Chapter 3: Survey research

Question 1

Three ways to measure a variable with their problems

Answer 1

1. Psychiological indicators: can be problematic because of confounding variables, measuring can be stressful, practical issue to test multiple people at once
2. Questionnaires: biases, accuracy of responses
3. Observations: differences between people, confirmation bias, behave differently when observed

Question 2

Reliability

Answer 2

Refers to consistency
A necessary condition for construct validity, but it is not because there is a high reliability that there immediately is a good validity!
Three types of reliability: test-retest, interrator and internal

Question 3

Test-retest reliability

Answer 3

We measure the variable at two measurement moments (same measure, same group of people)
High test-retest reliability: people have similar scores on both measures
Assumes stability in construct across measurement moments

Question 4

Interrater reliability

Answer 4

At least two raters who observe behavior (observation studies or qualitative research)
Reliability of observations is measured by correlate (r) scores of both raters → consistent: high correlations between scores
Correlations can only be used on interval variables, when rating a nominal variable we use Cohen’s Kappa

Question 5

Internal consistency reliability

Answer 5

Useful for multi-item scales measuring the same construct
We expect that items measuring the same construct correlate positively with each other → average inter-item correlation
Cronbach’s alpha to compute variable: the higher the score, the more reliable the results are (>0.70) → better alternative: omega

Question 6

Two subjective ways to assess validity

Answer 6

1. Face validity: does the measure appear to be a good indicator of the construct?
2. Content validity: does the measure contain all relevant aspects of the construct?

Question 7

Three empirical routes that use data to assess validity

Answer 7

1. Convergent validity
2. Discriminant validity
3. Criterion validity

Question 8

Convergent validity

Answer 8

You expect that there will be a (positive/negative) relationship between our measure and another variable we measured
How to measure: give people two questionnaires, evidence for convergent validity if the scores on both tests have a relationship with eachother

Question 9

Discriminant validity

Answer 9

You expect that there will be no relationship between our measure and another variable we measured
We don’t expect people who score high on a measure to score high/low on another scale or measure (that has nothing to do with the construct we measure)
There is evidence for discriminant validity when there is no or a very weak correlation between both variables

Question 10

Criterion validity

Answer 10

Is there evidence for a relationship between the measure and certain behaviors?
Can we predict how people would behave or how they would score on another measure based on this measure?
Makes use of the known-groups paradigm
Idea: think about a relationship of the variable with a certain outcome, collect data and check whether data supports expected pattern

Question 11

Experience sampling methods

Answer 11

A representative sampling of immediate experiences in one’s natural environment
Singal-contingent sampling: send a notification through an app on phone and they have to react to it
Event-contingent sampling: fill out a survey when something happens
Strengths: high ecological validity, less sensitive to memory bias and common method bias, ideal to study changes in variables over time
Weaknesses: high participant burden → high drop-out, testing effects or reactivity

Question 12

Four things to consider when developing a scale

Answer 12

1. Choosing the question format
2. Writing well-worded questions
3. Encouraging accurate responses
4. Validating the scale

Question 13

Choosing the question format

Answer 13

Open questions: a question and a textbox where people write their answers
Closed question: answer options like dichotomous (yes/no), nominal (gender), ordinal (degree), interval

Question 14

Four types of scales

Answer 14

1. Likert scale: bipolar, indicate to which extent you agree, 6-7 answer options is recommended to eliminate neutral middle category
2. Visual analogue scale: measuring pain level with ‘faces’ pain rating scale
3. Semantic differential scale: scale with two endpoints that are opposites of each other, indicate where they would fall in between both points
4. Guttman scale: statements and indicate whether or not they agree with them

Question 15

Writing well-worded questions

Answer 15

Pay attention to:
Leading questions: formulated in a way that you automatically tend to choose a certain option
Double-barreled questions: questions that contain two questions and ask you to rate two different things at a time
Double negotiations: too complex
Order of questions: first questions might influence way people react to the next ones

Question 16

Encouraging accurate answers

Answer 16

How much can we rely on self-reports: we tend to report more than we actually know + recollection bias
Respons bias

Question 17

Four types of response biases

Answer 17

1. Acquiescence: always selecting the same answer because you’re not paying attention
2. Fence sitting: always choosing the middle/neutral option to not pick a side
3. Social desirability: portraying a better picture of yourself by choosing the option you think is socially acceptable
4. Faking bad (malingering): portraying yourself worse than you are

Question 18

Three types of miscomprehensions

Answer 18

1. Instructional miscomprehensions: not reading the instructions
2. Sentential miscomprehensions: interpreting questions differently
3. Lexical miscomprehensions: not understanding something in the question

Question 19

Nine methods to detect ‘low quality data’

Answer 19

1. Self-report items: ‘did you answer truthfully?’
2. Bogus items: incorrect/impossible questions
3. Instructed items: ‘please indicate …’
4. Too fast response times
5. Longstring: choosing the same answer options for nine times in a row
6. Individual response variability
7. Psychometric synonyms: opposite questions
8. Personal reliability
9. Mahalonobis D: scores that are very different from others → way to detect outliers

Question 20

Observations

Answer 20

In a lot of situations, observations are better than self-reported data
Usually, there is a difference between hypothetical and real behavior in self-reports
Problems: observer bias, observer effect, reactivity

Question 21

Observer bias
Observer effect
Reactivity

Answer 21

Observer bias: observers see what they expect to see and would like to see it confirmed
Observer effect: participants behave in line with expectations of the observer
Reactivity: participants react to being observed and change their behavior

Question 22

Six ways to assure reliable and valid observations

Answer 22

1. Unobtrusive observations: observing people who don’t know they’re being observed
2. Wait it out: people can’t keep changing their behavior
3. Measure the behavior’s result
4. Use a codebook: describe what behavior you want to see
5. Interrater reliability: use multiple observers
6. Blind and double-blind design: participants don’t know the hypothesis / observer also doesn’t know hypothesis