Week 8 : Survey Research

Question 1

What is a survey?

Answer 1

• highly structured… close ended questions & fixed response options
• quantitative
• large sample size
• large number of questions

Question 2

Why survey? (3)

Answer 2

1. quantify change over time
2. effectiveness
3. breadth of topics

Question 3

Why survey?

1 - quantify change over time

Answer 3

• within-person change
• cohorts
• societal trends
• done through… cross sectional & longitudinal (repeated cross sectional & panel designs)
• MUST ask same questions every time/place when doing longitudinal

Question 4

Why survey?

2 - effectiveness

Answer 4

• standardized formats (consistent measures)
• short questions & multiple choice answers
• large sample possible
• representativeness (generalizability)… high external validity if used a probability sample

Question 5

Why survey?

3 - breadth of topics

Answer 5

• can include many questions/topics
• easy to replicate
• data for many studies (can be used by researchers with diverse research interests)

Question 6

Primary vs secondary data

Answer 6

• primary data… social scientists can design & carry out their own surveys
• secondary data… use & analyze data from large surveys carried out by others like the government & large universities

Question 7

Challenges of survey research (4)

Answer 7

1. nonresponse
2. coverage error
3. sampling error
4. measurement error

Question 8

Challenges of survey research (4)

1 - nonresponse

Answer 8

• Survey nonresponse… no participation at all
• Question nonresponse… no response to particular questions on the survey
• Systematic error… individuals who do and do not respond differ in systematic ways! (bias)

Question 9

challenges of survey research (4)

1 - nonresponse solutions

Answer 9

1. anticipate potential errors with a literature review
2. write good survey questions
3. organize the questions (professionalism is key!)
4. test & revise the survey questions before launch

Question 10

challenges of survey research (4)

2 - coverage error

Answer 10

• the sampling frame does not adequately capture all members of the target population
• either by systematically omitting some or by including others more than once
  *

Question 11

challenges of survey research (4)

2 - coverage error solutions

Answer 11

• Anticipate the potential errors (lit review)
• Address the errors in sampling strategy
• Acknowledge any systematic errors

Question 12

Challenges of survey research (4)

3 - sampling error

Answer 12

• sampling does not represent the population accurately
  *

Question 13

Challenges of survey research (4)

4 - measurement error

Answer 13

• occurs when the approach used to measure a particular variable affects the response provided
• Source of error is the survey design, the interviewer, the setting, etc.

Question 14

4 modes of administration for surveys

Answer 14

1. face-to-face
2. telephone surveys
3. mail or Self-administered questionnaires (SAQs) surveys
4. online surveys

Question 15

Criterea for deciding which mode of adiminstration to use

Answer 15

• cost
• response rate
• researcher control over data-collection process
• potential for interviewer effects

Question 16

1 - face-to-face

Answer 16

• interview schedule
• high cost, response rate, researcher control & potential for interviewer effects
• interviewers make observations about the setting too (para-data)
• ensures participants answer more honestly about like height & weight and visible things like that

Question 17

2 - telephone surveys

Answer 17

• Structured interview administered over the phone
• moderate cost, high response rate, moderate researcher control & potential for interviewer effects
• pros… interviewer can guide, less planning, can guage opinions on emerging issues
• cons… interviewer effects, fatigue, cell phones, ppl don’t wanna be bothered & no showcards (to remember responses)

Question 18

3 - Mail or Self-administered questionnaires (SAQs) surveys

Answer 18

• low cost, response rate, researcher control & potential for interviewer effects
• questionairre
• cons… more missing data/idks, can’t ensure targetted repsondent acc filled it out & very low response rate

Question 19

4 - Online surveys

Answer 19

• low cost, moderate response rate, moderate researcher control & low interviewer effects
• pros… little risk of data entry errors, easy nevigation & reach a lot of ppl
• cons… ‘diginal divide’ (access), can’t be probability sample, ethical concerns maybe

Question 20

Sampling for surveys…

Answer 20

• the goal is to quantify while elimintation systematic errors & using stats to estimate sampling errors
• probability samples… stratified sampling good but very expensive
• non-probability samples… systematic bias (Wier et al.), useful but not really

Question 21

Sampling for surveys - quota sampling

Answer 21

• Convenient sample (non-probability)
• Can have quota to mirror population characteristics
• Quota ≠ Strata
• e.g. online sample services , qualtrics, Amazon M Turk, etc.
• more affordable that probability samples
• Still: systematic error!

Question 22

ethical considerations…

Answer 22

• Most surveys are not sensitive, but some are! (e.g. HIV, drug use, sexual orientation)
• confidentiality… surveys obtain highly personal and sensitive information & if this information is made public, it could be a source of harm to respondents (use identification IDs)
• anonymity… unrealistic & unpractical, can’t do follow-ups if they can’t figure out who they are
• can cause minor psychological distress

Question 23

Typical steps to survey research

Answer 23

1. research questions
2. conceptualization & operationalization (Theory -> Hypotheses -> Concepts….. -> Variables)
3. sample & survey mode
4. survey questionairre (write & organize)
5. data collection
6. data processing
7. analysis

Question 24

Survey content (organization)

structure of questions

Answer 24

• The stem poses the question & the response categories provide the pre-set answers that the respondent may select

Question 25

# Survey content (organization) Closed-ended vs Open-ended questions

Answer 25

* closed ended... Answer options must be mutually exclusive & exhaustive, often used to measure the frequency of behaviours & many use a rating scale * open ended... only require a brief answer, more personal, can be acc question or just 'other: (specify)

Question 26

# survey content organization of questions...

Answer 26

* skip patterns & screener questions * establish trust & rapport early in the survey * avoid monotony & response set (switch between positively & negatively worded questions) * avoid order effects (improve by adding sections) * avoid priming effects

Question 27

# survey content Single Item vs Composite…

Answer 27

* single item... one variable = one survey question * composite variable... one variable = many survey questions & also need to specify how you will combine them (either indexing or scaling)

Question 28

# Characteristics of High-Quality Questions 4 characteristics of high-quality questions...

Answer 28

1. use clear & simple language 2. be precise 3. exercise caution when dealing with sensitive topics 4. avoid leading questions

Question 29

# Characteristics of High-Quality Questions 1 - use clear & simple language

Answer 29

* brief, clear & concise * no academic jargon * double negatives should be avoided cuz they are difficult to understand (e.g. "do you favor or oppose not legalizing physician- assisted suicide?")

Question 30

# Characteristis of high-quality questions 2 - be precise

Answer 30

* avoid double barelled questions... * e.g. "How much confidence do you have in the Prime Minister’s ability to handle domestic and foreign policy” * do not assume prior knowledge of respondent

Question 31

# Characteristis of high-quality questions 3 - caution with sensitive topics

Answer 31

* use neutral language that does not stigmatize the behaviour of interest * questions should avoid emotionally laden words that might upset respondents

Question 32

# Characteristis of high-quality questions 4 - avoid leading questions

Answer 32

* questions should be as fair & as even-handed as possible * veading questions are phrased to lead or guide responses to a particular answer, yielding results that support the organization's goals * may include a morally charged word that pushes respondents toward or away from a particular answer * avoid acquiescence bias, in which some respondents answer 'agree' to attitudinal questions (yes/no options)

Question 33

recap

Answer 33

* Thoughtful sampling can help a lot with coverage error and sampling error * Post-survey weighting can help with survey non- response * Some statistical procedures can help with question non- response (missing values) * But a large part of survey success lies in the questionnaire ○ Pretest ○ Preliminary data analysis Cognitive interviews