Class 8: Research and methods

Question 1

Stantcheva: What are the benefits of using survey data?

Answer 1

Surveys allow researchers to directly elicit otherwise invisible factors like beliefs, attitudes, and perceptions, creating controlled variation instead of relying on observational data on behavior

Question 2

Stantcheva: What is coverage error?

Answer 2

The difference between the potential pool of respondents and the target population -> for example with online survey, you cannot survey people who are not online, even though they might be in your target population

Question 3

Stantcheva: What is sampling error?

Answer 3

The difference between the potential pool of respondents and your planned sample (the fact you are only drawing a sample)

Question 4

Stantcheva: What is non-response error?

Answer 4

The difference between the target sample and actual sample, due to respondents ignoring the survey invitation or refusing to participate

Question 5

Stantcheva: What is attrition?

Answer 5

Respondents dropping out of the study before completing it -> if not random (for example connected to specific respondent characteristics), introduces bias

Question 6

Stantcheva: What is selection bias?

Answer 6

The difference between those who start they survey and those in the target population

Question 7

Stantcheva: What are 4 benefits of online surveys in terms of selection?

Answer 7

1. Flexibility for respondents
2. Convenience of technology for filling it out
3. Reaching hard-to-reach populations such as the youth, people in remote areas etc.
4. Variety of potential rewards for taking the survey

Question 8

Stantcheva: What are 3 good practices before respondents start the survey?

Answer 8

1. Provide information on how data is stored
2. Assure anonymity and confidentiality
3. Provide limited information about the purpose to not bias respondents

Question 9

Stantcheva: 3 ways to prevent/minimize attrition

Answer 9

1. Smooth respondent experience
2. Shorter survey
3. Incentives

Question 10

Stantcheva: What are 3 response biases?

Answer 10

1. Moderacy response bias: choosing a middle category every question
2. Extreme response bias: choosing extreme values every question
3. Response order bias: the order of the response options affects responses (systematically choosing first option)

Question 11

Stantcheva: What are cognitive-based and normative-based order effects?

Answer 11

Cognitive: priming (content becomes salient in later questions), carryover (answering questions similarly)

Normative: wanting to appear fair, consistent, and moderate

Question 12

Stantcheva: What is social desirability bias?

Answer 12

The desire of respondents to avoid embarassment and project a favorable image to others, resulting in not revealing their actual attitudes

Question 13

Stantcheva: 2 ways to reduce social desirability bias

Answer 13

1. Online surveys with no surveyor
2. Assurance of anonymity and confidentiality
3. Reminding people before sensitive questions of anonymity

Question 14

Stantcheva: What is acquiescene bias?

Answer 14

The tendency to answer questions in a positive way, such as systematically selecting agree, true, or yes

Question 15

Stantcheva: 2 ways to minimize acquiescene bias

Answer 15

1. Asking clear, unambiguous questions
2. Avoid questions that only have options agree/disagree, true/false, yes/no

Question 16

Stantcheva: What are 3 challenges of survey experiments?

Answer 16

1. Risk of confounding and pre-treatment contamination
2. Different respondents interpreting treatment in different ways
3. How well treatment mimics real-world treatment (external validity)

Question 17

Stantcheva: 3 types of survey experiments

Answer 17

1. Information treatments
2. Priming treatments
3. Factorial experiments (vignette and conjoint)

Question 18

Stantcheva: Between-subject vs. within-subject designs

Answer 18

Between-subject: each respondent only receives one treatment

Within-subject: Each respondent is subject to multiple treatments, in different orders

Question 19

Stantcheva: What is a conjoint experiment?

Answer 19

List descriptions of people and situations that vary attributes

Question 20

Stantcheva: 2 benefits and 1 disadvantage of factorial designs

Answer 20

They present realistic, hypothetical scenarios

They limit social desirability bias

Limited external validity, as people might not make similar choices in real life

Question 21

Stantcheva: How to measure persistence of treatment?

Answer 21

Follow-up surveys

Question 22

Wood et al.: How are left behind communities often defined, and how do Wood et al. define them?

Answer 22

Traditionally: Working class, socially excluded, lacking in education, and ethnically white

Wood et al.: Residents of places that are stigmatized by elite actors as harboring support for populist views due to those places having multiple forms of deprivation and a lack of education

Question 23

Wood et al.: What is the RQ?

Answer 23

What are the preferences of left behind communities for policy change processes related to populist politics?

Question 24

Wood et al.: What are the two methodological challenges of assessing policy preferences of left behind communities, and what does it lead researchers to do?

Answer 24

1. Preferences are difficult to asses because of emotional attachment and affective polarization
2. Because they are socially stigmatized as poor and lacking education, which creates social desirability bias and distrust of researchers and hestitancy to participate

This leads researchers to oversimply left behind views, often conflating them with populist narratives and ignoring the complexities of their views

Question 25

Wood et al.: What is stigmatization?

Answer 25

The process in which a particular group and individuals develop stigma as they are subjected to shame, scorn, ridicule or discrimination in their interactions with others

Question 26

Wood et al.: How do they overcome the methodological challenges?

Answer 26

By using photo elicitation which allows them to process emotions they have related to a subject and avoids social pressures associated with stigmatization

Question 27

Wood et al.: What is photo elicitation?

Answer 27

Participants consider a photograph of a contentious issue and clarify their preferences in dialogue with researchers following reflection, allowing them to speak freely about contentious issues

Question 28

Wood et al.: Research methods

Answer 28

418 interviews in 5 left-behind communities in the UK through street intercept recruitment method Focus on claim that they voted for Brexit to improve NHS investment by showing picture of bus: "We send the EU 350 million pounds a week. Let's fund our NHS instead" and asking what comes to mind and nudging them towards legitimacy and accountability Photo elicitation and dialogue with the researchers, who engaged positively and noted down their responses afterwards

Question 29

Wood et al.: What is the traditional understanding of why left-behind communities voted Leave?

Answer 29

They want to reduce immigration through Brexit to increase access to public goods such as the NHS

Question 30

Wood et al.: What 6 themes do they identify?

Answer 30

1. "Bullshit" and skepticism about the bus statement contrary to expectations of support 2. Distrust in politicians and distancing from politicians and politics 3. Need for NHS investments 4. Lack of clarity and conflicting views regarding responsibility for improving NHS 5. Lack of link between NHS and Brexit -> either failed to recognize or outright rejected link 6. Anti-immigration sentiment among a minority of hard Leavers

Question 31

Wood et al.: What is positive marginalization, and how does it work in this case?

Answer 31

Cognitive process through which stigmatized groups can reinterpret stigma positively by rejecting the logic of elite positions and then articulating distinct policy preferences through their own personal experiences In the themes bullshit and distrust of politicians, left-behind communities reject elite Leave positions and framings, and then articulate their own policy preferences of NHS investment

Question 32

Wood et al.: What do their results say about the general narrative of why left-behind communities voted leave?

Answer 32

There is no link identified between Brexit and wanting NHS investment There is dissonance between support for investment and distrust that it will happen as a result of Brexit

Question 33

Elkjær & Wlezien: What is the traditional wisdom regarding a don't know (DK) option in surveys on preferences and opinions?

Answer 33

Offering a DK response option encourages satisficing = respondents use the option even when they actually have a preference

Question 34

Elkjær & Wlezien: What do they add to the conventional wisdom on DK options?

Answer 34

That while it is correct, at the same time it downplays the possibility that omitting the option encourages respondents without preferences to give one (providing random responses that bias the results)

Question 35

Elkjær & Wlezien: Research methods

Answer 35

4800 US respondents IV: having the don't know option or not DV: Preferences on 8 policies After each question, the respondents who expressed a preference were asked how sure they felt about their answer Afterwards, respondents were asked to revisit 3 of their responses, where the control group now receive the DK treatment while those in the treatment do not

Question 36

Elkjær & Wlezien: What are their 5 main findings?

Answer 36

Respondents are more likely to provide non-responses when offered the DK option -> allowing them to skip a question is not effective The more information needed to answer the question, the more the DK option is used Respondents with lower levels of political knowledge are more likely to choose DK, whereas high-information respondents are virtually unaffected by the inclusion of a DK-response option Respondents offered the DK option are more confident in their responses Omitting the DK option can bias aggregate opinion toward 50-50 splits

Question 37

Elkjær & Wlezien: When should you include/exclude a DK option?

Answer 37

For high-salience issues where most respondents already have an opinion, DKs mostly reflect satisficing -> omit For low-salience issues, many respondents will not have an opinion or be unsure -> include

Question 38

Egami & Hartman: What is external validity?

Answer 38

Whether a study's results can be applied to other people, situations, treatments, or outcomes

Question 39

Egami & Hartman: What are the 4 types of external validity?

Answer 39

X-, T-, Y- and C-validity

Question 40

Egami & Hartman: What is X-validity?

Answer 40

Problem: Can the results be generalized to a different group of people/population? Is the sample representative of the target population? Solution: Adjust for differences between the study sample (for example students) and target population (for example overall population) using covariates

Question 41

Egami & Hartman: What is T-validity?

Answer 41

Problem: Would the same effect occur if the treatment were implemented differently? Does it reflect how it works in the real world? Does it seem realistic? Solution: Design experiment so the treatment matches real-world intervention as closely as possible

Question 42

Egami & Hartman: What is Y-validity?

Answer 42

Problem: Would the same effect be found if a different or more realistic outcome was measured? Do we measure the right thing? Solution: Use outcomes that are as close as possible to the real-world behavior or policy goal

Question 43

Egami & Hartman: What is C-validity?

Answer 43

Problem: Would the effect be the same in a different setting, time, or institutional environment (where/when)? Is this contingent on a specific setting? Solution: Adjust for or match on key "context moderators" that explain why results might differ across time, geography or institutions

Question 44

Egami & Hartman: What is effect-generalization and what is it useful for?

Answer 44

Estimating how big the treatment effect would be in a new setting Useful for X- and C-validity, because you can statistically adjust for differences in who is studied and where the study takes place

Question 45

Egami & Hartman: What is sign-generalization and what is it useful for?

Answer 45

Testing whether the direction of the effect (positive/negative) stays the same in other settings Useful for T- and Y-validity, as it requires fewer assumptions. Also helpful for X- and C-validity when effect-generalization is too hard

Question 46

Egami & Hartman: What are the 3 types of effect-generalization estimators?

Answer 46

Weighting-based: Adjusts the sample to resemble the target population by applying weights Outcome-based: Models the outcome based on characteristics and applies it to the target group Doubly-robust: Combines both weighting and outcome modeling

Question 47

Cavaillé et al.: What is preference intensity?

Answer 47

How strongly someone prefers one policy option over another, predicting whether they will act in opinion-congruent ways

Question 48

Cavaillé et al.: Why are usual measures of opinion insufficient?

Answer 48

They capture what people think, but not how much they care. They also allow cheap talk and partisan signaling, leading to bunching at the extremes without differentiating true commitment

Question 49

Cavaillé et al.: What are the advantages and disadvantages of Likert and Likert+?

Answer 49

Simplicity and lower drop-out rates! Likert rates support/opposition on simple scale, Likert+ adds follow-up question of how important the issue is to the respondents But puts respondents in a world where talk is cheap and there are no consequences for misrepresenting your opinion or making one up, for example to stay within party lines

Question 50

Cavaillé et al.: What are the advantages and disadvantages of Quadratic Voting for Survey Research (QVSR)?

Answer 50

QVSR gives respondents a fixed budget to buy favor/oppose votes, which prices increasing quadratically This better approximates real-world opportunity costs and induces people to realize what they care about the most, regardless of partisan views But is difficult and costly to scale up to larger surveys, requires higher cognitive engagement, and an appropriately sized fixed budget

Question 51

Cavaillé et al.: Research methods, incl. IV and DV

Answer 51

US citizens asked opinion on 10 policy issues, randomly varying method used to measure their opinion IV: Likert (1-3), Likert+, or QVSR (100 credits) DV: opinions on 10 policy choices + asked if they would donate money to advocacy group or write to Senator

Question 52

Cavaillé et al.: What are their main findings?

Answer 52

All 3 methods predict opinion-congruent behavior, but QVSR works best Individuals who choose end-of-scale response categories in Likert(+) end up de-bunching under the QVSR in ways that align with the donation/writing task QVSR helps distinguish between respondents who directly benefit from policy and respondents who don't. In question on gender equality, with Likert there is little variation based on gender, but QVSR shows that women are more likely than men to strongly support gender equality policies