Policy Analysis Exam Flashcards
(185 cards)
1
Q
What do the following acronyms stand for: T, C, RCT, PE, PIE, and SRS?
A
T = treatment or treatment group
C = control group
RCT = randomised controlled trials
PE = policy evaluation
PIE = policy impact evaluation
SRS = simple random sampling
2
Q
What are the two different categories of evaluation?
A
- Operational evaluations: examines how effectively programs were implemented and whether there are gaps between planned and realized outcomes.
- Impact evaluations: studies whether the changes in well-being are due to the program intervention and not to other factors.
3
Q
Define Policy Impact Evaluation (PIE).
A
- An assessment of the causal effect of a specific intervention on some measurable outcomes in a target population.
- This causal effect is called treatment / intervention impact.
- This also includes assessing if a policy reaches the goals for which it was implemented
4
Q
What questions should we ask when determining the difference between intervention design and evaluation design?
A
- Who is more in need?
- Who will be more reactive to the intervention?
- Who is less at risk of adverse outcomes?
- There are also political constraints.
5
Q
What is the main tradeoff in PIE?
A
Effectiveness and generalisability.
6
Q
What is the issue of heterogeneity?
A
- That the effects of a program and inputs affecting outcomes may vary over its expected lifetime.
- Thus, monitoring long-term as well as short-term outcomes may be of interest to policymakers.
7
Q
What are the four criteria for effective intervention designs?
A
- Efficacy: establish a detailed, plausible chain of causal mechanisms.
- Compliance: promote real participation to T.
- Cost-effectiveness.
- Standardisation: a reasonable degree of uniformity.
8
Q
What are some examples of social issues?
A
- Community support programs improve the health outcomes of babies.
- Reducing the size of classrooms in primary schools benefits disadvantaged students.
- Urban desegregation improves the occupational opportunities of ethnic minorities.
- Smoking marijuana enhances the risk of cancer.
9
Q
What is causality?
A
If x (cause, treatment, intervention) is changed, there will be a change in y (effect, impact, outcome).
10
Q
What are four reasons for which PIE are important?
A
- Informs evidence of outcomes.
- Cost-effectiveness.
- Accountability.
- Job opportunities.
11
Q
What does ex-ante PIE predict?
A
The policies impacts and its viability using data before the program intervention.
12
Q
What does ex-post PIE predict?
A
It predicts outcomes after programs have been implemented.
13
Q
What are reflexive comparisons?
A
A type of ex-post evaluation; they examine program impacts through the difference in participant outcomes before and after program implementation (or across participants and nonparticipants).
14
Q
A
15
Q
What is the main challenge across different types of policy evaluation?
A
To find a good counterfactual—namely, the situation a participating subject would have experienced had he or she not been exposed to the program.
16
Q
When were policy evaluations invented, and who uses them?
A
1980s. Policymakers, practitioners, private foundations, NGOs, associations, beneficiaries, proponents and skeptics. Mostly used in Anglo-Saxon countries.
17
Q
What is the minimum amount of time to complete a policy evaluation?
A
Six months
18
Q
What is the first step of every evaluation?
A
To determine who are the key stakeholders, which may include individuals who have:
- Access to the field,
- Data, context and information,
- Funding and support,
- Interferences.
19
Q
Why must the policy goal be specified clearly?
A
- For it to be measurable.
- The policymaker will state the outcome in a very generic and vague way – to the researcher that clearly identifies how to measure the efficacy of the policy.
- It is key, for a researcher, to establish the specific outcomes so that the policymakers (who come up with the idea) cannot argue against the results once released.
20
Q
Are ex-ante or ex-post evaluations preferable?
A
Ex-ante (may be referred to as simulations)
21
Q
What is input?
A
Set of resources required for a policy, including economic resources (data), human resources (IT employees to create an app).
22
Q
What is output?
A
The set of products/results/deliverables of the policy that comes from the actions of the policy. For example, how many people have downloaded the app?
23
Q
What are outcomes?
A
Outcomes refers to the causal impact and the net effect of the policy – how it changed the course of events of individuals and groups.
24
Q
What is the main difference between output and outcomes?
A
The causal effect and the fact that outputs focus on the results produced during the program, and not after.
25
What is the main tension between policymakers and researchers?
Time. There is a tension between policymakers that want to know the results as soon as possible, and the fact that researchers are interested in long-term outcomes and not short-term outcomes.
26
When the EU assesses policies, what do they evaluate?
1. Budget
2. Legal
3. Deliverables
27
What is the mixed methods approach?
It mixes quantitative and qualitative methods such as PIE and the process assessment.
28
What is the theory of change?
An explanaton of the process of change by outlining causal linkages in an initiative. It is best to start by the long-term outcomes and work backwards to treatment implementation, identifying hurdles.
29
Define nudging and provide an example.
An attempt at influencing people’s behaviour in a predictable way by using their biases and habits.
For example, sending people a reminder to book a doctor’s appointment, or placing healthy foods at eye level at supermarkets.
30
What is Kahneman's 'Dual Process Theory'?
At times, humans act irrationally and not in their interests because they have two systems of thought:
- System 1 (intuition): fast, unconscious, automatic, everyday decisions, error prone.
- System 2 (reasoning): slow, conscious, effortful, complex decisions, reliable.
31
What does the EAST Framework intend to do?
To encourage behaviour (nudges) to make it easy, attractive, social and timely.
32
What are the components of the EAST Framework and their biases?
* Easy: reducing effort. Biases: endowment effect, status quo effect, cognitive overload.
* Attractive: presenting benefits.Biases: availability bias, anchoring effect, loss aversion, optimism bias, scarcity bias.
* Social: harnessing social pressure. Biases: confirmation bias, harding, commitment bias, authority bias.
* Timely: prompting when they are most receptive. Biases: present bias, hyperbolic discounting, duration neglect, hot/cold states.
33
What is the endowment effect?
How individuals place more value on items they own, than the same items they don’t own.
Example: shares inherited from deceased relative.
34
What is the status quo effect?
How individuals prefer to maintain their current situation, and oppose actions that may change it.
Example: gender norms.
35
What is cognitive overload?
How individuals are not capable of processing more than a certain amount of information at a time.
Example: overloaded nurse.
36
What is the availability bias?
How individuals rely on information that comes readily to mind.
Example: plane crashes in contrast with car crashes.
37
What is the anchoring effect?
How individuals tend to rely and base their decisions on the first piece of information offered.
Example: products at discount.
38
What is loss aversion?
How individuals experience losses more intensely than equivalent gains.
Example: loss or gain of $100.
39
What is optimism bias?
How individuals overestimate the likelihood of positive events and underestimate the likelihood of negative events.
Example: refusal to have job security fund.
40
What is scarcity bias?
How the more difficult it is to acquire an item, the more value is placed upon it.
For example: the sale of luxury goods.
41
What is confirmation bias?
How individuals prefer information that confirms their current beliefs.
Example: certain news outlets.
42
What is herd mentality?
How individuals can be influenced by the majority.
Example: panic reactions in crowds.
43
What is commitment bias?
How individuals tend to stay committed to their past behaviours, particularly those exhibited in public.
Example: public commitment to exercise.
44
What is authority bias?
How individuals believe opinions of authority figures over others.
Example: doctors and dentists on toothpaste advertisements.
45
What is present bias?
How individuals give stronger weight to payoffs that are closer to present time.
Example: preferring $10 today, to $20 tomorrow.
46
What is hyperbolic discounting?
How individuals value immediate though smaller rewards than long-term, larger rewards.
Example: free shipping with bundle purchases.
47
What is duration neglect?
How individuals’ judgement of the unpleasantness of painful experiences depends very little on the duration of them.
Example: mothers giving birth.
48
What is the hot/cold empathy gap?
How individuals underestimate the influences of visceral drives on their own attitudes, preferences and behaviours, such as hunger.
Example: (cold) physicians medicating (hot) patients.
49
How can a researcher design an effective treatment group ex-ante?
While difficult:
* Use empirical evidence of previous PIE and PE,
* Have deep qualitative knowledge of the context,
* Conduct pilot studies, and
* Use implementation analysis.
50
List the four types of correlational approaches.
* Ecological and comparative analyses
* Time comparisons: pre-post
* Matched-case control studies
* Difference-in-differences
51
List the three types of causal approaches (PIE).
* RCT
* Regression discontinuity
* Instrumental variables
52
What are the steps required to determine the outcomes? (provided visual)
1. Policy issue, policy goal, outcomes
2. Or - input, intervention, output, outcomes
53
What are the steps/types of implementation in policy evaluation?
* Budget analysis
* Legal audits
* Reception of the beneficiaries
* Feedbacks from implementers
* Analysis of the deliverables
54
Define treatment impact, or intervention effect?
Changes in outcomes that are attributable to the treatment.
The difference between the outcome some time after the intervention has been implemented, and the outcome had the intervention not been implemented.
55
Define counterfactual.
The state of the world that participants would have experienced in the absence of the intervention.
That is, the outcome had the intervention not been implemented.
56
Define average treatment (intervention) effect.
A measure used to compare treatments in randomised experiments, evaluation of policy interventions, and medical trials.
It measures the difference in mean outcomes between units assigned to the treatment and units assigned to the control.
57
What is the fundamental problem of causal inference?
Intervention and the counterfactual cannot be observed simultaneously.
Individual-level causal inference is impossible.
58
How is a counterfactual constructed?
By making a control group, which involves selecting a group of individuals who did not participate in the intervention.
Goal: to reliably attribute any difference in outcome between the treated and control group after the treatment. RCTs are good for this.
59
How do correlational approaches work?
Correlational approaches to PIE use comparisons over space/time and related variations of policies and outcomes to approximate the counterfactual in the absence of random assignment.
The issue is it has a large number of confounding factors, and the comparison between the groups are not really comparable.
Thus, it is difficult to isolate the specific causal impact of the intervention.
60
What are pre-post correlational approaches?
An assessment conducted at the start and at the end of a term, in order to observe changes caused by an intervention.
61
Define the diff-in-diff approach.
Differences-in-differences: calculate the effect of a treatment (independent variable) on an outcome (dependent variable) by comparing the average change over time in the outcome variable for the treatment group to the average change over time for the control group.
For example, testing minimum wage in Paris, but not in Marseille, and observing the differences between outcomes.
It tries to mitigate the effects of extraneous factors and selection bias. However, it is still subject to biases such as mean regression, reverse causality, and omitted variable bias.
62
What are the levels of diff-in-diff approaches?
* Post-intervention assessment,
* Pre & post intervention assessment,
* DFD for time trend,
* DFD with follow-up and fade out or maturation,
* DFD with follow-up and concomitant factors,
* Matched-case control study with DFD and follow-up,
* Matched-case control study.
63
How does the ecological / comparative approach work?
It considers the context of public administration, many different disciplines, and takes a comparative approach across regions and governments.
For example, when studying the impact of urban segregation on employment outcomes of minority groups, we must consider ethnicity, SES, cultural attitudes, sector composition, industrialisation, globalization, GDP per capita, current policies (employment, educational, welfare).
64
What are the advantages and disadvantages of the ecological / comparative approach?
Advantages: holistic, context, policy
Disadvantages: complex, subjectivity, generalisability, bad data
65
When are causal inferences impossible?
When there is only distributional information about the data and no information about how the data was generated.
66
What is the purpose of PIE?
Learn lessons and have accountability
67
Define independent and dependent variables - and how they interact?
Independent variable are manipulated to affect the dependent variable.
Independent variable: the cause of a change.
Dependent variable: the observation of or the change itself.
Example of variables: gender, nationality, age, income, having a Velib pass, receiving the culture pass.
68
Define frequency distribution.
A representation that displays the number of observations within a given interval; it shows the frequency of each variable.
69
Define absolute frequency.
A count of the number of times something has occured.
70
Define relative frequency.
The number of times something has occurred divided by its frequency.
71
Define cumulative frequency.
The sum of relative frequencies.
72
Define bivariate analysis.
A statistical method examining how two different things are related.
73
What is the Pearson correlation coefficient?
A number between 1 and -1 that indicates the strength and direction of a relationship between two variables.
74
What is a confounding variable?
A third variable that influences both the independent and dependent variable, causing a spurious association.
75
What are a spurious relationships?
False statistical relationships.
76
What is selection bias?
How participants in a study differ systematically from the population of interest, leading to a systematic error in association or outcome.
Example: health studies that recruit participants directly from clinics miss all the cases who don't attend those clinics or seek care during the study.
77
What are matched-case control studies?
In individually matched studies, the population of interest is identified and cases are randomly sampled or selected based on particular inclusion criteria.
78
What are the advantages and disadvantages of matched-case control studies?
Advantage: it removes the original confounding effect by matching factors, such as age or sex.
Disadvantages: if the matching factor is associated with the exposure of interest, it introduces a new bias because the matching factor itself is confounded.
Another is overmatching, which happens when cases and controls are matched on variables that have a high relevance to the exposure. For example, in a study of smoking and lung cancer, matching on the variable “carrying a cigarette lighter.”
79
In RCTs, what is the relationship between the two groups?
In RCTs, the two groups are perfectly equivalent because they are drawn from the same population of people who accepted to participate in the study.
80
Do RCTs have external validity?
No, they cannot be generalised to individuals who refused to participate. However, they do have internal validity.
81
Define randomisation.
If two participants are randomly extracted from the same population, on average they will display the same values on all parameters, or be equivalent.
82
RCTs = (3 factors)
RCTs = randomisation + manipulation + control
83
How is randomisation sold?
By:
* Showing the delayed/compensatory treatment;
* Explaining why comparability is important;
* Explaining that there are not enough resources to treat everyone;
* Using ‘action research’ instead of experiments.
84
Always use random assignment if it is ethical. If it is not...?
* It can be a temporary exclusion,
* It can be compensated for,
* Consider everyone has the same chance of getting it,
* Consider it benefits greater society.
85
With policy evaluations, are there any other ethical duties to consider?
There is an ethical duty to consider whether a policy evaluation will have negative outcomes, which is why there is informed and revocable consent, and an institutional review board.
86
What are the five types of randomisation structure?
1. Nested structure
2. Individual-level randomisation
3. Individual-level randomisation with blocking: divide individuals into blocks and randomise within them
- Pros: higher statistical power
- Cons: higher risk of contamination than non-blocked designs
4. Cluster randomisation (can be between schools or classes): assigning groups to randomisation instead of individuals
- Pros: more practical and cost-effective, less risk of contamination
- Cons: less statistical power, higher N and data collection costs
5. Cluster randomisation with blocking
87
Define factorial design.
An experiment that has multiple factors or independent variables.
88
Define placebos.
A treatment specifically designed to have no effect.
89
What methods can a researcher use to analyse experimental data?
* Ex-ante equivalence analysis
* Simple comparison of group means or relative frequencies of the outcome variables
* More advanced techniques for treatment compliance
90
What is the best possible design to assess impacts ex-ante?
PIE via RCTs
91
To assess the total quality of a policy evaluation, what can a researcher measure?
Internal and external validity – which must be valid causal inferences, generalisable to the relevant population that can benefit from the policy.
92
What are quasi-experimental methods of PIE?
It is the same as a true experiment, except it does not rely on random sampling and requires that subjects be assigned to groups based on non-random criteria.
93
Define Randomised Control Trials (RCTs)?
A trial in which subjects are randomly assigned to one of two groups – the experimental group receiving the intervention that is being tested, and the control group receiving an alternative treatment.
Subjects must be willing to participate in the intervention, and then halved into these groups.
94
What is internal validity?
The degree of confidence that the causal relationship is not influenced by other factors and variables.
95
When are quasi-experimental designs useful?
- When experiments are unethical or logistically difficult
- When randomisation is not accepted
- When the policy has been implemented
96
Who assigns treatment and control groups in quasi-experimental designs?
A third party
97
What are the four types of quasi-experimental designs?
- Regression discontinuity designs (RDD)
- Instrumental variables (IV)
- Difference-in-differences (DiD)
- Matching methods / propensity scores
98
What are some examples of regression discontinuity designs (RDDs)?
- Effect of training program for youth unemployment – assignment based on age.
- Effect of conditional cash transfers on contraceptive behaviour – assignment based on income.
- Effect of broadening access to the healthcare system on low-income children’s health – assignment based on income.
99
What is the intuition behind regression discontinuity designs (RDDs)?
1. Policies often use thresholds to determine eligibility
2. These thresholds are arbitrary: subjects located around them are almost identical
3. With RDD, those just above and below the threshold can be compared
100
What is the difference between randomised control trials (RCTs) and regression discontinuity designs (RDDs)?
RCT: researcher randomly assigns control and treatment group.
RDD: researcher randomly assigns based on a cut-off value.
101
What are the three limitations of regression discontinuity designs (RDDs)?
1. External validity
2. Must be many observations around the cutoff point
3. If the same cutoff is used for other programs, treatment effects are confounded
102
Why are instrumental variables used?
They are used to provide true effects, rather than biased effects.
103
What are some examples of instrumental variables in use?
1. Effect of contraceptive knowledge on number of babies – Z = mass media exposure
2. Effect of housing vouchers and educational attainment – Z = random assignment
104
Explain what exclusion and relevance variables must do.
Exclusion: it must have no direct effect on the variable of interest.
Relevance: it must have a significant correlation with the instrumented variable.
105
What are the limitations of instrumental variables?
1. The estimates of treatment impacts have high uncertainty when the relationship between variables is weak. (Test: F-test)
2. The exclusion variable is not respected and must be controlled for.
106
Identify and define the three families of causal effects?
1. Average treatment effect (ATE): the overall effect of the policy on the whole sample or population.
2. Conditional treatment effect (CTE): the overall effect of the policy on a specific subgroup.
3. Local average treatment effect (LATE): the effect of the policy on compliers in the whole sample or population.
107
Why sample?
Saves money and time - hence improves the quality of evaluation.
108
What are the five types of probability sampling?
1. Simple random
2. Systematic random
3. Area
4. Cluster
5. Stratified
109
What are the five types of non-probability sampling?
1. Voluntary
2. Convenience
3. Purposive
4. Snowball
5. Quota
110
What is the objective of sampling?
The objective of sampling is to approximate the characteristics that are relevant to the question about a larger population.
111
Identify and define the four limitations of sampling.
1. Population-specific error: when researchers do not understand who they should survey.
2. Selection error: when respondents self-select their participation in the study (only those that are interested respond).
3. Sample frame error: when the wrong sub-population is used to select a sample.
4. Non-response error: when potential respondents are not successfully contacted or refuse to respond.
112
Define confidence interval.
An indication of how precise the sample is and the degree of uncertainty of an estimate.
It is used to quantify sampling uncertainty.
113
What are the steps to sampling?
1. Define the target population
2. Find a reliable list of units of this population
3. Choose the most appropriate sampling method
4. Determine sample size
5. Implement sampling plan
114
Define statistical significance.
A measure of how ‘rare’ the results are, under the assumption that the null hypothesis is true.
115
Define probability sampling and provide an example.
Probability sampling: the randomised selection of a sample from a population.
For example, after a training course for the unemployed, a short questionnaire on customer satisfaction is left and participants may or may not fill it in.
116
Define non-probability sampling and provide an example.
Non-probability sampling: the subjective selection of a sample from a population.
For example, interviewing every house on a given street.
117
Define simple random sampling (SRS) and explain the procedure.
Simple random sampling: a type of probability sampling in which the researcher randomly selects a subset of participants from a population.
Procedure:
1. Assign a number to each unit in the list
2. Draw randomly N numbers
3. Select the units corresponding to these numbers
118
Define systematic sampling and explain the procedure.
Systematic sampling: a type of probability sampling where researchers select members of the population at a regular interval; for example, alphabetical.
Procedure:
1. Assign a number to each unit in the list
2. Determine k = population size / sample size
3. Draw randomly one number, select it, then select every kth element
119
Define stratified sampling and explain the procedure.
Stratified sampling: a type of probability sampling where researchers divide subjects into subgroups called strata based on characteristics they share; for example, race or gender.
Procedure:
1. Identify one or more relevant predictors of the outcome.
2. Create sublists of the population based on the categories of this predictor. For example, men and women.
3. For every stratum, draw a number of units corresponding to its share in the population (proportional stratification).
4. With these sublists, select the units using SRS or SS.
120
Define quota sampling and explain the procedure.
Quota sampling: a non-probability sampling method that relies on the non-random selection of a predetermined number or proportional of units.
Procedure:
1. Identify one or more relevant predictors of the outcome.
2. Create sublists of the population based on the categories of this predictor. For example, men and women.
3. For every stratum, draw a number of units corresponding to its share in the population (proportional stratification).
4. Select the units freely.
121
What are the drawbacks of quota sampling?
1. It is not random.
2. It is exposed to selective bias. For example, interviewers will interview easy interviewees more often.
122
Define clustered sampling and explain the procedure.
Clustered sampling: a probability sampling method in which researchers divide a population into clusters and then randomly select some of these clusters as the sample; for example, districts or schools.
Procedure:
1. Divide the population into groups of homogeneous units (clusters), usually based on geographical proximity.
2. Draw a random sample of these clusters, using SRS or systematic sampling.
3. All units from the selected clusters are selected (or can be drawn randomly from within each cluster).
123
How do researchers choose the best sampling method?
1. List research goal
2. Identify potential sampling methods
3. Test them
4. Choose the best
124
Define external validity.
External validity: the degree of confidence that the causal relationship being tested is not influenced by other factors or variables.
125
Define attrition.
Attrition: the loss of study units from a sample.
126
Define non-response bias.
Non-response bias: when those unwilling or unable to take part in a research study are different from those who do.
127
What does N signify?
The total number of individuals or observations in a sample.
128
What function is chosen in Excel to sample and randomise?
RANDBETWEEN
129
What four features must be calculated to determine sample size?
1. Population size
2. Margin of error
3. Confidence level
4. Expected variance
130
Define standard deviation.
Standard deviation: a measure of how dispersed the data is in relation to the mean.
131
Define effect size.
Effect size: how meaningful the difference between variables or the difference between groups is.
Null effect = 0
Small effect size = -0.05 to -0.15
Moderate effect size = -0.15 to 0.30
Large effect size = 0.30 and above
132
Define minimum detectable effect size (MDES).
Minimum detectable effect size (MDES): the effect below which we cannot precisely distinguish the effect from zero, even if it exists.
133
Define explained variance.
Explained variance: the proportion to which a design takes into account the variation of a data set.
134
Define Cohen's d (effect size).
Cohen’s d (effect size): measures the size difference between two groups.
135
Define null result (or effect).
Null result (or effect): a result without the expected content.
136
What is sensitivity analysis?
Sensitivity analysis: used to identify how much variations in the input values for a given variable impact the results for a model.
137
What are the seven steps to be taken to design a study?
1. Choose a randomisation type
2. Choose the MDES
3. Choose participants assigned to treatment
4. Estimate explained variance
5. If cluster randomisation, estimate RHO
6. Conduct sensitivity analysis
7. Integrate non-response and attrition rates
138
What information should be collected to increase explained variance?
1. Pre-treatment values (pretest)
2. Individual predictors
3. Contextual predictors
139
Define pretest.
Pretest: when researchers test questionnaires on some members of the treatment group to evaluate the reliability and validity of them before they are distributed.
140
What are three reasons why the pretest is important?
1. Measures the equivalence of the treatment and control groups
2. Analyses the dynamic effects of the treatment
3. Increases the precision of estimates (considerably)
141
Define statistical power (or sensitivity).
The likelihood of a significance test detecting an effect when there actually is one.
142
What are the five factors that affect statistical power?
1. N
2. Number of treatment and control groups
3. Percentage assigned to treatment group
4. Randomisation design
5. Sampling design
143
What are the seven types of data collection methods?
1. Experiment
2. Survey
3. Interview/focus group
4. Observation
5. Ethnography
6. Archival research
7. Secondary data collection
144
Define experiment data collection.
To test a causal relationship.
145
Define survey data collection.
To understand the general characteristics or opinions of a group of people.
146
Define interview or focus group data collection.
To gain an in-depth understanding of perceptions or opinions on a topic.
147
Define observation data collection.
To understand something in its natural setting.
148
Define ethnography data collection.
To study the culture of a community or organisation first-hand.
149
Define archival research data collection.
To understand current or historical events, conditions, or practices.
150
Define secondary data collection.
To analyse data from populations that cannot be accessed first-hand.
151
What is administrative data?
Data that organisations collect about their operations
152
What is big data?
Large and hard-to-manage volumes of data that inundate organisations.
153
What are computer-assisted personal interviews (CAPI)? List their advantages and disadvantages.
Data collected face-to-face with the assistance of technological devices that capture the responses.
Advantages: population coverage, reliable data, can be long and complex.
Disadvantages: expensive.
154
What are computer-assisted telephone interviews (CATI)? List their advantages and disadvantages.
Data collected from a trained interviewer calling from a call centre.
Advantages: population coverage, reliable data, cheap.
Disadvantages: response rates, can’t be long and complex.
155
What are computer-assisted web interviews (CAWI)? List their advantages and disadvantages.
Data collected through online surveys or interviews.
Advantages: cheap.
Disadvantages: response rates, can’t be long and complex, population coverage and unreliable data.
156
What does the quality of a study rely on?
1. Internal validity
2. External validity
3. Measurement
157
Identify the three aspects of internal validity and define each.
1. Initial equivalence: before the manipulation of the causal variable, participants in different conditions are the same on average on all the measured variables.
2. Treatment contamination: when members of the control group inadvertently receive the treatment or are exposed to the intervention.
3. Treatment replacement: when members of the treatment group are put back into the population before other members are sampled.
158
What are the five aspects of external validity?
1. Population list
2. Response rate
3. N
4. Sampling design
5. Randomisation design
159
What are the three aspects of measurement?
1. Time frame
2. Coverage
3. Validity of the indicators
160
What are the two types of external validity?
1. Population validity: how generalisable the findings are to other populations.
2. Ecological validity: how generalisable the findings are to other situations or settings.
161
What is the tradeoff between internal and external validity?
The more generalisable, the less extraneous variables are accounted for; and vice versa.
162
What are the seven factors that determine coverage?
1. Primary and secondary outcomes
2. Primary and instrumental outcomes
3. Positive and negative outcomes
4. Reception of the intervention
5. Baseline information to enhance statistical power
6. Contextual and individual moderators
163
Define heterogeneity analysis.
A measure of the differences between participants in a study.
164
What is an indicator?
An observed value of a variable.
165
Define reliability.
The extent to which results can be reproduced when the research is repeated under the same conditions.
166
Define validity.
The extent to which results measure what they are supposed to measure.
167
What are the four types of questionnaire designs? Define them.
1. Multiple choice questions
2. Open-ended questions
3. Likert scale: a scale used to represent people’s agreement or disagreement towards a statement or question on a topic
4. Semantic differentials: a scale used to represent a person’s attitude towards something using bipolar adjectives
168
What are the two golden rules for designing questionnaires?
1. Run a pretest
2. Keep in mind that people are not always accurate. They are subject to social desirability, short memory, superficiality, and are easy to influence
For example, respondents understate alcohol and drug use, and tax evasion and racial bias; while they overstate church attendance, charitable contributions, and the likelihood they will vote in elections.
169
Why does wording matter in questionnaires?
1. Clarity and understanding
2. Consistency
3. Bias prevention
4. Sensitivity
5. Response options
170
What is the guideline for questionnaires?
1. Must be short
2. Must be simple
3. Not assume ‘they know’
171
What is the guideline for answers?
1. Exhaustive answers
2. Mutually exclusive
3. Include don’t know, or don’t want to answer
172
What two elements should be avoided in questionnaires?
1. Double negatives
2. Jargon and unfamiliar abbreviations
173
What variables are important when thinking about questions and answers?
Education level, age, and nationality
174
How do aspects of quantitative and qualitative approaches to program evaluation differ?
Quantitative:
- Standardised data collection
- Random sampling
- Large samples
- High reproducibility
Qualitative:
- Flexible data collection
- Ad hoc sampling
- Small samples for in-depth analysis
- Low reproducibility
175
Define reproducibility.
When researchers arrive at the same results using their own data and methods.
176
Define replicability.
When separate researchers arrive at the same results using the original researchers’ material.
177
What are the three functions of program evaluation?
Function 1: Monitoring evaluation delivery
Function 2: Understanding the limitations of RCTs
Function 3: Intervention design (legal and economic issues)
178
Who should be interviewed and sampled?
1. Participants: including non-compliers and leavers
2. Secondary subjects: including policymakers, implementers, and external observers
179
What are the three most common methods of data collection?
1. Individual interviews
2. Direct observation
3. Focus groups
180
How is the global quality of an evaluation study measured?
1. External validity
2. Internal validity
3. Measurement
4. Process evaluation
5. Acknowledgment of limitations
181
What are the four threats to external validity?
1. Targeting of a highly selected population. Solution: multi-site RCTs.
2. Sampling: non-randomised and small N. Solutions: List of the population, Sampling method, Ex ante power calculations.
3. Enrolment: refusal to participate, attrition. Solutions: Emphasise benefits and importance of evaluation, Minimise hidden costs of participation (time), Get support from trusted stakeholders, Use delayed treatment (waiting list).
4. Data collection: refusal to respond, missing, longitudinal attrition. Solutions: Proper method, Short and simple questionnaires, Pretest, Response incentives (money, small gifts), Interviewers’ quality and training.
182
What are longitudinal studies?
A correlational type of research in which researchers observe and collect data on a number of variables without trying to influence those variables.
183
What are the three threats to internal validity?
1. Different participation, response and attrition rates. Solutions: Delayed treatment, Different incentives, Initial agreement.
2. Spillover-effects between treatment and control group. Solutions: Delayed treatment (may cause loss of long-term outcomes), Proper randomisation design, Survey controls to combat contamination, Reduce circulation of materials and the content of the intervention.
3. Treatment replacement. Solution: delayed treatment.
184
To whom and how is data disseminated after the study has been completed?
- Policymakers: report with executive summary; workshops and meetings.
- Practitioners: websites, podcasts, social media, blogs; kit of materials with instructions.
- Public / Participants: websites, podcasts, social media, blogs; short reports.
- Researchers: research protocols; scientific journals, books and conferences; data repositories and meta-analyses.
185
What are the three ideals of data dissemination?
1. Independence
2. Simplicity
3. Replicability
