Policy Evaluation Flashcards
how do we measure the impact of a programme?
d = (Y | D=1) - (Y | D=0)
What is the funamental problem of causal interference?
We cannot observe D=1 and D=0 at the same time
Who wrote about the fundamental problem of causal interference and when?
Holland 1986
What is a counterfactual?
Outcome in the world where the intervention did not happen
What assumptions do we need for a good counterfactual?
1) Same average characteristics
2) Same reaction to treatment if both groups were exposed
3) Neither exposed to any other interventions
What is the Before- After comparison?
Using the before treatment as a counterfactual
What are the issues with the before after comparison?
The outcome for the individual may have changed even if they weren’t treated.
What is the key assumption for before after comparisons?
The outcome for the individual would have been the same as it was before the intervention
What is the treated vs non treated comparison?
Comparing those treated with those who weren’t treated by a programme
What is the key assumption for treated vs non treated?
That both trends are the same if they were in the same conditions
What are positive and negative selection biases
Positive selection bias - impact is overestimated
negative selection bias - Impact is underestimated
What is the idea of randomised selection
Using random assignment as an allocation rule
-provides a good counterfactual
How does randomised selection work?
- Random evaluation sample picked from population
- Random treatment group picked from the evaluation sample, comparison group is those left over
What can be said about the characteristics of the two groups in randomised selection?
Average observable and unobservable characteristics will be identical
-random sample in large population - normal distribution
What are the advantages and disadvantages of randomised selection?
Adv: estimated impact is the true impact of that particular population
Disadv: External validity, hard to generalise the result
What are some of the complications associated with random selection?
1) Programme may have already finished
2) Non-excludable policies eg smoking bans
3) Unfeasible to randomise, eg road tunnels
4) Ethical reasons - health reasons
What is regression discontinuity design?
- non-experimental impact evaluation method
- Creates counterfactual from exogenous eligibility rule of program participation
- many policies use index or threshold to decide who can enrol
- compares those nearest the threshold to get average characteristics similar
What are the two conditions/ assumptions for RDD?
1) Continuous eligibility index for ranking individuals
2) Exogenous clearly defined cut off point for participation
Who conducted the merit awards impact on future performance experiment?
Thistlewaite and Campbell
1960
What are the problems with RDD?
- The closer you get to the threshold the better BUT there’s little data available.
- Trade off between statistical power and selection bias (close to threshold No selection bias but little stat power)
- Limited external validity - only applies to those close to the cut off
What are the main advantages and disadvantages of RDD?
Adv)
- Close to threshold impact is very accurate
- Mimics random selection
Disadvantages)
- only a local impact
- Lack of data around cut off limits meaningful estimates
Wha is the difference-in-differences method?
- non experimental
- Explores changes in outcomes over time between treated and non treated.
- Cross between before/After and Treated vs Non-Treated
What is the key assumption for DID method?
- Parallel/common trends assumption
- trends of treated and comparison must be identical before the programme
How do we measure the DID value?
DID= [ Yt(post) - Yt(pre) ] - [ (Yc(post) - Yc(pre) ]
