Topic 6 - Impacts Evaluation of Development Policies and Programs Flashcards
What is Impact Evaluation?
To determine if and to what extent the change in something is due to intervention of a policy and if the change from that policy was cost effective.
Trying to establish causality
What is the issue in establishing causality between intervention (X) and output (Y)?
The diff types of causalities:
1.CAUSALITY: 𝑋 → 𝑌
2.REVERSE CAUSALITY: 𝑌 → X
3. Simultaneous change 𝑋 → 𝑌 and 𝑌 → X
4. SPURIOUS CORRELATION/OMITTED VARIABLE BIAS:
Observed X/Y correlation due to some common Z.
In finding a method what is crucial?
Random selection crucial
Intervention/ Program (P)
What is the ideal impact of an intervention?
(𝛿i) = Yi (1) - Yi (0)
Change in outcome in future greater than outcome without intervention
What are some commonly used methods for impact evaluation?
- Experimental design with randomization (Randomised Control Trial (RCT) approach)
- Matching methods
- Difference-in-Differences techniques (Diff-in-diff)
- Roll-out approach and Panel analysis
- Regression discontinuity design (RDD)
- Event analysis approach
- Instrumental variables approach (IV)
What is the RCTs (Randomised Control Tests)
->Randomisation in constructing TREATMENT and CONTROL groups from an eligible population
Two random groups one placebo and one with medicine for instance
How do you measure the impact of a RCT?
Avg treatment effect:
ATE = Y bar t - Y bar c
Y bar t = Avg outcome in Treatment group
Y bar c = Avg outcome in contrl group
What are the potential behavioural problem with randomised experiments?
- Hawthorne effect - modify their behavior simply because they are being observed
- Contamination and Spill-over effects
- John Henry effect:
When control group behave different to make up for the disadv of being the control group
- Dropout / Attrition effects - those not who dropped out
- Pioneer / Partial equilibrium effect
What are type 1 and type 2 errors in research (RCT)
Type I “false positives” – When the tester validates a statistically significant difference even though there isn’t one
Type II - false negative, also known as an error of omission
In what situation may RCT not be applicable?
When the event has already occurred
What is the Propensity Scoring Matching (PSM)?
Select Control (non-participants) group (C) with many observable essential characteristics as Treated group (T)
What is the process of the PSM?
- For each participant (i )
Find non-participants (mi ) with a very close propensity score (prob of participation) based on several variables (X)
- Impact of program is average difference in outcomes between treated i and his/her matched control.
= Treatment on the Treated (ToT) = 𝑇oT = 1/𝑁𝑇∑𝑖∈𝑇( 𝑌𝑖 − 𝑌𝑚(𝑖))
What are the assumptions the Difference-in-difference (DiD) approach?
Make crit assumption about occurence in absense of program
What are the stages the DiD approach take to calculate the approach?
- Change in outcome variable (Y) among participants over some period
∆𝑌𝑝 = Program Impact + Potential Role of other factors
- Compute change in same outcome among non-participants over same period
∆𝑌𝑛𝑃 = Potential Role of other factors
∆𝑌𝑝− ∆𝑌𝑛p = Programme Impact
What is the graph in the DiD display?
How the diff stages work
SEE GRAPH IN NOTES