Designing and Analyzing Experiments Flashcards
T/F: The least powerful way to identify cause and effect relationships is conducting controlled experiments.
False: its the most powerful
What happens to results when the compliance of an experiment is imperfect?
The result becomes the average intent-to-treat effect (AITTE).
Define the three controlled experiment types.
1) field exeriments: aim to be similar to the real world
2) lab experiments: carried out artificially
3) A/B tests: aim to evaluate different versions of the same product
Define controlled assignment and what does it do?
When you select the value of x to the observations. This is to eliminate self-selction and eliminate reverse causality by not letting the outcome y affect x in any way.
T/F: The average intent-to-treat effect will be a little bit smaller than the average treatment effect if the degree of noncompliance was small.
True
T/F: Compliance is random
False: its rarely random.
Define the Hawthorne effect.
Both treated and nontreated individuals change behavior due to being observed
Define the placebo effect
Treated individuals change behavior due to be in treated (and not the treatment effect)
Define the John Henry effect.
Non-treated individuals change behavior due to being in comparison group
What are the 3 ways to randomize observations?
- Benchmark - simple random draw and treat
- Cluster - clusters of units selected
- Stratified - create strata and then randomize within strata
What are the two decisions that are relevant for all experiments?
- Number of subjects
- Proportion of subjects assigned to treatment
What are covariates?
Variables that effect the dependent variable, but are not of interest in the study. Need to check to see if these are balanced in experiments.
What does it mean when we say our data is balanced?
This means the distribution of covariates in the treatment and controlled groups are similar. If they are unbalanced, one or more of the covaraites will occur either more or less often in the treatment or control group.
When do we say that compliance is perfect? Imperfect?
Compliance is perfect if everyone in the treatment and control group stays in the group they were assigned to throughout the experiment. It becomes imperfect once someone changes groups in the middle of the experiment.
What is a spillover?
A spillover is when an event causes a change in one group, but can indirectly cause a change in another group as well, either positively or negatively
