1
Q
What is a special case of cross - sectional design?
A
Case control studies. They are observational cross - sectional design. It is a comparison of “ affected “ cases to controls.
2
Q
What are the limitations is of the case control studies?
A
- The fact that they are retrospective: exposure in the past poses possible recall problems
- “Reversed causation”: e.g. cases may change crucial behaviors or memories as response to disease. (may overstate or downstate …)
- P (XlY) does not equal P(YIX) we want to know the chance of cancer given smoking (not the other way around)
- Comparability of the cases and controls
- Selection bias: cases/ controls/etc.
3
Q
What is the purpose of an experiment?
A
- The purpose of an experiment is to investigate a causal relationship between two (or more) variables. This is done by means of removing influence of other variables so that the effect of the intervention can be clearly seen. Themost distinguishing aspect of the experimental design is that we don’t only measure but we also manipulate. By intervening and actually manipulating the X - construct ourselves, we are in principle, better able to tackle confounding and make causal inferences about the effect of X.
4
Q
What are the distinguishing characteristics of the experimental design?
A
- Manipulation of the independent variable by the researcher.
- Allocation of units to treatments?
- Control over conditious.
5
Q
What are the conditions necessary for an experiment?
A
- The researcher is able to manipulate the independent variable x (the intervention).
- Ethical challenges are resolvable (regarding possible physical and for psychological harm) to conducting the experiment.
- Practical possibilities, funding, etc.
6
Q
What are the required characteristics of an intervention?
A
- The manipulation should:
- only wiggle X, not other stuff.
- should wiggle the entirety of X, not a small part of X
- should not run into plausible non - compliance.
- should be specified in detail.
7
Q
What does a basic experimental design scheme look like?
A
- It uses a pre -test as a point of reference to check whether there has been a change in Y over time.
Table ->
8
Q
What could be a problem for basic experimental design?
A
- You cannot be sure that the value at T2 is caused by the treatment.
9
Q
What does a control group experimental design scheme look like?
A
- A control group is added to filter out the influences of external events /extraneous variables.
10
Q
What could be a problem for control group experimental design?
A
- Cannot be sure that group differences in Y at T2 are caused by the treatment.
11
Q
What does a randomized control group design in experimental design look like?
A
-The idea is to filter out the influence of external events (or e.g. extraneous variables) through having a control group AND to filter out the influence of differences between groups caused by chance by creating comparable groups through randomisation.
12
Q
What could be done in order to create equivalent comparable groups?
A
- Randomisation, matching and homogenisation/restriction
13
Q
What is randomisation in experimental design?
A
- Research units are assigned to groups by means of a lottery system. So every unit has an independent chance of becoming part of either the experimental or control group).
14
Q
What is matching?
A
- In matching pairs of subjects are formed a on the basis of anticipated (relevant) similarity on one or morevariables. Pairs are split between groups. So equivalence is created based on known and measured characteristics.
15
Q
What is homogenisation/ restriction?
A
- It restricts sample to a certain value (range) of a variable (it is often bad for external validity)
16
Q
When to use randomisation and matching?
A
- Randomization when you have a large sample.
- Matching when you have:
- a small sample
- you know all confounding variables
- no choice (e.g. no budget)
17
Q
What are the possible problems with randomised control group design in experimental design?
A
- Cannot be sure that group mean Y at T2 are caused only by the treatment. Internal validity may be affected:
- Extraneous (not measured) variables: i.e external events occurring between pre and post test (history);
- When dealing people and animals: maturation effects.
- Pre - test effect (conditioning): i.e. existing situation is changed by taking pre-test/ learning effect (information learned in pre - test)
- Complex interactions like:
- Pre - test * intervention
- (Pre - test) intervention * extraneous variable ( not dealt with in the Solomon design)
18
Q
What are the interaction effects?
A
- It is when the effect of the combination of two variables (or conditions) is different than simply the sum of separate effects. An example of this is when pre -test and treatment each have their own effects, but exposure to both renders more effect, than only adding up the separate effects.
19
Q
What is the Solomon 4 control group design?
A
- The idea is to filter out the influence of the pre-test effects (and the interaction effects), two groups are added to the randomized control group design. The underlying idea for dependent observed variable Y:
- 02 = O1 + change in (x) + change in (P) + change in (E) + change in (X * P)
- 04 = O3 (= O1) + change in (P) + change in (E)
- 05 = O1 + change in (x) + change in (E)
- O6 = O11 + change in (E)
- We’re interested in delta X, the effect of the treatment. In the first group, all possible biases are present and also the effect of the treatment. Extraneous effects (external events, maturation), effect of the pre-test, interaction effect between the treatment and the pre-test. In the second group, there’s no treatment. So only the pre-test effect and the extraneous affects are present. In the third group, we have the treatment effect that we’re interested in and the extraneous effect. If we have good randomisation, we can assume the observation at t1 would have been the same as in group 1 or 2. Then we can use group 4 – again assuming we have good randomisation – to determine the extraneous effect. And group 3 (and 4) for the treatment effect.
20
Q
What are the issues with the Solomon four group design?
A
- It is costly and time consuming. Also it does not deal with the interaction effect of the intervention and extraneous variable.
21
Q
How to decide on whether to do a pre-test or not?
A
- Depends on how confident we can be that our randomisation resulted in comparable groups and on whether we expect pre - test effects and whether pre - testing is practically and ethically preferable or not.
22
Q
What is an experimenter effect?
A
- When researcher affects the experimental groups differently from the control group, because he/ she thinks there is an effect of the intervention.
23
Q
What is contagion/ contamination/interference/ diffusion/ spillover/ etc. effects?
A
- It is when the control and the experimental group have contact, they exchange information on the experiment and thereby influence the results. Applicable not just for humans or animals, but it can also be soil il plots (if I choose two adjacent plots to test a fertiliser treatment, the control plot may be affected by the experimental plot). To avoid that just make sure to have buffer space between the experimental soil plots and the control soil plats.)
24
Q
What is differential compliance in experimental design?
A
- Differential compliance is an issue especially when working with humans. I may have made sure to break the link between educational level and getting a treatment (take a pill), but this source of confounding may reassert itself afterwards if educational level affects rate of compliance (e.g. higher educated more stubborn in following up orders to follow their prescribed treatment)
25
What is differential dropout in experimental design?
- Dropout always affects external validity, but in the experimental design, it’s also relevant for internal validity. You make inferences about experimental effects by comparing post-test means of groups, assuming you can compare them. Perhaps you could compare them at the start of the experiment, but if dropout was differential, e.g. healthier people were more likely to drop out before the post-test in the experimental group than in the control group, you may come to wrong conclusion about the effect of X on Y for your units under study.
26
What are some other considerations for experimental design (Time lag, Scale effect, ATE vs. subgroup effects, ecological validity)?
- Time lag: how long after the manipulation do you measure the outcome? What if there are lagged effects?
- Scale effect: if job training has great effects on finding a job in a small-scale experiment and we implement free job trainings nation-wide as a result, the labor market will be flooded with people with job training, so it loses its distinctive value and will not have the same effect. Also think back to the example of the biological water turbidity treatment. Sometimes small-scale finds no effects, while it would likely be effective when scaled up. Sometimes, it’s the other way around.
- Average Treatment Effects (ATE) vs subgroup effects: on average something may not work, but perhaps that’s because it works badly for some and really well for others. How to identify relevant subgroup differences?
- What if ecological validity is a worry? Should you always go for a lab experiment anyway?
27
Why in experimental design you have to think about ecological validity?
- Because the lab is not the real world or because there is less control when you make the world your lab.
