Chapter 10: Quasi-experiments, small-N designs, replication Flashcards
(28 cards)
Quasi-experiments
The independent variable is not manipulated, but rather observed or measured
There is no random assignment
Use a quasi-independent variable: independent variable is not manipulated
You can come up with a lot of confounding variables because the groups are non-equivalent
Nonequivalent control group
When groups differ with regards to the variable, but are also likely to differ on a lot of other aspects
There could be selection effects due to the lack of random assignment
Interrupted time-series design
Looking at the dependent variable across time → the independent variable is actually time
No control group or random assignment
Nonequivalent groups interrupted time-series design
Combines two ideas: looking at a pattern over time and adding a control group
History threats
An external event that happened at the same time as your independent variable and has an effect
Problematic, especially in interrupted time series design and especially when you don’t have a control group
Attrition threat
We see a change over time because some people dropped out of the study
If these dropout systematically differed from the other people in the study, this could be an alternative explanation for the effect
Can be ruled out when researchers delete the data of the dropouts
Instrumentation threats
We see a change of scores because we used different instruments
Four reasons to use quasi-experiments
- Real-world opportunities
- Realism
- Ethics (main reason)
- High construct validity
Difference between correlational studies and quasi-experiments
Similarities: both designs don’t have random assignment and there are no manipulated variables (all measured)
Differences: sample is intentionally chosen in quasi-experiment, goal of quasi-experiment is to make claims
Correlational: studies with participant variables are intended to document similarities and differences due to social identities, development or personality
Quasi: focus is less on individual differences and more on potential interventions such as laws, education or media exposure
Small-N experiment
Using very small samples in experiments
Also referred to as ‘case studies’, because they often look at one person
Used when we want to learn something about a certain relationship, so we conduct repeated experiments
Two advantages of small-N designs
- Control: it’s a very controlled setting
- Special cases: we can look at unique people
Two disadvantages of small-N designs
- Confounds are possible
- External validity
Three types of small-N designs
- Stable baseline design
- Multiple baseline design
- Reversal design
Stable baseline design
We start with multiple baseline measures: give a person a list of words and ask him to memorize as many as possible
At a certain point, we teach him how to use ‘expanded rehearsal’ and after this, we check again how many words he memorized
Result: number of memorized words increased after the new strategy → conclusion: method has a causal effect
! Maybe placebo effect, design confound or history threat
Multiple baseline design
Introduce a baseline period for each child in the study, during which they come in the lab and are asked to get as close to a dog as they feel comfortable to
Next, we introduce the treatment: they have to calm down and take another step, after which they get a reward (! important: treatment is introduced at different times for the different kids)
History threat is unlikely, maturation is possible but can’t explain the entire pattern
Reversal design
Start with baseline measures for both variables: we observe how frequently patients use the coping mechanism and measure the depressive symptoms
Second phase: we introduce treatment
Reverse the treatment to know if the reduction in symptoms is caused by coping behavior: person stops using the new coping behavior → if the depressive behavior increases again, then the treatment has a causal effect
We end the experiment by introducing the treatment again (ethical)
Validity of small-N experiments
Internal: can be very good if the study is designed well
External: can be problematic depending on the goals of the study
Construct: can be very good if the variables are well defined and observations are accurate
Statistical: not always relevant here because we look at individuals
Replication crisis
A lot of research can’t be replicated (social psychology is the hardest)
Research: 62% of studies were successfully replicated if they conclude a study as successful when they detect a significant effect in the same direction as the original study using the same statistical test
Problematic to some extent, bus also an opportunity to realize that science needs to be done in a different way
Open science practices
Making sure that science is done in a way that is transparant and open: making everything openly accessible and sharing as much as possible) → solution to replication crisis
Three types of replication
- Direct replication: doing the exact same thing as the original study, just in a different sample
- Conceptual replication: we test the same hypothesis in a different sample, but now we operationalize the variables in a different way
- Replication-plus-extension: we replicate a study, but we add something
Three reasons why studies are not replicable
- Effects are sensitive to contextual effects (fa. cultural context)
- Limited number of replication attempts
- Problem with original study (fa. a false positive)
Meta-analysis
A systematic review and empirical analysis of results of multiple studies
We try to summarize findings from different studies and come to an overarching general conclusion
Five steps of meta-analysis
- Formulate research question
- Search for ‘primary studies’
- Coding selected ‘primary studies’
- Meta-analyse the effect sizes of primary studies
- Interpret meta-analytical effect size
Two strengths of meta-analysis
- Summary of scientific evidence, a large amount of literature and primary studies
- High degree of credibility
