The Replication Crisis and Meta-Analysis Flashcards
reasons for the replication crisis
Questionable research practices
Tendency towards publishing novel findings
File drawer effect
Confirmatory hypothesis testing
P-value rounding
Optional stopping of data collection
Conflicts of interest
Vague publication of methods
overcoming the replication crisis
Replication studies that with larger sample sizes
Pre-registration and peer review before data collection
Online repositories for data
Reduce p-value threshold
Report effect sizes and Cis
Replicate novel findings before publication
Meta-analysis
what is meta analysis
Statistical technique used to combine findings from individual studies with the same, or very similar, research question
Often carried out alongside a systematic review
Often used to assess the clinical effectiveness of healthcare interventions
meta analysis and effect size
Meta-analysis commonly uses Cohen’s d of the correlation coefficient
A weighted avg is calculated to provide a precise estimate of the overall treatment effect
when is meta analysis appropriate?
Applicable to collections of research that:
○ Are empirical
○ Quantitative results
○ Examine the same constructs and relationships
○ Have findings that can be configured in a comparable statistical form
○ Care comparable in general
advantages of meta-analysis
○ Can handle many studies
○ Improves estimates of the size of the effect (increases sample size and power)
○ Results can be generalised to a larger population
○ Settles inconsistencies in findings
disadvantages of meta-analysis
- Cant resolve the file drawer problem
- Relies heavily on researcher competence
- Agender-driven bias
- Time and effort
steps in meta-analysis
- Define variables of interest
- Plan database search
- Obtain research reports/studies
- Critically appraise studies for inclusion
- Calculate effect sizes for each study
potential sources for identification of documents for meta-analysis
○ Computer databases
○ Authors working in the research domain
○ Conference programs
○ Trial register
○ Review articles
○ Hand searching relevant journals
selecting studies for meta analysis
- Formulate your research question
- Conduct literature search
- Select studies to include based on relevance and quality criteria
- All studies you wish to include must report means and standard deviation scores for the outcome measure (both experimental and control)
meta analysis models
fixed effect model
random effects model
fixed effect model
○ Assumes homogeneity (every study estimates the same population mean)
○ Unrealistic
random effects model
○ Assumes that effects may vary from study to study
○ Routine choice
standardised mean differences (SMD)
an effect size measure used when studies have measured the same outcome using different scales/measures
SMD of 0
indicates that both conditions have equivalent effects (e.g., control and intervention/treatment)
