Chapter 14: Replication & Transparency Flashcards
Replicable/Reproducible
Getting the same results when conducting a study again
Replication is part of interrogating statistical validity
Findings should be replicable if they are to be considered of high quality
Journals often don’t want to publish replication - but this is decreasing now
Researchers may not want to do replication studies (want to do something new to advance career)
Goals of Replication
Identify type 1 errors (false positives)
Generalize to new populations
Verify underlying hypothesis
Failed replication does not mean theory is not true
Large scale reproducibility projects
Troubling trio: be more skeptical about these studies, ensure they have been replicated (Low power (small sample sizes), High p value, Surprising results)
Direct Replication
Repeat an original study as closely as possible to see if the effect is the same in the newly collected data
Cannot replicate the initial study in every detail
E.g. conducted at different times of the year, with different participants, etc
Any threats to internal validity or construct validity flaws carry over to direct replication studies
Conceptual Replication
Explore the same research question but use different procedures
Conceptual variables stay the same
Procedures for operationalizing conceptual variables change
Replication Plus Extension
Replicate original experiment and add variables to test additional questions
Scientific Literature
Series of related studies, conducted by various researchers, that have tested similar variables
E.g. literature on alcohol and aggression, or on note taking format
Review Article/Literature Review
Collect all the studies on a topic and consider them together
Summarize the literature in a narrative way
Describing what the studies typically show
Explaining how the body of evidence supports a theory
Meta-Analysis
Mathematical summary of scientific literature
Quantitative technique
Mathematically averaging the results of all studies that have tested the same variables to see what conclusion the whole body of evidence supports
Pros/Cons of Meta-Analysis
Contain data from articles in empirical journals which have been peer reviewed (ensure quality)
Only as good as the initial study quality
File drawer problem
Valuable because assess weight of the evidence
File Drawer Problem
Meta analysis might overestimate the true size of an effect because negligible effects, or even opposite effects, have not been included in collection process
In history, stronger relationships were more likely to be published than negligible effects
Researchers should contact colleagues and request both published and unpublished data
Questionable Research Practices
Underreporting null findings: leads to evidence appearing stronger than it is
HARKing
p-Hacking
Small samples: few chance values can influence the data (estimates become imprecise/less replicable)
HARKing
Hypothesizing after the results are known
Predictions that happen before data are collected are more convincing
Harking misleads readers about the strength of the evidence
p-Hacking
Exploratory practice of analyzing results
Removing outliers, compute scores in different ways, run different types of statistics
Not intentional, but bias can creep in
Misleading when others are not told about all the different ways the data were analyzed and only the strongest version is reported
Open Science
Practice of sharing one’s data and materials freshly so others can collaborate, use, and verify results
Open materials and data
Open Materials
All study materials are reported publicly
Combats underreporting null effects
