Replication Crisis Flashcards
What is p hacking or data fishing?
Involves not having a clear hypothesis before data collection, running lots of analyses on the data to see what significant. Relevant for large data sets like ours. Hard to detect
What is HARKing?
Hypothesising after results known. Instead of writing hypothesis a priori (before running the analysis) you write it after you know the results to make sure it’s supported
What is Salami-slicing?
Instead of writing one publiction sharing all the results from the study, creating many papers looking at individual aspects of a dataset
Which of the below apply to the Replication Crisis?
A. Replications tend not to advance a researcher’s career much, and are harder to publish, so may not be attempted often
B. Journals tend not to publish non-significant findings. Leads to file-drawer effect - non-significant studies not being written up because publication is unlikely
C. Published significant results are more likely to be flukes, due to publication bias towards significant results (p < .05 means about 5% of significant findings will be a Type 1 error)
D. Original Studies may have been conducted with questionable research practices
All of them
Difference between replication and reproducibility?
Reproducibility is same data, same results.
Replication is getting same effect or results for new study, new samplle
What perecentage of studies couldl not be replicated?
one third
Why should we not draw conclusions from a single study?
1/3 can’t be replicated.
What does replicability depend on?
Assumptions such as
Normality - parametric vs non parametric methods
Choosing the correct mdoel
Issue with getting assumption of normal distribution wrong?
Inflating false positive rate
Why is it not always appropraite to fit a regression line?
Because perhaps variables dont have a linear relationship and there is not good theoretical reason to do so.
SO, are you fitting the correct model for the data both statistically and theoretically
Replicability depends on:
Assumptions but also transparents methods
Can you repeat the tests in another lab
are justifable deicison and hypotheses made before the results are known
correcting for multiple tests
researcher degrees of freedom
Why is methodological transparency so important in research?
Because excluding things like tests used or how many conducted means that we can’t evaluate if they have corrected for false positives . Hard to know which conclusions to draw
Why is methodological transparency so important in research?
Because excluding things like tests used or how many conducted means that we can’t evaluate if they have corrected for false positives . Hard to know which conclusions to draw
What is researcher degrees of freedom?
the decisions that we all make about how to analyse our data or what tests to run which might be different from researcher to researcher or lab to lab, because there are more than one right way to do things.
How can researchers fix the fact that linear or non linear methods could be valid for different reasons/garden path argument?
Be transparent about methods and pre register hypothesis/analysis plan before access to data.
