open science Flashcards
1
Q
reproducibility
A
- the idea of taking a dataset (which another research may have collected) and running the same analysis as that researcher and getting the same results.
2
Q
replicability
A
- the idea of taking the methods (research designs, stimuli, etc) from a previously run study, re-running, and getting the same results
3
Q
a spectre is haunting psychology
A
- the spectre of failed replications
- several large-scale replication attempts have shown that many classic findings in the psychology literature can not be replicated.
- some estimates suggest that > 50% of findings aren’t replicable.
- this has prompted some to claim that psychology is in a state of crisis.
- replication crisis - the observation that independent researchers dont get the same findings when they re-run classic studies
4
Q
what is the cause of the crisis
A
- how statistics and statistical procedures are used and abused in psych
- bias in which studies get published and which do not
- the typical use of small sample sizes in psych
- lack of clearly defined theories in psychological science
- these causes probably aren’t independent but are likely to be interconnected and related to each other
5
Q
bias in publishing
A
- the vast majority of published papers in psych journals report findings that support the tested hypothesis
- but how is this possible?
1) maybe psych researchers as psychic and they always test hypotheses that turn out to be true
2) maybe the hypotheses they’re testing a trivial
3) maybe there is some sort of bias in publishing
4) or maybe they only report the results that support their hypothesis
6
Q
bias in publishing 2
A
- one source of bias in publishing of psych studies is that journal editors and peer review might not want to publish studies when they dont support the tested hypotheses
- this might especially be the case when new studies dont show support for a famous or influential theory
- editors/reviewers might be more likely to suspect there’s some kind of a problem with the new study
- researchers might also choose not to submit studies for publication if they dont support the tested hypothesis
7
Q
bias in research practices
A
- it is very easy for researchers to engage in certain practices that invalidate their results.
- these practices make it so that researchers are more likely to find results that support a tested theory even if that theory isnt true.
- some examples include:
1) Running a statistical test, looking at the result, collecting more data, re-running the statistical test…. rinse, repeat… until you find the desired result.
2) Collecting data under many different conditions and only reporting the conditions that produce the desired result.
8
Q
combating bias
A
- but if these are problems, then what is the solution?
- one solution that has been proposed is pre-registration
- the idea od pre-registration has been covered in popular media, e.g., its been written about in the guardian on several occasions
9
Q
pre-registration and combating bias
A
- pre-registration can get around publication bias by allowing editors and reviewers to judge whether a study is likely to produce reliable results before the results are known.
- pre-registration can also get around certain kinds of experimenter and statistical biases by making researchers specify their statistical and study methods in advance
10
Q
pre-registration and combating bias 2
A
- pre-registration means that before conducting a study, researchers plan their study in detail
1) specifying the theory they plan to test and all of their hypotheses: - this means they cant change their hypothesis to make it fit whatever their data happened to show
- they cant cherry-pick their data or engage in subtle procedures to make the data fit their hypotheses
2) by outlining their plans in detail, reviewers can judge: - whether the methods are scientifically rigorous
- whether the study is likely to produce clear results
- they have to do this all before seeing the results, which might otherwise bias their decision
- in a special form of pre-registration known as a registered report, a journal actually agrees to publish a study before the data are collected
- this is possible because the pre-registration plan gives enough detail for editors/reviewers to judge whether the study is scientifically sound
11
Q
pre-registration in action
A
- in 2003, a paper was published claiming to show that merely looking at numbers would cause a shift in attention to either the left or right side of space
- this finding was very influential with more than 700 subsequent studies citing this finding or building on it
- some published studies tried to replicate it, most showed successful replications and very few failed replications
12
Q
but is it true?
A
- if your spoke to people at scientific conferences then many researchers would tell you that they couldn’t successfully replicate the effect
- but this wasn’t reflected in the scientific literature where most published papers on the effect showed that it could be replicated and where scientists continued to cite the original finding believing it to be true
- but why?
- the original finding was published in an extremely prestigious journal and it quickly became influential
- this means it probably got accepted as something like an established fact
13
Q
overturning established findings
A
- once a finding is accepted as an established fact then journal editors and reviewers might be reluctant to publish studies that don’t support the original finding
- note that it can sometimes, but not always, be very reasonable to not believe the results of a new study…
- e.g., if i did a study that showed that gravity doesn’t exist, then what its more likely? that gravity doesn’t exist or that my study is wrong?
- but for other examples, it might be that the established theory is wrong
- its best to try and judge studies based on their models rather than being influenced bu what the results are
- registered reports make this possible
- registered report - submitting a detailed plan for your research to a journal before you have collected the data
14
Q
an example registered report
A
- In 2017 I put together a registered report that involved a replication attempt of the original 2003 attentional cuing finding and some additional experiments to attempt to understand the mechanism that produced the effect.
- I then approached a journal with this plan to see if they were willing to publish the study if I did it according to the plan.
- The plan was sent out for review to be checked and then the journal agreed that they would publish it if I did it according to the plan.
- I then gathered together 30+ psychological scientists from 17 universities around the world and we ran the experiment on over 1300 participants.
- What did we find?
- We found absolutely no evidence for the original finding.
- No evidence that the additional manipulations modulate the size of the effect.
- Now scientists can move on from this finding, but a lot of resources have already been wasted studying it.
- This finding is a unique case, there are likely many zombie findings in psychology.
15
Q
the reproducibility crisis
A
- we can say a study is reproducible if we can take a dataset and re-run the analysis described in that journal article and get the same numbers
- its difficult to test because researchers dont typically share their data
- but data sharing is becoming more common, which means we might be able to test it
