Special Topic: The Replication Crisis

Question 1

The Replication Crisis

Answer 1

● What is it?
● Why do we care?
● How bad is it?
● When you read papers, how do you know you can trust the findings?

Question 2

Does learning about the replication crisis help?

Answer 2

● Undergraduate students who learned about replication for 1 hour…

○ Showed high endorsement that media attention was not a accurate indication of study reliability

○ Showed high agreement with suggestions about transparency and replication (e.g., publishing null
findings, making data open)

○ Understood the problems with flexible decision making in statistics

○ Showed slightly less trust in psychological findings

○ Showed greater appreciation for study design

○ Showed no decline in the desire to pursue graduate school

Question 3

salmon experiment?

Answer 3

• a salmon was put in a fmri scan
• their was activated in the areas in the human brain when thinking about various social situations
• but the salmon was dead
• because - They did multiple tests with the guarantee that a % of them will be significant

False positive rate: % of results that will show as significant, tied to alpha

Question 4

we see big effect sizes without replication to see if they indeed can be infared

Question 5

The replication crisis:

Answer 5

Many psychology studies done in the past do not reproduce by modern standards.

• Effects nonexistent or in opposite direction
• Previously significant results no longer significant
  -they might be crerry picked or luck
• Some effects exist, many do not (before: many big effects
  existed)
• But why?

Question 6

p hacking

Answer 6

• re-making the reserch until it fit into what the resercher wants
• get a lot of participants but at soon as it gets what they want they stop with the study

Question 7

File drawer problem

Answer 7

• Doing lots of studies, knowing that alpha % of our studies (e.g. 5% for
  alpha = .05) will show a false positive
• Then, discarding the studies that didn’t support our hypothesis and
  did not produce significant results
• All that gets published is the study that happened to have significant results!

The rest remain in the file drawer

Question 8

Significance incentives

Answer 8

• Once upon a time (and still today), journals only wanted to publish
  significant findings

* Incentive to get published (and get a better chance at a job/tenure):
Publish significant findings

• But most findings in psychology are not significant

Question 9

Why is this a problem?

Answer 9

• The studies we cite or build upon may show effects that are not
  reliable or that don’t really exist
• We lose our sense of the true state of cognitive processes
• We are incentivized to discard studies that don’t show significant
  results, leaving a lot of great science in the file drawer

Question 10

What can we do about the replication crisis?

Answer 10

1. Move beyond the p-value
2. Be upfront about what you plan to do
3. Be transparent in your work

Question 11

1. Move beyond the p-value

Answer 11

• your p-value should not be dancing around
• Effects in psychology are not single points; they are distributions

using distribution over p-values

• there should have a trashold

Question 12

Using the distribution: Instead of looking at p-values..

Answer 12

• Compare confidence intervals
• Compare the distributions themselves, (how they overlap)

Question 13

2. Be upfront about what you plan to do

its very easy

Answer 13

Pre-registration helps keep us accountable

• By reporting all studies and analyses you plan to conduct, insignificant
  results can’t be put in the file drawer
• Incentive: some journals will promise to publish your results
  regardless of their significance if you pre-register your analyses with
  them

Question 14

3. Be transparent in your work

Answer 14

• We want to replicate and see if our work can be replicated
• Replication only works when we know the conditions under which the original study took place!

Question 15

Letting our studies replicated can feel scary…

Answer 15

• Perspective: it is better, more honest science to let our work be
  critically evaluated and tested by others
• We build on each others’ work –

Question 16

Be transparent in your work

Answer 16

How can we know what others did (and others, what we did?)

• Share your data
• Share your methods
• Be open to critiques/corrections if your work wasn’t replicated by
  others

Question 17

Data sharing

Answer 17

t’s easy to share your code, data, materials, etc. on a site like OSF

• Others can look at the raw data, try and replicate your analyses, and much more

Question 18

What should I take away from this?

Answer 18

• Many past studies in psychology cannot be replicated because:
  -They were not fully transparent in their procedures and findings
  -They just “happened” to be the one significant study out of many
• True effect sizes in psychology are smaller than we’ve been led to
  believe
  -There are fewer significant effects
• We can address these problems by:
  -Incorporating all of your data into the analysis
  -Being accountable to your research plans
  -Being transparent with procedures and findings