Reproducibility Crisis

Question 1

Brian Wansink

Answer 1

• experiments on eating behaviours
• abused statistical procedures to look like research was successful
• p-hacking and HARKing

Question 2

What is ‘Crisis of Reproducibility’?

Answer 2

published research can’t be replicated/reproduced

• misuse of statistics
• not just about statistics

Question 3

Why researchers use statistics?

Answer 3

• to find relationships between variables if they think they are linked
• ignore noise and try find relationships that hold true ‘in general’

Question 4

What hypothesis do researchers test for?

Answer 4

null hypothesis / statistical tests that estimate how well the data supports the null hypothesis
- rarely test if a relationship exists but test to see if no relationship exists

Question 5

what is the null hypothesis?

Answer 5

hypothesis that no relationship (statistical significance) exists

Question 6

What is the p-value?

Answer 6

• probability that null hypothesis is true (probability that results are due to chance)
• lower p-value –> less likely null hypothesis is true; more reasonable to reject null hypothesis
• higher p-value –> more likely null hypothesis is true (no relationship exists); accept null hypothesis

Question 7

What would be the null hypothesis for study testing to see if there’s an extra $1000 annual income per year of schooling?

Answer 7

• there is no relationship between years of schooling and income

Question 8

What does a p-value of 0.55 mean?

Answer 8

• 55% chance that null hypothesis is true / 55% chance that there’s no relationship between years of school and income
• expect a statistical test this extreme 55% of the time

Question 9

What does a p-value of 0.01 mean?

Answer 9

• 1% chance that null hypothesis is true
• very unlikely that there’s no relationship between 2 values
• very unlikely to get p-value this extreme

Question 10

Most common cut off for statistical significance

Answer 10

p < 0.05

- researchers only incorrectly reject null hypothesis 5% of the time (1 in 20)

Question 11

What is the issue with p-value cut off being <0.05?

Answer 11

• p<0.05 so 1 in 20 (5%) chance that null hypothesis is true
• so for every 20 tests run, you will incorrectly reject null hypothesis 1 time (null hypothesis is true in in 20 times)

Question 12

What is p-hacking?

Answer 12

• Repeating a statistical test to get false positives / false statistically significant results

Question 13

What is HARKing?

Answer 13

Hypothesis after results are known

- can’t collect data then frame hypothesis around data

Question 14

What is the risk of a false positive with the accepted p value for a statistically significant result

Answer 14

• 1 in 20 chance of false positive (rejecting null hypothesis when it is true and there’s actually no relationship)

Question 15

Examples of why research can be wrong

Answer 15

• small sample size
• publishing studies with small effects
• relying on a small number of studies
• generating new hypothesis to fit data
• flexibility in research design
• intellectual bias
• conflict of interest
• competition to produce positive results

Question 16

Is flexibility in research design good?

Answer 16

NO - shouldn’t change research design to fit data

Question 17

Why does so much research fail to replicate?

Answer 17

• bad method
• researchers not constrained
• culture in research to get new and exciting results

Question 18

Example of how researchers edit their data

Answer 18

• exclude outliers from analysis
• p-hacking
• HARKing
• stopping collecting data when they achieved their desired results
• looking for effect in subgroups instead of whole populations

Question 19

How do researchers get away with this sloppy science

Answer 19

• they have freedom in research
• ## they justify altering methods midway as flexibility

Question 20

Can researchers justify altering research design before it’s complete

Answer 20

not good but sometimes there’s good reasons, e.g. :

• study causes harm
• trial working (can’t deny control group)

Question 21

Why is it bad that researchers don’t share their data/methods

Answer 21

can’t be checked and critiqued

- privacy restrictions / archive fails

Question 22

What effect does researchers wanting positive results have on fellow research?

Answer 22

publication bais towards pos results:

• studies aren’t replicated
• neg results turned into pos results
• neg results aren’t publishes

Question 23

What can be done to improve statistical use in research?

Answer 23

• better training
• p < 0.01 (null hypothesis incorrectly rejected 1 in 100 times)
• confidence intervals instead of significance testing
• make raw data available