Week 1- P Hacking and Harking

Question 1

What is the main critical thing determining if a manuscript gets published or not?

Answer 1

The results (although we should have no control over it)

Question 2

What is best for science?

Answer 2

Publishing rigorous research regardless if the results support the hypothesis or not (null results are still important!)

Question 3

What is best for scientists?

Answer 3

Publishing lots of results (care for quantity>quality)

Question 4

How is Koert van Ittersum and Brian Wansink’s research on food portions and extroversion lacking in credibility?

Answer 4

-No error bars
-Small sample size (unrepresentative)
-6 to 12-year-olds will not be alike (unrepresentative)

Question 5

What is a median-split?

Answer 5

Determining the median where below=low and above=high and groups are delegated based off that (mathematically makes no sense as numbers next to each other not alike BUT numbers in same group are even if far apart)

Question 6

Give 3 examples of poor research

Answer 6

1.Wunsink kept analysing food data until significance.

2.Prof Diederik Stape made own data on spreadsheet and had 58 papers retracted.

3.Amy Cuddy claimed power posing (certain poses) would increase brain chemicals and hormones to increase confidence (which just isn’t true)

Question 7

True or false: The better/prestigious the journal means the less likely someone will replicate the results BUT more likely to retract the paper

Answer 7

True (prestigious science journals struggle to attain average reliability)

Question 8

What is the file drawer problem?

Answer 8

Researchers will conduct pre-defined analysis and publish successful findings but file drawer “unsuccessful ones” aka null findings. (usually when p-hacking fails)

Question 9

What did John et al. (2012) find when determining the number of researchers involved in questionable research practises? (QRP)

Answer 9

More worse QRP means fewer self-admission rate e.g., falsifying data

Question 10

What does it indicate if lots of observations are just under the 0.05 significance?

Answer 10

Manipulation of data must have occurred somehow because it mathematically just be linear. (as probability is not a false concept)

Question 11

Define P-Hacking

Answer 11

The method of manipulating data to achieve significant results

Question 12

Give 6 P-Hacking methods

Answer 12

1.Multiple analyses

2.Omitting information (removing certain variables from the analysis)

3.Controlling for variables

4.Analyse part way through then collect more data and repeat until significance is reached

5.Changing the DV

6.Removing outliers (although sometimes they can drive significant results)

Question 13

True or false: Nonparametric correlations (non-normal distribution) create more sensible results with outliers

Answer 13

True

Question 14

Why can multiple analyses create problems?

Answer 14

Different methods can lead to different conclusions despite testing for the same thing. (worse with more complex analyses)

Question 15

What was the PACE trial?

Answer 15

-Compared adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy with specialist medical care for chronic fatigue syndrome (hard to treat) (PACE): a randomised trial (White et al., 2011) suggesting it would exceed SMC

-Cost 5million

-Still used to inform treatment in the UK

Question 16

What did White et al. (2011) CLAIM they found?

Answer 16

Adding CBT and GET onto SMC significantly improves outcomes

Question 17

What did White et al. (2011) originally design?

Answer 17

-Originally the trial was designed to explore objective results (6min walking test and Aerobic fitness)

-These did not improve (published 4 years after the main paper)

-Other objective measures e.g. returning to work, did not show improvements (and were delayed in publication)

-Objectively it was ACTUALLY ineffective as a treatment

Question 18

How did White et al. (2011) P-Hack the data?

Answer 18

-The PACE trial’s published protocol defined ‘recovery’ as requiring an SF-36 Physical Functioning (SF36-PF) questionnaire score of at least 85 out of 100 originally

-The trial’s entry criteria THEN required a score of 65 or under-recovery, which was taken to indicate that patients’ fatigue was disabling[2].

-However, the cut-off for an acceptable level of physical functioning was changed to 60

-Literally, a participant could have been recruited with an SF-36 score of 65, finished the trial with a score of 60 (i.e. GOT WORSE) yet be described as a positive outcome

Question 19

True or false: Mentions of margins or on the edge of significance is a red flag.

Answer 19

True

Question 20

Why is analysing part way through and then collecting more data to stop when significance has reached a problem?

Answer 20

Because it massively increases false positive rates (more checkups = more likely around a 25% rate)

Question 21

How is Favardi and Cox’s usage of attentional bias to alcohol in order to reduce consumption done wrong?

Answer 21

-A 20-minute lab study won’t fix the strong conditioning
-Hawthorne effect=the awareness of being observed results in people writing fewer measurements of alcohol units down down

Question 22

Why do control groups need to be equivalent?

Answer 22

-No removal of active ingredients e.g., doesn’t give a point of comparison or allowance to make a statement about the experimental condition

-For example, waiting list controls (e.g., waiting for nicotine) are NOT a control condition (just use a non-active patch to make it a CC)

Question 23

What’s the false positive rate of p<.05?

Answer 23

30% (Colquhoun, 2014)

Question 24

How can we redefine significance?

Answer 24

-Benjamin et al (2017) argue that new effects should be considered sig only if p<.005.

-This corresponds to a Bayes factor of 14-26 which represents ‘substantial’ to ‘strong’ evidence according to Bayes classifications.

-As a result, it reduces the false-positive rate to ‘reasonable levels’ (5%).

Question 25

According to Lakens et al (2017), why does lowering the p-value to .005 NOT improve replicability?

Answer 25

-P<.005 won’t reduce p-hacking (just make it harder)

-Increased sample size needed to be powered (favours established research establishments)

-Would reduce number of replication studies

-Cost more money (bias in favour of established scientists?)

-May increase false negatives

-Instead argues we should define alpha levels when designing study

Question 26

True or false: Computerised tests have poor reliability

Answer 26

True up to the point that results should be discarded

Question 27

What’s Preregistration?

Answer 27

-Registering your intentions for analysis in advance (means you get caught out if you change it so avoid temptation)

-Justify sample sizes (ideally from power analyses)

Question 28

How do we preregister?

Answer 28

1.What is the main question being asked or hypothesis tested in this study

2.Describe the key dependent variable(s) specifying how they will be measured

3.How many conditions will participants be assigned to?

4.Specify exactly which analyses you will conduct

5.Any secondary analyses? (It’s the use of existing research data to find an answer to a question that was different from the original work)

6.How many observations will be collected or what will be the sample size

7.Anything else you would like to pre-register?

8.Has any data already been collected

Question 29

What is open data?

Answer 29

-Shared data not keeping it as your own

-Add it to the open science framework (preregistration too and means you won’t put purposely manipulated wrong data)

-Must be included in the ethics that you can share data

Question 30

What is Clarke’s third law (updated by Gelman)?

Answer 30

“Any sufficiently crappy research is indistinguishable from fraud”