Week 4

Question 1

Early philosophers

Answer 1

• First big ideas about mind and science

Question 2

Enlightenment

Answer 2

• Growing questions about mind, mechanism, empiricism

Question 3

Early psychologists

Answer 3

• How to be experimentalists
• Study perception, consciousness, intelligence

Question 4

Psychoanalysts

Answer 4

• The importance of the unconscious, inner conflict

Question 5

Behaviourists

Answer 5

• No more mind silliness
• Behaviour only

Question 6

Cognitive revolution

Answer 6

• The mind is back in psychology
• Study as information processing

Question 7

Paradigm shift

Answer 7

• Dominant schools of thought about how to study the mind scientifically have changed
• Zeitgeists
• Often periods of upheaval, revolution

Question 8

Reproducibility

Answer 8

• The extent to which consistent results are observed when scientific studies are repeated
• Major demarcation between science and pseudo-science
• Scientific claims should not gain credence by virtue of status/authority of their originator

Question 9

Science

Answer 9

• Systematic observation
• Ruthless peer review
• Considers all evidence
• Invites criticism
• Repeated results
• Limited claims
• Specific terms, operational definitions
• Engages community
• Changes with new evidence
• Follows evidence where it leads

Question 10

Pseudoscience

Answer 10

• Anecdotal evidence
• No peer review
• Considers only positive evidence
• Dismisses criticism
• Non-repeatable results
• Grandiose claims
• Vague terms and ideas – science-y jargon
• Isolated
• Dogmatic and unyielding
• Starts with a conclusion, works back to confirm

Question 11

How to collect data

Answer 11

• Generate a hypothesis
• Is it interesting
• Collect some data
• Maybe the first study doesn’t work so you fix it by changing some variables
• Repeat step 3 until you have enough studies to publish

Question 12

Stapel – 2010

Answer 12

• Prolific Dutch Social psychologist was investigated for fraud
• He often supplied the data to his grad students
• His grad students working in the lab remarked that stats for different studies showed similar means and SDs
• After investigation, admitting his fraud and found 25 published papers were based on fabricated data
• 58 papers were retracted

Question 13

Daryll Bem

Answer 13

• ESP study
• Claimed people had precognition
• Picking between pictures behind curtains
• The study had issues of reproduction

Question 14

Common bad research practices

Answer 14

• Stopping data collection once p is smaller than .05
• Analyse many measures but only report significant ones
• Collect and analyse many conditions but only report significant
• Using covariates to get significance
• Excluding participants
• Transforming data to get p<.05

Question 15

Open science collaboration

Answer 15

• 100 replications of pieces of research from 3 prominent journals
• Formed in 2011 with around 60 members
• Grew to 270 scientists from over 50 countries
• 97% of original studies reported significant effects
• 36% of replications had significant effects in the same direction

Question 16

Replication crisis factors – how did the crisis happen

Answer 16

• Enormous pressure and incentive to produce many papers
• Over-interest in counter-intuitive findings as sexy
• Confirmation biases by researchers promoting questionable research practices
• Lack of accountability and transparency

Question 17

Confirmation bias

Answer 17

• Tendency to seek out information that verifies your theory (validation)
• And not seek out information that falsifies your theory (falsification)

Question 18

Drawbacks of the replication crisis

Answer 18

• Failed replications may be the new trend
• It creates a culture of paranoia and moral righteousness

Question 19

Dan Gilbert – reaction to the crisis

Answer 19

• So-called replicators are shameless little bullies and second stringers

Question 20

Power pose replication crisis

Answer 20

• Carney, Cuddy and Yap 2010
• As evidence has come in over these past 2+ years
• My views have updated to reflect the evidence
• I do not believe that power pose effects are real

Question 21

Schnall replication crisis

Answer 21

• One of her major papers failed to be replicated
• Schnall wrote a response commenting on damage to her career
• Roberts commented that damage to her career was less important than the PhDs she ruined for being honest

Question 22

Increase replication

Answer 22

• We must ensure that findings are robust and replicable
• Direct/conceptual replications should be a part of research pipeline

Question 23

Beware P-hacking

Answer 23

• Exploring researcher degrees of freedom to find a significant effect on
• Changing degrees of freedom to find a significant effect
• Implicit bias or explicit data manipulation

Question 24

Definitions of power

Answer 24

• The probability of finding an effect in your study if the effect is real
• The probability that a test of significance will detect a possible deviation from the null hypothesis, should such a deviation exist
• The probability of avoiding a type 2 error

Question 25

Boost your power

Answer 25

- Studies can be underpowered - Due to misunderstanding of power - Large studies are more expensive and time-consuming - We need to publish more papers, more frequently

Question 26

How do you increase power

Answer 26

- Larger sample sizes

Question 27

Open data principles

Answer 27

- Making data, materials and analysis available online - So that others can replicate, check and reproduce your work

Question 28

Confirmatory/exploratory research

Answer 28

- Confirmatory – hypothesis testing - Exploratory – studying around the topic - Exploratory research is okay - But must not be presented as confirmatory - Should be followed by confirmatory

Question 29

HARK-ing

Answer 29

- Hypothesising after results are known

Question 30

How to conduct confirmatory research

Answer 30

- Decide study details a priori - Hypotheses to test, number of subjects, conditions, DVs - Only then start recruiting - Pre-register your study

Question 31

Open science practices in teaching

Answer 31

- Ensure the next generation moves on from the reproducibility crisis - Teach the importance of conducting well-powered studies - Encourage critical evaluation of published studies in terms of open science practices - Open science practice leads to more reliable, reproducible science

Question 32

Open science practices as reviewers

Answer 32

- Signatories will not offer comprehensive review for any manuscript that does not meet the minimum requirements - Neither will they recommend the publication - Stimuli and materials should be made publicly available - Data should be made publicly available - Documents containing details for interpreting any data files or analysis code should be made available - The location of all these files should be advertised in the manuscript and all files should be hosted by a reliable third party

Question 33

Incentives for pursuit of new ideas

Answer 33

- Publications, grant income, employment, promotion, tenure, fame

Question 34

Issues with rewarding science practices

Answer 34

- Poor motivators - May prompt bad practices for fame

Question 35

Goals for reward open science practices

Answer 35

- Be tolerant of lower output if doing correctly - Reward food practices for pre-registered studies regardless of outcome - Reward good practices for high powered studies - More time for research

Question 36

Solutions to the replicability crisis

Answer 36

- More replications - Beware of p-hacking - Boost your power - Open data, open materials, open analysis - Conduct pre-registered confirmatory studies - Incorporate open science practices in teaches - Insist on open science practice as reviewers - Reward open science practices