Lecture 11: Psychological science into the 21st century: The standard paradigm, the replication crisis, and Open Science

Question 1

Isn’t it remarkable that you come across a chapter in a psychology textbook entitled
“Is psychology a science?”

Points to…

Answer 1

• insecurity of psychologists with regard to their own scientific field
• healthy self-critical attitute (typical property of psychology, seen less in other areas)

Question 2

the image of the psychologist is determined by the media

Answer 2

there, psychologists rarely act in a scientific role. much more often you see psychologists that have never done research

Question 3

verschil tussen hoe kinderen scientists en psychologen zien, en een groot verschil tussen “perceived difference in knowledge between different kinds of scientists and the average person”

Answer 3

oke

Question 4

what also plays a role

Answer 4

criticism from within

Question 5

criticism from within =

Answer 5

• Psychoanalysis: Psychology is superficial
• Critical psychology: psychology wrongly treats man as a physical object
• Humanistic psychology: scientific psychology is cool and distant, when it should be concerned with people
• Feminist and postcolonial psychology: scientific psychology is a matter for Western white men

Question 6

so psychology has a hard time with itself and with the outside world… voorbeelden

Answer 6

• Psychologists regularly have to defend themselves
• Partly against stereotypes, partly against justified objections
• Bad experiences with Popper and Freud
• So for psychology, the demarcation problem is very acute…
• Maybe methodology can come to the rescue?

Question 7

methodolatry=

Answer 7

Worship of a method that employs it uncritically regardless of ever- changing particulars and steadfastly ignoring past negative results.

Question 8

wat is er met methodology en methodolatry

Answer 8

• Methodology plays an important role in psychology
• General principles like striving for clarity, focus on testing, the provisional nature of science
• Plus specific attention to issues such as uncertainty (statistics), induction, confounds
• Strong orientation towards empirical data + statistical analysis
• This is all good. But…
• A prevailing viewpoint in psychology is that science if characterizes solely by method
• “Whoever follows the scientific method is
  scientific”
• But make no mistake! You can also follow the scientific method very superficially…
• Are you scientific just because you follow the
  precepts?

Question 9

methodology kan daardoor dus ook …. worden

Answer 9

methodolatry

Question 10

wat is de analogie van Feynman

Answer 10

Cargo cult science, in WOII the Americans build bases on the Pacific islands, and a cariety of religious practices spontaneously arose from that (planes, chocolate etc)

-> he suggested that science can work the same way, it looks like science but it isnt.

Question 11

wat was feynman zijn boodschap

Answer 11

science is not just doing experiments and statistical analyses, it is an attitude

Question 12

quote van Feynman

Answer 12

[T]here is one feature I notice that is generally
missing in Cargo Cult Science. That is the idea
that we all hope you have learned in studying
science in school—we never explicitly say what
this is, but just hope that you catch on by all
the examples of scientific investigation. It is
interesting, therefore, to bring it out now and
speak of it explicitly. It’s a kind of scientific
integrity, a principle of scientific thought that
corresponds to a kind of utter honesty—a kind
of leaning over backwards.

The first principle is that you must not fool
yourself—and you are the easiest person to
fool.

Question 13

mertons core values

Answer 13

1. communalism
2. universalism
3. disinterestedness
4. organized skepticism

Question 14

communalism =

Answer 14

scientific products belong to no one

Question 15

universalism =

Answer 15

truth claims are judged the same, no matter who makes them (je hoeft niet perse een phd te hebben etc)

Question 16

disinterestedness=

Answer 16

scientists are not interested in the outcome of research

Question 17

organized skepticism=

Answer 17

ideas are cracked down on and rigorously tested, regardless of who proposes them

Question 18

hoe was de communalism tijdens 20th century

Answer 18

• articles were published behind a publisher paywall
• psychologists often regarded their data as property, did not share data or materials etc

Question 19

universalism hoe zit dat nu

Answer 19

• no explicit favouring groups in psychology, however clearly power structures (students, editorial positions, leadership of scientific org)
• also underrepresentation of women, minorities, scientists from developing countries
• WEIRD demographic
• shared value is not automatically a reality

Question 20

WEIRD demographic

Answer 20

westerd
educated
industrialized
rich
democratic

Question 21

disinterestedness hoe zit dat nu

Answer 21

• psychologists and other researchers are interested in success
• the temptation to make things better than they are can be considerable
• fraud and questionable research practices

Question 22

organized skepticism hoe zit dat nu

Answer 22

• scientific culture separates the person and the idea, therefore distrust is not personal (but sometimes understood as suspicion)
• however, much of psychology in 20th century appears to have been too gullible

Question 23

the scientific attitude and mertons norms are not actualized all the time, this does not make them useless! why?

Answer 23

they are regulative ideals: things we strive for. striving for an ideal can be useful, even if this ideal is strictly unattainable.

Question 24

science has room for.. not for …

Answer 24

skepticism

cynicism

Question 25

The standard of psychological research in the 20th century:

Answer 25

• Create a theory (usually a verbal story)
• Come up with an experiment (usually small n)
• Test hypothesis (usually with a significance test)
• Publish results (usually without full disclosure and without data)

Question 26

who introduced the notion of a scientific paradigm

Answer 26

Kuhn (1968)

Question 27

Vul’s Voodoo correlations

Answer 27

• The neuroscientist Edward Vul lamented the small studies in neuroscience
• Identified extremely high correlations between e.g. personality and fMRI results (r>.8)
• In fact these values were higher than should be possible given the reliability of the measures used
• Argued that these results could not possibly be truthful

Question 28

Bargh’s fantastic effects

Answer 28

• Some psychologists claimed fantastic effects
• Priming people with words like “old” and “elderly” leads people to walk more slowly?
• People who are lonely take hotter showers?
• These effects raised more than a few eyebrows
• The research results seemed too good to be true

Question 29

so who pioneered behavioural priming research

Answer 29

John Bargh

Question 30

4 types of questionable research practices

Answer 30

• p hacking
• publication bias
• HARKing
• Dropping experimental conditions that ‘dont work’

Question 31

p-hacking

Answer 31

deleting outliers, adding covariates, splitting the sample, selecting variables until you find a significant effect

Question 32

publication bias

Answer 32

only publishing positive results

Question 33

HARKing

Answer 33

Hypothesizing after the Results are Known

Question 34

Bem’s Feeling the Future

Answer 34

• Psychologist Daryll Bem purported to have shown that ESP exists
• Extraordinary claims that basically nobody believed
• But… Bem had hardly done anything out of the ordinary
• Most people drew the conclusion that this meant that something was wrong with our methods

Question 35

wat was er mis met de methodes

Answer 35

• samples sizes varied across studies without a stated reason
• different studies appear to have been lumped together or split apart
• its not clear which analyses were planned in advance
• one tailed tests in the absence of a directional prediction
• p values very close to 0.05
• how many other studies were run but not reported?

Question 36

the stapel case

Answer 36

• Social psychologist Diederik Stapel published heavily in leading journals
• Amazing effects that made their way into newspaper headlines
• Three brave Ph.D. students suspected something was wrong
• Data for 55 of his papers were entirely made up!

Diederik Stapel (1966).

Question 37

Important difference of fraud with QRPs:

Answer 37

Fraud is intentional, QRPs are not

Question 38

replication crisis

Answer 38

In 2013, a group of psychologists – the Open
Science Collaboration – decided to assess
replicability of 100 psychological studies
* The results were published in Science and suggested that many studies could not be replicated
* This marked the beginning of the replication crisis (now known as the crisis of confidence)

Question 39

pillars of open science (3)

Answer 39

• Open Data: So that anybody can assess the
  evidence
• Open Materials: So that anybody can replicate the study
• Preregistration: So that a posterori tinkering with analyses becomes visible

Question 40

relationship between mertonian norms and open science

Answer 40

• Wagenmakers et al. (2021) relate open
  science to Merton’s norms
• Argue that Openness and transparancy
  instantiate these norms
• Open science = science done right?

Question 41

Open Science institutionalized

Answer 41

• Center for Open Science
• Open Science Framework
• Psychological Accelerator
• GitHub code sharing
• Findable, Accessible, Interoperable, Reusable (FAIR) data
• Transparency and Openness Promotion (TOP) guidelines
• Diamond Open Access journals

Question 42

the Open Science Framework

Answer 42

• The OSF is a repository
• Allows researchers to make all of their materials available
• Now standard in psychology

Question 43

the Manylabs project

Answer 43

• In five Manylabs projects, different labs collaborated to replicate psychological phenomena
• Bad news: some of psychology’s most sexy findings don’t replicate
• Good news: many findings replicate well, be it often with smaller effect sizes
• The Manylabs setup has been taken up in other fields (ManyBabies, Many Birds, etc.)

Question 44

the Psychological Science Accelerator

Answer 44

• The Psychological Science Accelerator (PSA) aims to accelerate cumulative reliable knowledge
• Countering WEIRDness through diversity is a main aim
• PSA can rapidly employ huge distributed studies to evaluate research findings

Question 45

criticisms of open science

Answer 45

• Increased bureaucracy and more work for researchers
• Most data are never downloaded, many preregistrations aren’t adhered to, badges aren’t checked
• Too much focus on replication, not enough on creativity
• Open Science is expensive and can compound inequalities
• Open Science is not always feasible or desirable
• Some approaches may decline (e.g. field studies, qualitative research)

Question 46

• “How scientific is psychology?” remains an important question
• Clinging to methodological precepts risks degrading into methodolatry
• Scientific culture requires more than rule following: the broad norms of Merton, for instance
• In the 20th century, the psychological paradigm did not always adhere to these norms
• Result: the replication crisis
• Current transformation to Open Science is in progress, but of course comes with its own risks and pifalls…

Answer 46

oke