Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

PSYC 306 > Lecture 22: Replication Study Designs > Flashcards

Lecture 22: Replication Study Designs Flashcards

1
Q

replicability

A

whether a published study’s findings can be repeated in a different lab, using the same or similar research methods

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

the replication crisis in psychology

A

a failure for published research findings to be reproduced in other labs when they followed the same or similar research methods as the published findings

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

the gold standard

A

a scientific finding’s replicability is often considered the best possible evidence for the accuracy of a finding

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

reproducibility

A

the ability of a different researcher to reproduce another researcher’s published analyses, given the original dataset and computer code for the statistics used

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

U.S. National Science Foundation (NSF) subcommittee’s definition of reproducibility

A
  • Reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator
  • A second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

replication in social and cognitive psychology studies

A

Only 36% of social psychology and cognitive psychology studies published in scientific journals were replicated at the p < .05 level

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

replication rates of cognitive vs. social psychology

A
  • Cognitive psychology (50%)
  • Social psychology (26%)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

why do researchers think there are lower rates of replication in social psychology?

A

Many researchers think that the lower rates in social psychology are due to the frequent use of surveys and the lack of control over who answers surveys

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

replication in other science domains

A
  • 70% reported a failure to replicate the results of other studies
  • 87% of chemists
  • 77% of biologists
  • 69% of physicists and engineers
  • 67% of medical researchers
  • 64% of environmental and Earth scientists
  • 62% of all other respondents
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

replicating one’s own experiment

A

50% had failed to replicate their experiment

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

do researchers agree that there is a replication crisis?

A

The majority of researchers surveyed agree that there is a replication crisis

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

is the replication crisis specific to psychology?

A

no

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

are most labs working to enhance replicability

A

yes

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

common methods to enhance replicability

A
  • Better document the research methods used
  • Run the study again
  • Ask the lab member to replicate the study
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

why are replication rates lower for psychology?

A

the use of human participants can introduce significant variability

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

failures to replicate in social psychology

A
  • Stanford Prison Experiment: was conducted by Zimbardo and was widely criticized for ethical concerns and has faced skepticism regarding the generalizability of its findings. Subsequent replications and reevaluations have raised doubts about the validity of the original study
  • The Bystander Effect: suggests that individuals are less likely to help in emergencies when others are present. This has faced mixed replication results. Situational and contextual factors may play a significant role in whether or not bystanders intervene
  • Stereotype Threat: individuals from stigmatized groups underperform due to the fear of confirming stereotypes. This has faced mixed replication results, particularly in real-world settings
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

failures to replicate in cognitive psychology

A
  • Spotlight Attention: individuals focus their attention on a specific area of the visual field. This has faced challenges in replication. Some studies have found that attention is more distributed and flexible than originally theorized
  • The Dual-Process Model of Memory: posits that memory consists of two separate systems, one for implicit and another for explicit memory. This model has faced scrutiny, and some researchers have proposed alternative models of memory
  • The Mirror Neuron System: a system believed to play a role in understanding and imitating the actions of others and has faced some challenges in replication. Some studies have reported inconsistencies in the neural responses associated with mirror neurons
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

failures to replicate in developmental psychology

A
  • The Critical Period Hypothesis for Language Acquisition: implies that there is a specific window during which language learning is optimal. Some studies have reported individual variations and exceptions to this hypothesis
  • Attachment Theory: describes different attachment styles in children. Some replication studies have suggested that attachment styles are more fluid and context-dependent than originally proposed
  • The Mozart Effect on Infant Intelligence: suggests that listening to classical music, particularly Mozart, can boost children’s cognitive abilities, has faced some replication challenges. Subsequent studies have not consistently replicated the idea that listening to Mozart leads to long-term cognitive enhancements
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

failures to replicate in clinical psychology

A
  • The Dodo Bird Verdict: refers to the idea that all psychotherapies are equally effective and that it is the therapeutic relationship that accounts for positive outcomes. Specific therapeutic approaches, such as CBT, can have different effectiveness for certain conditions
  • The Power of Positive Thinking: the idea that a positive attitude can lead to improved physical health outcomes has faced challenges. Replication studies have not produced consistent results
  • The Clinical Efficacy of Memory Recovery Techniques: some memory recovery techniques, such as hypnosis and guided imagery, have been used in therapy to help individuals recall repressed memories. However, replication studies have shown that the accuracy of such recovered memories can be unreliable
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

failures to replicate in neuroscience

A
  • The Amygdala’s Role in Fear Processing: while the amygdala is traditionally associated with fear processing and emotional responses, some replication studies have reported variations in the amygdala’s involvement in fear- and non-fear-related tasks
  • The Left Brain-Right Brain Distinction: the notion that the left hemisphere of the brain is primarily associated with logical thinking and the right is associated with creativity and emotion has faced challenges. Neuroscience research has shown that both hemispheres are involved in a wide range of cognitive functions
  • The Brain-Training Effect: some studies have suggested that engaging in cognitive training exercises can improve cognitive function and memory in older adults. However, replication efforts have produced mixed results, with some studies failing to consistently replicate the benefits of brain-training programs
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

causes of the replication crisis

A
  1. ignoring or misunderstanding statistics
  2. publication bias
  3. falsifying data
  4. quality of replication
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

ignoring or misunderstanding statistics

A
  • Misunderstanding the null hypothesis and the meaning of p-values
  • Small sample sizes
  • Effect size and power
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

publication bias

A

the way we conduct, publish, distribute, and fund our science

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

examples of falsifying data

A
  • D. Stapel, 2011: psychology professor at Tilburg University who had 55 cases of fabricated data in social psychology
  • Marc Hauser, 2012: psychology professor at Harvard who was accused of faking results on morality and cognition
  • Karen Ruggiero, 2001: psychology professor at the University of Texas who was accused of faking results on discrimination research
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Q

quality of replication

A

failure to follow the original procedures

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
26
Q

causes of failures in quality of replication

A
  • This can be due to the decision-making of the replicator
  • This can be due to incomplete method descriptions by the first researcher
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
27
Q

HARKing (Hypothesizing After the Results are Known)

A

involves formulating or changing hypotheses after analyzing the data. In this practice, researchers may first explore their data without a specific hypothesis and then generate a hypothesis based on what they find. This can lead to confirmation bias, as it gives the appearance that the results were predicted when they were discovered post hoc

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
28
Q

why does HARKing go against the principle of research strategy?

A

it shows that the IV causes (precedes) the change in the DV

29
Q

SHARKing

A
  • Secretly HARKing in the introduction section
  • “Publicly presenting in the Introduction section of an article hypothesis that emerged from post hoc analyses and treating them as if they were a priori”
  • Instead of making hypotheses based on results from all existing studies (standard practice), the researcher hypothesizes after knowing the results from the data at hand
30
Q

is SHARKing justified?

A

no, it’s never justified in science

31
Q

THARKing

A
  • Transparently (openly) HARKing in the discussion section
  • “Clearly and transparently presenting new hypotheses that were derived from post hoc results in the Discussion section of an article”
32
Q

is THARKing justified?

A
  • Can promote effectiveness and efficiency of science
  • Ethically required in some cases
  • THARKing is justified in science, according to Hollenbeck
33
Q

Hollenbeck’s two case studies

A
  1. A researcher desperate to get a job takes 30 of the shortest and most easily obtained survey measures and creates a pair of long questionnaires… they run a new survey find some significant correlations and publish them as a priori hypotheses. No one can replicate the findings. (example of HARKing)
  2. Epidemiologists test 100 patients on new drug to protect against virus. Correlation between treatment (drug) and survival rate = r of .1 (small). Some researchers notice that females react differently to the drug than males. The researchers re-evaluate the findings by peak estrogen age of participants and publish a short report as a post-hoc analysis. Others replicate the findings. (example of THARKing)
34
Q

Null Hypothesis Significance Testing (NHST)

A

an assumption that the null hypothesis means that there is no significant difference between the groups or conditions being compared

35
Q

example of Null Hypothesis Significance Testing (NHST)

A

condition A yields the same outcome as condition B

36
Q

issue with NHST

A

With a large enough N, virtually every study would yield significant results

37
Q

p-hacking

A

the unethical and questionable practice of manipulating or “hacking” statistical analyses to achieve statistically significant results. Researchers may engage in various data techniques (trying multiple outcome measures, analyses, or excluding data points selectively) until a statistically significant result is obtained. This can lead to false-positive findings and misrepresentation of the true state of affairs.

38
Q

methods of p-hacking

A
  • Stop collecting data once p < .05
  • Analyze many measures but report only those with p < .05
  • Collect and analyze many conditions, but only report those with p < .05
  • Use covaraites to get p < .05
  • Exclude participants to get p < .05
  • Transform the data to get p < .05
39
Q

cherry-picking data

A

Researchers may selectively report only the data or results that support their hypotheses while disregarding data that contradicts their expected outcomes. This practice can lead to a biased representation of the research findings.

40
Q

data fabrication and falsification

A

Intentionally creating or altering research data to support desired outcomes. It is a clear violation of research ethics and can have severe consequences, including academic and professional repercussions.

41
Q

the file drawer problem

A

Studies with non-significant or null results are less likely to be published or reported compared to studies with positive or significant findings.

42
Q

solution to the file-drawer problem

A

publish a meta-analysis that analyzes data from all articles on a particular topic (including outcomes that show no effect – whether published or not)

43
Q

selective reporting

A

Researchers and academic institutions tend to favour the publication of studies that reveal a significant effect or a novel discovery, leading to an imbalance in the scientific literature.

44
Q

solution to selective reporting

A

shift the balance toward journals that agree to publish null results when the research methods are judged in advance to be strong

45
Q

incomplete knowledge

A

When non-significant results are not published or made
publicly available, the scientific community may have an incomplete or biased view of a particular research question, as only a portion of the available data is accessible.

46
Q

solution to incomplete knowledge

A

larger sample sizes, reduce problems of internal validity and confounding variables

47
Q

replication challenges

A

When other researchers attempt to replicate findings based on published studies, they may encounter difficulties if non-published studies with null results exist in the “file drawer” and have not been considered in the replication effort.

48
Q

solutions to replication challenges

A

Open science practices (registration of research studies, publication of null findings) have been advocated to improve transparency and the reliability of scientific research

49
Q

types of publication biases

A
  1. the file drawer problem
  2. selective reporting
  3. incomplete knowledge
  4. replication challenges
50
Q

pre-registration of research methods

A

A detailed plan for research methods that are filed online (open) ahead of data collection

51
Q

what does pre-registration include?

A

includes writing what your hypotheses are, which specific methodologies you’ll use, and which analyses you’ll conduct. These are set in stone

52
Q

is there review prior to data collection with pre-registation?

A

no

53
Q

what do you get when you complete pre-registration?

A

a DOI

54
Q

DOI

A

a unique and permanent string of letters and numbers that identifies each article filled out online

55
Q

can you pre-register studies that use secondary data?

A

yes

56
Q

where can you find templates for pre-registration?

A
  • It depends on the scientific field, but usually online
  • For psychology, you use PsychArchives online
57
Q

sections of pre-registration

A
  • Title page
  • Introduction
  • Methods: sampling participants, IV, DV, materials, procedure
  • Planned analyses
58
Q

registered reports

A

peer review of the research methods prior to data collection

59
Q

when are registered reports accepted?

A

if the authors follow through with the registered methodology

60
Q

traditional review method for scientific articles

A

is only Stage 2 peer review when it is too late for the authors to adjust their research methods

61
Q

stage 1 peer review

A

registered report method (review the research methods)

62
Q

stage 2 peer review

A

traditional review of contents (did the authors use the research methods they promised)

63
Q

pre-registered vs. registered reports

A

registered reports have stage 1 review and in-principle acceptance but pre-registered reports don’t

64
Q

prevalence of pre-registration vs. registration

A

There are many more pre-registered than registered studies

65
Q

benefits of registered reports

A
  • Encourages the creation of sound research questions and a priori methods and analysis plans
  • Encourages open data and transparency
  • Reduces publication bias
  • Reduces p-hacking and HARKING
  • Gives you guaranteed publication
  • Gives you peer review when it’s most helpful
  • Reduces selection bias
66
Q

which of the benefits of registration reports DO NOT apply do pre-registration reports

A
  • Reduces publication bias
  • Gives you guaranteed publication
67
Q

does pre-registration improve replication rates?

A
  • Evidence suggests that pre-registration alone does not improve replication rates
  • But, it seems to improve perceptions of higher-quality research
68
Q

does registration improve replication rates?

A
  • Evidence suggests that registered reports improve replication rates
  • This doesn’t solve publication bias
69
Q

why tharking may have value

A
  • If we were just left to our powers of deductive, researchers may miss almost everything
  • The history of science is replete with accidental findings (33-50% of findings)
  • Historically low power in primary studies
  • Most approaches to grounded theory rely on the richness of qualitative observations that cannot be replicated in a strictly technical sense
  • Improves clarity in writing and exposition
  • There are ethical costs of failing to leverage costly data

PSYC 306 (17 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions