Issues in research methods

Question 1

Who created the hypothetico-deductive method?

Answer 1

Karl Popper

Question 2

What are the 8 stages of the hypothetico-deductive method?

Answer 2

Theory -> hypothesis -> operationalisation of concepts -> selection of participants -> survey studies/experimental designs -> data collection -> data analysis -> findings

Question 3

What did Popper (1972) argue about how we know what to test?

Answer 3

Research begins with the identification of a particular problem/issue and further suggested that there are 2 possible sources of research ideas: casual observation and previous research

Question 4

What is casual observation in the generation of research ideas?

Answer 4

A researcher spots a new phenomenon for the first time and decides it is worthy of investigation

Question 5

How does previous research inform new reserach ideas?

Answer 5

Can motivate new research projects for a number of reasons e.g. replication/generalisation/original findings/testing ideas in a new context/zooming in to look at underlying processes

Question 6

What do surveys measure?

Answer 6

Variables as they naturally occur e.g. physical activity and wellbeing in general public
- variables measured, not manipulated
- Dependent on sampling, may be generalised to wider population

Question 7

What is the function of experiments?

Answer 7

Manipulating variables to isolate their effects to establish a causal relationship through randomisation, and holding other factors constant

Question 8

What is systematic variation?

Answer 8

Variation that can be explained by model (statistic)

Question 9

What is unsystematic variation?

Answer 9

Variation that cannot be explained by model (statistic)

Question 10

What is the equation for a test statistic?

Answer 10

Variance explained by model (systematic) / variance not explained by model (unsystematic) = test statistic

Effect / error = tests statistic

Question 11

What is H1?

Answer 11

The effect you expect to find (alternative hypothesis) - related to systematic variation

Question 12

What is H0?

Answer 12

Null hypothesis - no evidence of effect - related to unsystematic variation

Question 13

What is the process of hypothesis testing?

Answer 13

1. pose hypotheses (one or two tailed)
2. analyse data (test model - error:effect ratio)
3. calculate probability of getting result if null hypothesis is true (p > .050, result is significant)
4. When to reject/fail to reject null hypothesis (don’t accept alt/experimental hyp, just infer evidence of effect)

Question 14

What is a type 1 error?

Answer 14

Incorrect rejection of null hypothesis
False positive
Conclude there is an effect when there is none

Question 15

What is a type 2 error?

Answer 15

Incorrect rejection of alternative hypothesis
False negative
Conclude there is no effect when there is one

Question 16

What are 2 limitations of relying on hypothesis testing (significance)?

Answer 16

1. focus on whether or not result is significant statistically, but not necessarily in broader sense
2. does not give indication of size of statistical effect

Question 17

What does the alpha level predict?

Answer 17

Probability of making a type 1 error - saying there is an effect when there isnt (typical value = 0.05)

Question 18

What does the beta level predict?

Answer 18

Probability of making a type 2 error - saying there is no effect when there is one (typical value = .20)

Question 20

What do effect sizes attempt to address?

Answer 20

Type 1 errors - can be significant but too small to be practically meaningful

Question 21

What are effect sizes also referred to?

Answer 21

Magnitude of statistical effect found

Question 22

What do effect sizes allow us to investigate?

Answer 22

How big a difference / relationship has been found
To compare findings across studies to gauge real-world importance

Question 23

What does the type of effect size used depend on?

Answer 23

The type of test used

Question 24

What are the values for small, medium, and large effects for Cohen’s d?

Answer 24

Small = .2
Med = .5
Large = .8

Question 25

What are the values for small, medium, and large effects for Pearson's r?

Answer 25

Small = .1 Med = .3 Large = .5

Question 26

What are the values for small, medium, and large effects for omega ω?

Answer 26

Small = .1 Med = .3 Large = .5

Question 27

What are the values for small, medium, and large effects for Eta-squared η2?

Answer 27

Small = .01 Med = .059 Large = .138

Question 28

What do power analyses attempt to address?

Answer 28

Control for type 2 errors - tells us statistical power associated with particular test

Question 29

What are the 2 approaches to power analyses?

Answer 29

1. A priori - before collecting data, determines sample size needed to reliably find an effect if there is one to find 2. Post-hoc - power estimated after collection and calculating inferential statistics

Question 30

What are the 10 steps of model building?

Answer 30

1. Pose hypotheses 2. Calculate a priori power analysis to determine sample 3. Collect data 4. Calculate descriptive statistics 5. Identify whether the data meet the assumptions of parametric statistics 6. Conduct the test (e.g., obtain t-value and p-value) 7. Identify whether the result is significant (p < .05) 8. calculate effect size 9. Reject or fail to reject the null hypothesis (based on p-value) 10. Write it up (discussion/conclusions can be framed by effect size)

Question 31

How is the mean calculated?

Answer 31

𝑥 ̅ = Σx/N Add up all the scores (Σx) and divide by the number of scores/number of participants (N) This gives an indication of the central tendency of a data set

Question 32

How is the variance calculated?

Answer 32

s2 = Σd2 / N-1 The sum of all the squared differences (Σd2) divided by the number of scores/number of participants minus 1 (N-1) This provides an indication of the spread in the data. The higher the variance, the more spread, i.e., the mean may be a poor representation of the data. The smaller the number, the less spread – so the mean is more reliable.

Question 33

How is standard deviation calculated?

Answer 33

s = √(Σd2 / N-1) The sum of all the squared differences (Σd2) divided by the number of scores/number of participants minus 1 (N-1). This is then square rooted. The same as the variance, just we square root the variance. Reduces the value down, making it easier to interpret.

Question 34

What are parametric tests?

Answer 34

Make assumptions about the data Normally distributed Homogeneity of variance Usually only for ratio/ interval data Used for e.g., group differences in equally sized groups

Question 35

What are non-parametric tests?

Answer 35

Make no assumptions about the data Violation of normality assumption (e.g., if data are very skewed / sparse) Used if you have ordinal data Or sometimes where you have small group sizes

Question 36

What is the publish or perish issue of research culture?

Answer 36

Career progression dependent on research success

Question 37

What is the file drawer problem of research culture?

Answer 37

Non-significant results less likely to be published

Question 38

What does the research culture lead to?

Answer 38

Cheatign - focusing too much on finding a significant result can lead to problematic research practices such as p-hacking

Question 39

What can problematic research practices lead to?

Answer 39

- a lack of trust in science - Over-inflated effects reported in literature - Effects that do not replicate (replication crisis)

Question 40

What are the 4 core principles of research integrity?

Answer 40

Honesty Accountability Professional courtesy and fairness Good stewardship

Question 41

What is the replication crisis?

Answer 41

Reports on an attempt to replicate 100 psychological studies and found… - Only 36% of studies could be replicated (i.e., found significant results as reported in the original study). - The average effect size of the replications was smaller than the original study (on average half the size of the original effect). - More ‘surprising’ findings were less likely to be successfully replicated. - Social psychology findings were less likely to replicate than those in cognitive psychology.

Question 42

Why don't studies replicate?

Answer 42

1. publication bias 2. failure to control for bias 3. p-hacking 4. low statistical power 5. poor quality control

Question 43

What are the 7 principles of open science?

Answer 43

1. Assessment (comment/peer review; determine impact of research output and researchers) 2. Preperation (define and crowdsource research priorities; organise project, team, collaborations; get funding/contract) 3. Discovery (search lit/data/code; get access; ger alerts/recommendations; read/view; annotate) 4. Analysis (collect, mine, extract data / experiment; share protocols/notebooks/workflows; analyse) 5. Writing (write/code; visualise; cite; translate) 6. Publication (archive/share publications, data, and codes; select journal to submit to and publish) 7. Outreach (Archive/share posters and presentations; tell about research outside academia; researcher profiles/networks)

Question 44

What is p-hacking?

Answer 44

the inappropriate manipulation of data analysis to enable a favoured result to be presented as statistically significant

Question 45

What is HARKing?

Answer 45

Hypothesizing After Results are Known, is a questionable research practice where a researcher presents a hypothesis developed after observing the results of a study as if it were a pre-existing hypothesis, one developed before the study began

Question 46

How can pre-registration help questionable research practice?

Answer 46

- Pre-registration encourages more “up-front” thinking about a study. - Researchers have to carefully consider each stage of the research process, from generating hypotheses to data analysis. - The aim is to reduce the risk of questionable research practices (QRPs) and biases during the research process (e.g., HARKing and P-hacking). - It doesn’t stop anyone from cheating, but does try to ensure that the research process is transparent, well thought through, and ensures that the inferential statistics are truly confirmatory (planned in advance). - It can be time consuming, as the process requires researchers to think through every stage of the process, and consider how they will deal with issues if they occur (e.g., data that do not meet the assumptions of their chosen analysis). But this in itself often leads to better quality research.