Effect Size and Statistical Power

Question 1

Think about the things that might make an experiment have a significant result

Answer 1

• If you are measuring something that is easy to measure
• Huge difference exists in real life (e.g., IQ of toddlers vs. university students)
• If you have a really good measurement method
• Everything controlled for by experimental design (e.g., test the same participants in two conditions)
• If you have lots of resources
• Test lots of participants
• Take many measurements from each participant
• [Ethically unsound: keep doing the experiment again until it is significant!]

Question 2

Effect Size

Answer 2

• Why does it matter?
• Small effects are often not meaningful in real life (e.g., if an intervention reduced depressive symptoms by 1/100th of an SD but took 8 hours a day for five years to complete)
• People might want to compare different experiments from the literature
• Significance tells us which group did better or which relation is present, but effect size tells us how big is the difference or how strong is the relationship
• If you know how big an effect is, you can estimate how difficult it will be to measure a significant result: Power Analysis
• This allows you to calculate how many participants to test/ how many measurements to take from each participant

Question 3

Effect Size Numerical Examples (Cohen, 1988 p 10)

Answer 3

• If the percentage of males in the population of psychiatric patients bearing a diagnosis of paranoid schizophrenia is 52%. and the effect is measured as a departure from the hypothesized 50%. the ES is 2%; if it is 60%, the ES is 10%, a larger ES.

Question 4

Standardised Effect Sizes

Answer 4

• Correlation between two variables, the r-statistic itself is the effect size:
• r = .1, weak; r = .5, moderate; r = .7, strong; r = .9 very strong
• Differences between means in t-tests Cohen’s d:
• d = .2 small; d = .5, medium; d = .8, large
• ANOVA/F-tests (variability of DV explained by IV) effect size is η² (Eta squared)
• η² = .01, small; η² = .059, medium; η² = .138, large

Question 5

Cohen’s d

Answer 5

Just as z-scores represent points of data in a standardised way. Cohen’s d expresses the size of a difference in a standardised way.

Cohen’s 𝑑= (M_2-M_1)/〖SD〗_Pooled

And where M and SD are the mean and
standard deviation of each measure/group

Individuals in the experimental group remembered an average of 6.7 words (SD = 2.1), whilst individuals in the control group remembered and average of 6.2 words (SD = 2.3). Calculate the effect size:

Question 6

How do we measure small effects?

Answer 6

• If effects are small, they are harder to measure

* This means we need more statistical power

Question 7

Think about the things that might make an experiment have a significant result

Answer 7

• If you are measuring something that is easy to measure
- Huge difference exists in real life (e.g., IQ of toddlers vs. university students)
• If you have a really good measurement method
- Everything controlled for by experimental design (e.g., test the same participants in two conditions)
• If you have lots of resources
- Test lots of participants
- Take many measurements from each participant
- [Ethically unsound: keep doing the experiment again until it is significant!]

Question 8

Power is influenced by

Answer 8

• Effect Size
• Sample Size
• Alpha Level
• Directional Hypotheses (i.e., 1-tailed)
• Variance of the population

Question 9

What is power?

Answer 9

The power of a statistical test of a null hypothesis is the probability that it will lead to the rejection of the null hypothesis, i.e., the probability that it will result in the conclusion that the phenomenon exists.
Cohen (1988, p4)

Question 10

How much power should we have

Answer 10

• Typically researchers aim for a power of 0.8
- This means that they have an 80% chance of finding a significant effect if that effect really exists: E.g., if they ran an experiment 100 times, they would expect 80 of the experiments to be significant.
• Why not 100%
- With large enough power, spurious tiny effects can lead to a significant result
• Such effects may not be attributable to a real-life phenomenon

Question 11

summary

Answer 11

• Power is the probability that an experiment will lead to a significant result if there really is an effect in the population
• This depends on:
• Effect Size
• The size of the effect being measured
• The variance in the data
• The number of participants being tested
• The power of a study can influence the resources needed to measure a significant result