Significance and power Flashcards
(19 cards)
NHST
Null Hypothesis Significance Testing
Principles of NHST
-Assume H0 is true
-Fit model to data, get a test statistic
-Calculate probability of getting test statistic, assuming H0 is true (p)
Misuse of NHST (ASA, 2016)
-P values don’t indicate probability that results occurred by chance or specific hypothesis is true
-Statistical significance ≠ practical importance
-p-value is poor measure of evidence
Type I error (α)
False positive (rejecting true null)
Type II error (β)
False negative (failing to detect true effect)
Power
-1-β
-The probability of detecting an effect if it exists
-Typically set at 0.8 (β=0.2) (Cohen, 1992)
Factors affecting power
-Effect size
-Sample size
-Alpha level (α)
-Variability
-Design type
-Stats test used
How effect size affects power
-Standardised measure of effect magnitude
-Depends on Cohen’s d (t-tests), Pearson’s r (correlations), partial eta² (ANOVA)
How sample size affects power
-Larger samples=more power
-Bigger effects need fewer pps
-Smaller effects need more pps
How alpha level affects power
-Significance threshold
-Common values: .05, .01, .001
-Lower α= reduced type I error, but also lower power
Familywise error rate
-Running multiple tests increases type I error
-Use Bonferroni correction to control error rate
Bonferroni correction
𝑃𝑐𝑟𝑖𝑡= 𝛼/𝑘
One-tailed hypothesis
-Predicts direction
-More powerful (α concentrated in one tail)
-Use cautiously, higher chance of detecting effect
Two-tailed hypothesis
-No predicted direction
-Α split between two tails
-Recommended for most studies
Study design and power
-Within-subjects: more powerful, fewer pps needed
-e.g. to detect a medium effect (d=0.5) with 80% power
-Between-groups: 128 pps (64 per group)
-Within-groups: 34 pps
Power analysis
-Post-hoc
-A priori
Post-hoc
After data is collected
A priori
Before study to determine required sample size
G*Power
Recommended for power calculations
