Research methods Flashcards
1
Q
The misuse of NHST (Null Hypothesis of Significance Testing)
A
- The American Statistical Association (2016) outlined principles on the misuse of p values in significance testing
- P-values are not measuring the probability of getting results by chance, or that a specific hypothesis is true
- Statistical significance is not the same as practical importance
- The p-value alone is not a good measure of evidence regarding model or hypothesis
2
Q
Type 1 and Type 2 errors
A
- Type 1 = incorrectly accepting alternative hypothesis
- Type 2 = incorrectly accepting null hypothesis
3
Q
Power
A
- The probability of finding an effect assuming one exists in the population
- Calculated as 1-B
- B is the probability of not finding the effect (usually 0.2 as stated by Cohen)
4
Q
What effects power? 3 factors
A
- Effect size: an objective and standardised measure of the magnitude of an effect (larger value = bigger effect size)
Depends on test concluded – cohen’s d, pearson r, partial eta squared (ANOVA) - Number of participants: more participants = more ‘signal’, less ‘noise’. You should choose sample size depending on the expected effect size (larger effect size = fewer pp’s, smaller effect size = more pp’s)
- Alpha level: the probability of obtaining a Typer 1 error. We compare our p value to this criterion when testing significance
- Other factors: variability, design, test choice
5
Q
Problems with alpha testing
A
- If we run multiple tests, this will increase the rate at which we might get a type 1 error (family wise experimental error rate)
- We can account for this by limiting the number of test or by using corrections such as Bonferroni correction (but this reduces statistical power)
6
Q
What is the difference between one and two-tailed tests
A
- One-tailed- we hypothesise there will be a difference in scores, and we’re specific about which score will be higher (α=.05 at one end)
- Two-tailed- We hypothesise there will be a difference in scores, but this could be in either direction (α= .025 at both ends)
- For a one-tailed test, our p-value is half of the two-tailed p-value
7
Q
Which type of test do I run?
A
- One-tailed tests are more powerful as a is higher
- However, there are several caveats and considerations so in most cases, it is recommended that run a two-tailed test
8
Q
Power and study design:
A
- Within-subjects studies are more powerful than between-subjects studies
- To run a t-test with a: two-tailed design, medium effect size, a level of 0.05, power level of 0.8
- 1) Calculate the power we have obtained in a study post-hoc
- 2) Calculate how many participants we need to collect for a study a priori (this can be done using statistical programs such as G*Power)
9
Q
What is analysis of variance?
A
- Analysis of variance (ANOVA) is an extension of the t-test
- it allows us to test whether 3 or more population means are the same, without reducing power
10
Q
Assumptions of ANOVA
A
- the scores were sampled randomly and are independent
- roughly normal distribution
- roughly equal number of participants in the groups
- roughly equal variance for each condition
11
Q
The basis of the ANOVA test
A
- analysis of variance is a way to compare multiple conditioned in a single, powerful test
- It was invented by Fisher (so its test statistic is F)
- It compares the amount of variance explained by our experiment with the variance that is unexplained
12
Q
Between-groups ANOVA
A
- The aim of ANOVA is to compare the ‘amount of variance explained by our experiment with the variance that is unexplained’
- For between-group designs:
- A) the explained variance is the variance between group
- B) the unexplained is the variance within a group
- The calculation is referred to as the mean squared (MS) error
13
Q
Degrees of freedom
A
- There are degrees of freedom associated with both variance values:
- A) degrees of freedom between conditions
- B) residual degrees of freedom
- ANOVA critical values require 2 d.f. values, one for each aspect of the variance
- We must report both
14
Q
Pair-wise comparisons
A
- ANOVA tells us whether groups differ or not
- How do we know which particular conditions?
- Run the multiple comparisons (those we were trying to avoid0
- Some of these are ‘planned comparisons’, some are ‘post-hoc’
- Correct for multiple comparisons
15
Q
Versions of ANOVA
A
- Analysis of variance (ANOVA) – one factor ANOVA and multifactor ANOVA
- Multivariate analysis of variance (MANOVA) – extension of ANOVA for multiple dependent variables
- Analysis of covariance (ANCOVA) – extension of ANOVA to handle continuous variables (e.g. correlations)
16
Q
What is ANOVA based on? (for between-groups)
A
A) the variance explained by the experiment (the effect)
B) the residual (remaining) variance that cannot be explained (noise)
- For between-group design, the variance comes from only two sources:
A) variance between groups (explained)
B) variance within groups (unexplained)
17
Q
Between group vs repeated measures for ANOVA
A
- For repeated-measure design, there are three possible sources of variance:
A) variance between conditions
B) variance between subjects (individual differences)
C) residual (unexplained) variance - In the between-group study, the variance between subjects fell under the category ‘unexplained’
18
Q
What is the F-ratio and MS unexplained formulas?
A
- F = MS explained / MS unexplained
- MS explained is the variance between conditioned
- MS unexplained is the remaining variance after accounting for individual differences
- MS unexplained = MS total – MS explained – MS ind diffs
- MS ind diffs is the variance between subjects within a condition
- MS total is the variance of all subjects in all conditions
19
Q
What is multi-factorial ANOVA?
A
- Like repeated-measures ANOVA
- Factors can all be within-subject, all between-group or a ‘mixed’ design
- We can have a ‘main’ effect or a variety of ‘interactions
- Main effect = one of the factors (IV) consistently affects the (DV) in the same way
- Interaction = the effect of one factor depends on the presence of another
20
Q
What is 2x2 ANOVA?
A
- The multifactorial ANOVA is a single test
- It returns multiple F values (one for each main effect to be checked and one for the interaction)
- With one two levels, there is no need for post-hoc tests
- So 2x2 is just a single test (no family-wise error)
21
Q
What are contingency tables?
A
- A table of frequencies for how often an often an observation occurs in a category
- Categories must be mutually exclusive and exhaustive
22
Q
What is the Chi-Square test?
A
- Devised by Karl Pearson in 1900, also known as Pearson’s chi-square
- Calculates how often a particular observation falls into a category based on how many were expected by chance
- Null hypothesis = the frequencies observed were expected by chance
- Alternative hypothesis = the frequencies observed reflect real differences in categories
- Assumptions: 1. Independence (each person can only contribute to one cell of a contingency table) 2. Expected frequencies (all expected counts should be greater than 1 and no more than 20% of expected counts should be less than 5)
23
Q
Violating expected frequencies: and how to reverse it
A
- Results in a loss of power
- How to reverse this:
- A) use an ‘Exact’ test instead
- B) remove data across one variable
- C) collapse levels of one variable
- D) collect more data
- E) accept the loss of power
24
Q
Chi-square by hand: one IV
A
- Calculate expected frequencies
- Calculate Chi-Square value based on observed and expected frequencies
- Compare Chi-Square value against a critical values table
- To interpret a table, we need to know our degrees of freedom, and our desired alpha value (degrees of freedom = number of categories – 1)
25
Chi-square by hand: two IVs
- With two IVs, the difference will be in calculating the expected values in each case
- To calculate expected frequencies for two IVs, we need to calculate expected frequencies of specific cells
26
What is the binomial test?
- Compares observed and expected frequencies for variables with only two levels
- E.g. are there more pp’s in our sample from the USA than we would expect by chance?
27
Inferring from samples
- Statistical power: probability of seeing a true positive
- Alpha (a): the highest acceptable risk of a false positive (typically 5%)
28
Publication bias & the file drawer problem
- Researchers biased towards results which support their theories
- Significant results are more likely to be published
- Many journals value novelty and surprising results
- Non-significant results are often not published
- Non-significant replications are hard to publish
- Researchers are under pressure to find significant results
29
Researcher degrees of freedom
- There are many valid ways to analyse a given dataset:
- A) different statistical tests
- B) different variables
- C) different rules for excluding outliers
30
P-hacking & HARKING
- P-hacking is a way to cheat/lie with statistics
- For any test, we accept a 5% probability of a false positive
- P-hacking: performing the analysis in different ways to get p<.05 and only reporting the significant results
- This results in false positive: we cannot trust the results
- HARKING: hypothesising after the results known
31
Multiverse analysis
- Run many possible analysis
- See how many get a significant result
- Munoz & Young (2018) analysed the data with N = 1152 regressions
- Less than 5% had a significant effect
32
Two problems: True and False positives
- Significant results easer to publish, including false positives
- Many papers are underpowered, true positive are not seen
- Leads to many false positives in the literature
33
How to solve the reproducibility crisis? 3 ways
1. Open materials – share the exact materials (instructions, program, stimuli), makes it easier for others to replicate
2. Open data – share the raw data so other researchers can perform the analysis and see how other variables/ analyses affect the results
3. Preregistration – plan the study in advance, including materials, planned analysis (e.g. open science framework), prevents p-hacking and HARKING. Researchers can compare your pre-registration to the final study
34
Why do we need to visualise data?
- Makes it easier to understand datasets
- E.g. Florence Nightingale used data visualisation to highlight British soldiers living conditions in the Crimean War (1858)
35
What is the purpose of data visualisation?
- Data analysis process: check that assumptions have been met and understand the relationships between variables before inferential analysis
- Report writing and publication process: show clear relationships between variables and help the reader interpret the data in the way you want them to
36
Types of graphs: 3 types
1. Checking data assumptions – graphs can be useful for checking data assumptions before running statistical tests (e.g. histograms, boxplots)
2. Summarising descriptives – graphs can also help to summarise lots of descriptive statistics (e.g. bar charts, clustered bar charts)
3. Graphing relationships – we can use scatterplots to graph relationships between variables (and sometimes check assumptions)
37
What makes a good graph? 7 things
- Tufte (2001) and the American Psychological Association (2021) suggest that:
A) images are clear
B) units of measurement are provided
C) axes are clearly labelled
D) elements in the figure are clearly labelled or explained
E) avoid distorting the data
F) induce the reader to think about the underlying messages of the figure
G) avoid using chartjunk (the use of unnecessary or misleading elements in the design of a graph
38
Bad graphs in public health data
- no x and y axis labels
- dates on the x axis are in a random order
- colours of bars are not consistent across clusters of bars
39
What is ‘parametric’?
- They are based on some commonly used parameters (e.g. standard deviation), which assume a normal distribution
- If the data is not normally distributed, then those parameters might not be meaningful
- Most data on a ratio or interval scale are normally distributed
- When data are heavily skewed, the step size between points on the scale is probably not constant
- When data are ordinal you may not have a normal distribution and parameters like SD, SEM, variance may no longer capture the data
40
Non-parametric test: Mann-Whitney U
- If we arrange all the data into an ascending sequence of scores:
a) When the null hypothesis is true, we would expect the group labels then to be randomly distributed
b) When the null hypothesis is false, we would expect the scores of the two groups to be clustered at either end of the sequence
41
Non-parametric test: Wilcoxon T
- Test is based on the difference between the 2 scores for each subject
- It uses the direction (which score is greater) and magnitude of the differences (how much greater)
- If H0 is true, then the differences in one direction will be as large as the differences in the other
- Whenever, we have tied ranks, we take the average of the range of ranks that the ties cover and allocate this to the value of the ties
42
Non-parametric test: Kruskal-Wallis H
- When the null hypothesis is true, we expect a random distribution of ranks across groups
- When the null hypothesis is false, we expect a systematic distribution of ranks across groups
43
Non-parametric test: Friedman test
- Rank within each subject
- Does not tell us which conditions differ
44
What are resampling techniques?
- A few basic ideas that can be modified and reused
- No equations or tables to look up – maths is easier
- Fewer assumptions so more accurate if assumptions not met
- Resampling does require:
- A) a computer
- B) some programming
45
Two common uses for resampling
1. Permutation tests – for comparing groups/ conditions (e.g. t-test replacement). Shuffle data according to your conditions
2. Bootstrap resampling – for generating intervals (e.g. make error bars). Resample-with-replacement the values in a sample
- Others: Jack-knife, Monte-Carlo method
46
Resampling for hypothesis testing
- The point of inferential statistics:
- To determine the probability that the differences we measured were caused by sampling error
- The principle of resampling techniques is to measure that sampling error by repeating the sampling process many times
- We can determine the likely error in introduced by the sampling by looking at the variability in the resampling
47
Resampling: Permutation tests (between-subject randomisation tests)
- We assume that these are real and sensible values
- We do not assume anything about their distribution
- Repeat process many times, forcing the null hypothesis to be true, and check how extreme the real value was
- No equation needed, except for the statistic of interest e.g. mean
- No table needed: the data, themselves, give the p value
48
Bootstrap resamples
- Can be used to calculate confidence intervals (e.g. standard error of the mean, confidence interval of a mean like 95%)
- They can also determine whether some test value is inside or outside the 95% confidence interval
- Used for confidence of simple values (mean) or for fitted parameters (gradient of a line)
- SEM is the standard deviation of the means of all possible samples, and it can be estimated from the standard deviation of the bootstrap means
- Bootstrapping generalises easily to more complex models than just the mean
49
Bootstrap advantages
- Very general method: any type of model can be used and confidence intervals on any of its parameters can be estimated
- Can also be used to perform hypothesis testing (for one-sample tests)
- Not based on any assumptions about the data
- No tables, no equations (except for the model)
50
Resampling summary: 4 steps
1. Stimulate recollecting the data
2. If the original data don’t look likely from your null distribution, then the null hypothesis is presumed not a good model of your data
3. These tests make very few assumptions about your data
4. They don’t throw away information
51
Bayes’ theorem: Key facts
- Thomas Bayes (1702-1761), statistician and philosopher
- Our perceived probability that something is true depends on the data about it as well as our previous expectations
- Bayes formalised that intuition into some equations
- It has become hugely important for many aspects of science
52
Posteriors, likelihoods and priors of Bayes Theorem
- Posterior: probability after seeing the data
- Prior: probability before seeing the data (expectation)
- Likelihood: evidence coming from the data itself
- Posterior = likelihood x prior
- (belief = evidence x expectation)
- One implication of Bayesian theory is that when something is surprising, we intuitively require better evidence to believe it
53
Bayesian Inferential Tests
- Null Hypothesis Significance Testing (NHST) – calculate the likelihood of this data when Null is H0. We reject H0 only if there’s good reason to do so. Failing to reject H0 doesn’t mean it was more likely than H1
- Instead, ‘Bayesians’ say – calculate the probability of H1 & H0 given the data. Calculate the ratio of those probabilities (i.e. Bayes Factor)
54
Bayes Factors
- These are the Bayesian equivalent of a p-value
- They operate in both directions
- Bayesians like to point out that BF is continuous – don’t need a hard boundary
- BF categories soon appeared too:
a) 0.0-1/3 evidence for H0
b) 1/3-3 not much evidence for anything
c) 3 evidence for the theory
55
Evidence for H0 in Bayesian theorem?
- In NHST H0 is always designed to be Null effect (no difference, no relationship etc.)
- In Bayesian statistics H0 is not necessarily the “Null” hypothesis. It can be any other hypothesis (so often referred to as M1/M2 rather than H0/H1)
- But often it is the Null effect (are the conditions the same) that we want to know about
- Mathematically Bayes Factors can provide support for the Null (e.g. when BF<1/3)
- To say 2 things are the same we have to say what level of “tolerance” (which Bayes tests also handle)
56
Bayesians don’t like “Frequentists”
- Bayesians refer to traditional statistics as “frequentists” because of the logic of the p value (they say p only tells you how ‘frequent’ this data would be assuming that H0 is true & they like to point out this isn’t the same as telling us how likely the original hypotheses are)
57
Credible intervals and comparison with traditional stats
- These are the Bayesian equivalent of Confidence intervals
- Aim to give you the “right” thing
Some key differences with traditional stats:
- Can also incorporate a priori knowledge explicitly in the test (but often used with a “flat” prior)
- Balanced test of whether H0/H1 is more likely, rather than assuming H0 unless confident about H1
- Can use any hypothesis as H0 (not necessarily the no-difference hypothesis)
- Philosophically, a better fit with what stats aims to do
