resampling statistics Flashcards
introduction, motivation, common uses of resampling, resampling for hypothesis testing, permutation tests, bootstrap resamples, other resample approaches, issues and concerns with resampling (26 cards)
what does resampling technique represent?
novel method that is assumption-free(er) but retains power
what is the motivation for using resampling statistics?
fewer assumptions - so more accurate if assumptions not met
what very generally is resampling statistics?
a few basic ideas that can be modified and reused
no equations or tables - to look up the maths is actually easier
thinking about the test forces us to think about our data
why are resampling approaches not popular?
new (1979 is recent for stats) and assumed (incorrectly) to be more complex
parametric statistics do a reasonably good job and are discussed in simple (ish) language in textbooks
resamples requires a computer, some programming (not available in SPSS)
a lot of people don’t like thinking about their data
what are common uses of resampling?
permutation tests
bootstrap resampling
what are permutation tests?
for comparing groups/conditions (e.g. t-test replacement)
shuffle data according to your conditions
what is bootstrap resampling?
for generating confidence intervals (e.g. make error bars)
resample-with-replacement the values in a sample
what is the point of inferential statistics?
to determine the probability that the differences we measured were caused by sampling error
what is the principle of resampling techniques?
to measure that sampling error by repeating the sampling process a large number of times
can determine the likely error introduced by the sampling by looking at the variability in the resample
what are the different types of permutation tests?
between-subject randomisation tests
within-subjects randomisation tests
how do between-subject randomisation tests?
want to determine the likelihood of getting differences this extreme if the data all came from a single “population”
what is the process of between-subject randomisation tests?
simulate running the experiment many times with the data coming from one population - check what range of values commonly occur
in practice - keep the measured values but shuffle them (randomly assigning them to two groups), count how often the difference between the new means is bigger than between the measures means
assume that these are real and sensible values but don’t assume anything about their distribution
repeat process a large number of times
what is the null distribution in between-subject randomisation tests?
distribution of expected experiment results if the null were true
what is the summary of between-subjects randomisation tests?
repeat simulated experiment a large number of times, forcing the null hypothesis to be true, and check how extreme the real value was
no equation needed except for the statistic of interest (e.g. mean)
no table needed, the data themselves give the p value
what is generalisation in between-subjects randomisation tests?
if our hypothesis is that the groups differ in diversity (standard deviation) rather than mean - repeat process large number of times
don’t need a whole new test if we change our opinion of what is interesting in the data
don’t need a parametric and non-parametric version of the test
very similar approach for within-subjects design
what is the within-subjects randomisation test?
now the populations that we randomise are within subjects
in each resample, values are shuffled for each subject rather than across the dataset
just randomise the sign of difference for each pair
repeat process a large number of times
how does number of participants affect the within-subjects randomisation test?
t-tests use n (number of subjects) in their equation
how is that accounted for here?
the sample size for the resamples has to be the same as the original data
the variance in the mean differences will automatically reflect the number of subjects
what are bootstrap resamples?
used to calculate confidence intervals (confidence interval of a mean, standard error of the mean)
also determine whether some test value is inside or outside the 95% confidence interval (like a one-sample test)
used for confidence of simple values (like mean) or for fitted parameters (like gradient of a line)
resample with replacement
how do you use bootstrap resamples to calculate SEM?
SEM = standard deviation of the means of all possible samples
can be estimated from the standard deviation of the bootstrap means
difference in SEM based on formula and SEM based on bootstrap resamples is due to fact that they are both estimates, calculated in two different ways
how can bootstrap resamples be used to calculate a confidence interval?
95% confidence interval from the bootstraps represents the range of values that 95% of the means take
what is bootstrapping with a model fit?
comparing the mean to a specific value is effectively having a very simple model of the world
bootstrapping generalises more easily to more complex models than just the mean
what are the advantages of the bootstrap?
very general method - any type of model can be used and confidence intervals of any of its parameters can be estimated
can also be used to perform hypothesis testing (for one-sample tests)
not based on any assumptions about the data
no tables, no equations (except for the model)
what are other resample approaches?
Jack-knife
Monte-Carlo method
what is the Jack-knife?
similar to bootstrap
rather than “randomly sampling with replacement”, resampling done by “selecting all data except one”
