analysing experimental studies Flashcards
how do we know if an interaction is significant?
look at the p-value of the interaction term - if it is significant, the p-value should be smaller than the alpha level
how do we know if one variable is significant?
F-stats tell us the overall significance of a model so if we only include on variable in the model it tells us the significance of the variance in our outcome based on the one variable included.
incremental F-test
a regular f-test is an incremental test of our model against a null/’empty’ model (just the intercept where all β = 0)
the incremental f-test evaluates the statistical significance of the improvement of variance explained in an outcome with the addition of further predictors - based on the difference in f-values between the two models
- the model with more predictors is called model 1 or the ‘full model’
- the model with less is called model 0 or ‘restricted
model’
incremental F-test equation
F (dfR - dfF), dfF = [ (SSRr - SSRf) / (dfR - dfF) ] / SSRf - dfF
where:
dfR = df of restricted model
dfF = df of full model
SSRr = residual sums of squares of restricted model
SSRf = residual sums of squares of full model
anova() function in R
used to perform an incremental f-test
provides the following results:
- residual df for both model
- SSresidual for both models
- difference in their dfs
- difference in their SS
- incremental F-test
- p-value for the significance of the test (needs to be smaller than our alpha value to be significant)
Nested vs non-nested models
nested = the predictors in one model are a subset of the predictors in the other (models must also be computed on the same data)
- can use incremental F-test
non-nested = there are unique variables in both of the models so there is no way of making them equivalent
- have to use AIC or BIC (smaller/more negative values indicate better fitting models)
AIC and BIC
both contain a primary correction - meaning they penalise models for being too complex. BIC is harsher in this aspect
AIC = n * ln( SSresidual / n ) + 2k
BIC = n * ln( SSresidual / n) + ( k* ln(n) )
- where ln = natural log function
AIC and BIC only make sense when used for model comparisons.
For BIC a difference of 10 can be used as a rule of thumb to suggest that one model is better than another (AIC does not do this) - we want the model with the smaller AIC or BIC value
why do we need constraints/a reference group?
we want a model that represents our data/observations but all we ‘know’ is what group an observation belongs to ( µ = β0 + βi) this creates problems as we don’t want to estimate too many variables (β0 means we will have one more parameter to estimate than group means)
- constraints fix this by making one of the group means β0
what is effects coding?
also called sum to zero coding
we compare groups to the grand mean (the mean of all observations)
we reduce the number of groups we have to
βj = 0 = all β values must sum to 0
typically used in experimental settings when there isn’t always an obvious reference group
how does the dummy coding constraint help?
example = 3 treatment variables
before dummy coding:
µA = β0 + βA
µB = β0 + βB
µC = β0 + βC
this is 4 βs for 3 group means
how dummy coding fixes this
µA = β0
µB = β0 + βB
µC = β0 + βC
how does the effects coding constraint help?
example = 3 treatment variables
before effects coding:
µA = µ + βA
µB = µ + βB
µC = µ + βC
where µ = grand mean
- this is still 4 things to estimate for 3 group means
how effects coding fixes this (sum to 0)
µA = β0
µB = β0 + βB
µC = β0 - (βA + βB)
effects coding results - general interpretation
β0 = µ = grand mean = reference group
- mean of predictors / k
- e.g. (µA + µB + µC) / 3 = µ
βj = difference between the coded group and the grand mean
- βj = µj - µ
steps in effects coding:
- create k-1 variables
- for all observations in the focal group, assign 1
- for all observations in the reference group, assign -1 (must sum to 0)
- for all other groups, assign 0
manual constraints testing:
allow us to test a wide variety of constraints so long as they can be written:
- as a linear combination of a population mean
- the associated weights (coefficients) sum to zero
manual constraints chunk variables together and compare them to other chunks to test if the mean of the chunks are significantly different
rules for assigning weight constraints:
- weights range between 1 and -1
- the group(s) in one chunk get positive weights and the other chunk gets negative weights
- the sum of the weights of comparison must be 0
- if a group is not involved in a comparison its weight is 0
- the weights assigned to the groups in the comparison = 1/number of groups e.g. if there is two groups in the positive chunk they will both be given a weighting of 1/2
- restrict yourself to running k-1 comparisons
- each contrast can only compare 2 chunks of variance
- once a group is singled out, it can not enter the other contrasts
- check if the contrasts are orthogonal
