Chapter 6 Beast of Bias

Question 1

Bootstrap

Answer 1

a technique from which the sampling distribution of a statistic is estimated by taking repeated samples (with replacement) from the data set (in effect, treating the data as a population from which smaller samples are taken). The statistic of interest (e.g., the mean, or b coefficient) is calculated for each sample, from which the sampling distribution of the statistic is estimated. The standard error of the statistic is estimated as the standard deviation of the sampling distribution created from the bootstrap samples. From this, confidence intervals and significance tests can be computed.

Question 2

Contaminated normal distribution

Answer 2

a normal-looking distribution that is contaminated by a small proportion of scores from a different distribution. These distributions are not normal and have too many scores in the tails (i.e., at the extremes). The effect of these heavy tails is to inflate the estimate of the population variance. This, in turn, makes significance tests lack power.

Question 3

Hartley’s Fmax

Answer 3

also known as the variance ratio, is the ratio of the variances between the group with the biggest variance and the group with the smallest variance. This ratio is compared to critical values in a table published by Hartley as a test of homogeneity of variance. Some general rules are that with sample sizes (n) of 10 per group, an Fmax less than 10 is more or less always going to be non-significant, with 15–20 per group the ratio needs to be less than about 5, and with samples of 30–60 the ratio should be below about 2 or 3.

Question 4

Heterogeneity of variance

Answer 4

the opposite of homogeneity of variance. This term means that the variance of one variable varies (i.e., is different) across levels of another variable.

Question 5

Heteroscedasticity

Answer 5

the opposite of homoscedasticity. This occurs when the residuals at each level of the predictor variables(s) have unequal variances. Put another way, at each point along any predictor variable, the spread of residuals is different.

Question 6

Homogeneity of variance

Answer 6

the assumption that the variance of one variable is stable (i.e., relatively similar) at all levels of another variable.

Question 7

Homoscedasticity

Answer 7

an assumption in regression analysis that the residuals at each level of the predictor variable(s) have similar variances. Put another way, at each point along any predictor variable, the spread of residuals should be fairly constant.

Question 8

Independence

Answer 8

the assumption that one data point does not influence another. When data come from people, it basically means that the behaviour of one person does not influence the behaviour of another.

Question 9

Kolmogorov–Smirnov test

Answer 9

a test of whether a distribution of scores is significantly different from a normal distribution. A significant value indicates a deviation from normality, but this test is notoriously affected by large samples in which small deviations from normality yield significant results.

Question 10

Levene’s test

Answer 10

tests the hypothesis that the variances in different groups are equal (i.e., the difference between the variances is zero). It basically does a one-way ANOVA on the deviations (i.e., the absolute value of the difference between each score and the mean of its group). A significant result indicates that the variances are significantly different – therefore, the assumption of homogeneity of variances has been violated. When sample sizes are large, small differences in group variances can produce a significant Levene’s test. I do not recommend using this test – instead interpret statistics that have been adjusted for the degree of heterogeneity in variances.

Question 11

M-estimator

Answer 11

a robust measure of location. One example is the median. In some cases it is a measure of location computed after outliers have been removed; unlike a trimmed mean, the amount of trimming used to remove outliers is determined empirically.

Question 12

Mixed normal distribution

Answer 12

a normal-looking distribution that is contaminated by a small proportion of scores from a different distribution. These distributions are not normal and have too many scores in the tails (i.e., at the extremes). The effect of these heavy tails is to inflate the estimate of the population variance. This, in turn, makes significance tests lack power.

Question 13

Outlier

Answer 13

an observation or observations very different from most others. Outliers bias statistics (e.g., the mean) and their standard errors and confidence intervals.

Question 14

P-P plot

Answer 14

short for ‘probability–probability plot’. A graph plotting the cumulative probability of a variable against the cumulative probability of a particular distribution (often a normal distribution). Like a Q-Q plot, if values fall on the diagonal of the plot then the variable shares the same distribution as the one specified. Deviations from the diagonal show deviations from the distribution of interest.

Question 15

Parametric test

Answer 15

a test that requires data from one of the large catalogue of distributions that statisticians have described. Normally this term is used for parametric tests based on the normal distribution, which require four basic assumptions that must be met for the test to be accurate: a normally distributed sampling distribution (see normal distribution), homogeneity of variance, interval or ratio data, and independence.

Question 16

Q-Q plot

Answer 16

short for ‘quantile–quantile plot’. A graph plotting the quantiles of a variable against the quantiles of a particular distribution (often a normal distribution). Like a P-P plot, if values fall on the diagonal of the plot then the variable shares the same distribution as the one specified. Deviations from the diagonal show deviations from the distribution of interest.

Question 17

Robust test

Answer 17

a term applied to a family of procedures to estimate statistics that are reliable even when the normal assumptions of the statistic are not met.

Question 18

Shapiro–Wilk test

Answer 18

a test of whether a distribution of scores is significantly different from a normal distribution. A significant value indicates a deviation from normality, but this test is notoriously affected by large samples in which small deviations from normality yield significant results.

Question 19

Transformation

Answer 19

the process of applying a mathematical function to all observations in a data set, usually to correct some distributional abnormality such as skew or kurtosis.

Question 20

Trimmed mean

Answer 20

a statistic used in many robust tests. It is a mean calculated using trimmed data. For example, a 20% trimmed mean is a mean calculated after the top and bottom 20% of ordered scores have been removed. Imagine we had 20 scores representing the annual income of students (in thousands), rounded to the nearest thousand: 0, 1, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 40. The mean income is 5 (£5000), which is biased by an outlier. A 10% trimmed mean will remove 10% of scores from the top and bottom of ordered scores before the mean is calculated. With 20 scores, removing 10% of scores involves removing the top and bottom two scores. This gives us: 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, the mean of which is 3.44. The mean depends on a symmetrical distribution to be accurate, but a trimmed mean produces accurate results even when the distribution is not symmetrical. There are more complex examples of robust methods such as the bootstrap.

Question 21

Variance ratio

Answer 21

also known as Hartley’s Fmax, is the ratio of the variances between the group with the biggest variance and the group with the smallest variance. This ratio is compared to critical values in a table published by Hartley as a test of homogeneity of variance. Some general rules are that with sample sizes (n) of 10 per group, an Fmax less than 10 is more or less always going to be non-significant, with 15–20 per group the ratio needs to be less than about 5, and with samples of 30–60 the ratio should be below about 2 or 3.

Question 22

Weighted least squares

Answer 22

a method of regression in which the parameters of the model are estimated using the method of least squares but observations are weighted by some other variable. Often they are weighted by the inverse of their variance to combat heteroscedasticity.