Family Wise Errors - L4

Question 1

What does the F value tell us?

Answer 1

An F-value tells us whether any differences exist between different levels of the independent variable

• It doesn’t tell us about which levels differ

Question 2

What is the F - Value also referred to as?

Answer 2

the omnibus* F

omnibus is latin for “for all”

Question 3

How do we do analysis with a nonsignificant omnibus?

Answer 3

With a nonsignificant omnibus F (i.e., F < Fa ), we are not confident that differences exist between levels; we can stop the analysis here

Question 3

How do we do analysis with a significant omnibus?

Answer 3

With a significant omnibus F, further analysis is required

Question 4

What is the planned type of analytical comparison?

Answer 4

Planned (a priori) comparisons
* These are comparisons about which we had a direct prediction prior to performing the experiment. These
are effects we expect to find a priori.

Question 5

What is the post-hoc type of analytical comparison?

Answer 5

• Post-hoc (a posteriori) comparisons
• If we had no principled reason to expect particular differences prior to performing the experiment, we can
  perform post hoc comparisons. These test effects about which we had no prior expectations.

Question 6

What is an example of a planned (a priori) comparison?

Answer 6

To extend the lecture style example, say we know that:
* Lecture Style 1 uses worksheets AND lectures
* Lecture Style 2 used lectures ONLY
* Lecture Style 3 used worksheets ONLY
* From previous research, we expect that both together will be more effective
than either alone. Thus, we expect a priori:

U1 > U2 and U1 > U3

Question 7

How do we represent a contrast (whereas the test whether the data supports an expected difference)?

Answer 7

The greek letter PSI (trident looking)

PSI = U1 - U2

Question 8

What do we use to test specific contrasts?

Answer 8

• In general, we use a design matrix (C ) to test specific contrasts. Each element
  of C is multiplied by one of the means (here M is a series of means):
  psi = CM = c1u1 + c2u2 + c3u3

Question 9

How do we test a planned comparison?

Answer 9

To test whether our planned comparison is significant, we’ll use the F-ratio approach again
* First, we need to compute the sum of squares
* As for the omnibus F, we next compute the mean squared error:
Finally, to obtain an F-ratio, we use the within-group MSE*

for calculations look at L4 slide 16 and 17

Question 10

What is s?

Answer 10

� : The number of subjects in each group

Question 11

What is C?

Answer 11

�: The coefficients from the design matrix

Question 12

What is a familywise error?

Answer 12

if we perform a sequence of statistical tests (including
comparisons) on our data, we accumulate Type I error

• This accumulated error is called family-wise error (often abbreviated FWE)

Question 13

What is an example of a family wise error?

Answer 13

• fMRI data is often analysed voxel-wise*
• If we have 64x64x64 voxels, this means we perform ~260K individual comparisons
• A threshold of � < 0.05 means that we expect a false positive 5% of the time
• Thus, even with random data (H0 is true) we expect ~13K (5% of 260K) significant voxels completely by chance

Question 14

What is error on individual tests called?

Answer 14

per comparison error

Question 15

What is an error on the whole set of tests called?

Answer 15

Error on the whole set of tests is called family-wise error

Question 16

What is the family wise error rate?

Answer 16

is the probability of making at least one
Type I error from comparisons

LOOK AT The maths slide 23

Question 17

When do we not have to correct for FWE?

Answer 17

, if our comparisons are planned (i.e., we have a principled a priori
expectation), we do not need to correct for FWE

Question 18

Why don’t we have to correct for FWE with a priori?

Answer 18

Because the probability of making an error is modified by our prior expectation that H0 is
false (an informal application of Bayes’ theorem)

Question 19

When do we have to correct for FWE?

Answer 19

if we haven’t planned our comparisons (i.e., we are performing post hoc
comparisons), we need to adjust our error rates to account for FWE

Question 20

What are the three major types of post hoc tests?

Answer 20

• Bonferroni correction
• Scheffé test
• Tukey’s Honest Significant Difference test

Question 21

What is Bonferroni correction?

Answer 21

• Bonferroni is the most conservative FWE correction:
• This completely accounts for FWE and ensures the FWER is below the desired level (e.g., 5%) 26
• However, with very many comparisons, it is
  overly conservative; i.e., ensures a very high
  false negative rate (failure to reject H0 even
  when it is really false, and thus failure to
  discover true effects)

Question 22

When is a bonferroni correction usually performed?

Answer 22

on t-tests,

Question 23

What is MSr?

Answer 23

the within-group mean squared error

Question 24

What is the Scheffe test?

Answer 24

• This is less conservative than Bonferroni correction

The Scheffé Test adjusts the critical F-value, using the number of treatment levels, a
(rather than total comparisons)

where a is the number of treatment levels, and F(df1, df2)a is the critical F value (omnibus or contrast) at threshold a

Question 25

What is the Tukey HSD test?

Answer 25

The Tukey Honestly Significant Difference (HSD) test uses a different statistical distribution than the F distribution to limit Type I error

• This is called the studentised range distribution Q, and can be used to estimate the probability of observing a range (max-min) in DV values for a given
  sample
• Like the F-test, the Tukey test uses a critical value, qa, df, a to determine whether
  a test is significant

Question 26

What is the the studentised range distribution Q?

Answer 26

a different statistical distribution than the F distribution to limit Type I error, can be used to
estimate the probability of observing a range (max-min) in DV values for a given sample ( Tukey test)

Question 27

Post hoc tests conclusions

Answer 27

Post hoc tests are conservative, meaning they reduce the chance of Type I
errors (false positives) by greatly increasing the chance of Type II errors (false
negatives)
* Only very robust effects will be detected
* Null results are not easily interpreted
* Many alternative tests have been proposed, with different trade-offs
* Many post hoc tests are available with statistical software packages such as SPSS,
Jamovi, JASP, or R

Question 28

How do you run an anova on SPSS?

Answer 28

• A one-way ANOVA can be run using:
• Click Contrasts to define planned comparisons
• Click Options to output descriptive and test the homogeneity of variance
• The Output Window shows you your results: