Lecture 18: Literature

Question 1

Familewise Type 1 Error rate

Answer 1

at least one Type 1 error in a set of tests

Question 2

The more hypotheses in the family, the more the familywise Type 1 error rate may be _______

Answer 2

inflated

Question 3

The exact amount of inflation of the familywise Type 1 error rate depends on the amount of _______/________ between the tests

Answer 3

dependence/correlation

Question 4

Multiplicity 2 other names

Answer 4

1. Multiple comparisons
2. Multiple testing

Question 5

Multiplicity

Answer 5

The problem of Type 1 errors being more frequent when there are multiple tests conducted

Question 5

How to counteract the problem of multiplicity

Answer 5

The Bonferroni procedure

Question 6

The Bonferroni procedure is implemented in 2 ways:

Answer 6

1. Adjusted alpha level: divide a-level by no. of tests
2. Adjusted p-value: multiply p-value by no. of tests

Question 7

Bonferroni procedure controls the _______ _____ _ _____ ____ but also reduces _______ _____ so testing multple hypotheses requires ______ ______ _____ to compensate

Answer 7

1. familywise Type 1 error rate
2. statistical power (increases type 2 error rate)
3. large sample size

Question 8

Because adjustment for multiplicity reduces statistical power, some researchers try to get out of adjusting by (2)

Answer 8

1. making excuses for not adjusting
2. Covering up how many tests were conducted! (p-hacking, data snooping)

Question 9

p-hacking

Answer 9

dishonest practice of conducting as many tests as necessary in order to get statistical significance without reporting all the tests that were conducted

Question 10

Data snooping

Answer 10

dishonest practice where the researcher looks at the data before deciding which test looks most promising (informal testing)

Question 11

How to handle multiplicity responsibly (4)

Answer 11

1. State hypotheses before the study begins
2. Report al tests conducted
3. Adjust for multiplicity and report method used

Question 12

It’s ok to come up with a hypotheses after looking at the data but they should be tested in a ___ ____ and with a ____ ___

Answer 12

new study new sample

Question 13

Can CIs be adjusted using the Bonferroni procedure? If so, how?

Answer 13

Yes!

You compute each CI at a confidence level equal to 1 minus the Bonferroni adjusted alpha level

eg, for 2 CIs each CI is computed at the 1 - 0.05/2 = 97.5% level

Question 14

Will we ever know what proportion of studies have a true null hypotheses?

Answer 14

No

Question 15

If all studies in a paper you read had true null hypotheses what does that mean?

Answer 15

100% of the statisticlly significant results are type 1 errors

Question 16

if none of the studies in a paper you read had true null hypotheses what does that mean?

Answer 16

then 0% of the statistically significant results are Type I errors

Question 17

File drawer problem

Answer 17

studies that didn’t produce statistically significant results don’t get reported in the literature