Topic 8: Multiplicity

Question 1

What is multiplicity

Answer 1

Multiplicity is the term to describe the increased risk of false positive (type 1) conclusions that arises when multiple statistical tests are carried out on a data set.

Question 2

Why does multiplicity occur?

Answer 2

Doing multiple statistical tests for multiple hypotheses means you have multiple effect estimates and multiple p values. Each test is performed with a small chance of error and you look across tests to make a conclusion, which increases the type one error rate.

Question 3

What is family wise error rate

Answer 3

Term to describe the increased error across p values associated with multiple tests contributing to your conclusion.

Question 4

What is the family wise type 1 error

Answer 4

The chance of a false positive conclusion in the trial as a whole

Question 5

What is required of the family wise type 1 error rate in order for trial outcomes to influence practice

Answer 5

It needs to be controlled.

Question 6

What is the p-value

Answer 6

The p-value arising from a statistical test represents the probability that the observed difference is due to chance.

Question 7

For two tests, each with a 95% prob of no error, what is the overall chance of error

Answer 7

0.95*0.95 = 0.9025. So overall chance of error is 9.75% (increased from 5%)

Question 8

For multiplicity, is it the number of p-values contributing to the conclusion or the number of p-values calculated

Answer 8

p-values contributing to your conclusion

Question 9

Is multiplicity an issue when you have multiple outcomes

Answer 9

Not if there is still only one hypothesis test contributing to the conclusion.

Question 10

What is data-dredging

Answer 10

A lot of tests carried out in order to try and find something important

Question 11

How can we avoid accusations of data dregdging

Answer 11

Don’t do any unplanned statistical tests. Do no more or less tests than those that have been planned in the trial protocol.

Question 12

Which tests in a trial need to be reported

Answer 12

All of them, regardless of whether they have positive outcomes or not.

Question 13

Do secondary analysis tests need to be reported too even if they don’t feed into the main conclusio

Answer 13

Yes

Question 14

What can only reporting positive outcomes lead to

Answer 14

Reporting bias

Question 15

How does reporting bias relate to multiplicity

Answer 15

It may not be clear if the results reported relate to all conducted tests, so its unclear how much of what has been reported has occured by chance of just excessive testing

Question 16

When is multiplicity an issue in a trial with multiple outcomes

Answer 16

If only one or the other outcome is required to significant, as opposed to both needing to be significant.

Question 17

When is multiplicity NOT an issue in a trial with multiple outcomes

Answer 17

When both are required to be significant, or if they are ordered so that the second is only tested if the first is significant. This means that the conclusion is only based on one result: effectiveness on both outcomes. Multiplicity is not an issue since the overall chance of error is not inflated.

Question 18

What is a hierarchical drug trial

Answer 18

When you’re comparing two doses of the same drug, but the lower dose would only be considered if the higher dose was found to be effective.

Question 19

What are two ways to overcome multiplicity for having multiple outcomes

Answer 19

Hierarchical strategy, you make conclusions in a set ordered way rather than conducting statistical tests for all hypothesis. Could also use a composite outcome.

Question 20

What is a composite outcome

Answer 20

Multiple end-points combined

Question 21

What is the issue with using composite outcomes to avoid multiplicity

Answer 21

They may be hard to interpret in terms of what the treatment differences are, what it relates to, and what the main driver of the treatment difference is.

Question 22

Why is using multiple treatment arms beneficial

Answer 22

You can answer two questions with only 50% more patients by having two interventions and a control - efficient.

Question 23

When is multiplicity an issue in trials with multiple treatment arms

Answer 23

When the conclusions simultaneously make reference to more than one treatment comparison - the chance of false positive claim is increased

Question 24

In what kind of multiple treatment arm trial is multiplicity particularly an issue

Answer 24

When arms are added and removed throughout the trial, so you don’t know at the start, which treatment arms will be used.

Question 25

How can you get around multiplicity in a multiple arm trial

Answer 25

Use a hierarchical design

Question 26

Give an example of a time where you might repeat analysis at different time points

Answer 26

A trial that has an interim analysis and then a final analysis.

Question 27

Give 3 reasons you might stop a trial early

Answer 27

Overwhelming evidence of improvement in efficacy. A lack of efficacy. Futility - a small chance the trial will show efficacy if you continue.

Question 28

What is an adaptive trial

Answer 28

When you can make changes throughout the trial, give it interim looks to make changes and check the original assumptions that were used to power the trial.

Question 29

In what phases are adaptive trials more common

Answer 29

Earlier phase trials

Question 30

Why is multiplicity an issue with repeating analysis at different time points/interim analyses.

Answer 30

The more looks at the data, the more tests, and the greater the chance of making a type 1 error.

Question 31

What does testing for interactions do to the power of the test

Answer 31

Decreases the power of each test - need more participants for equivalent power.

Question 32

Why do subgroup treatment estimates have wider confidence intervals than overall treatment effect

Answer 32

you normally need more patients to detect an interaction term than you would for a standard trial, and the study has been powered for the overall treatment effect.

Question 33

What did subgroup analyses use before using treatment interactions for analysis

Answer 33

Do a set of analyses on each subgroup and compare p-tests from the same test on different populations.

Question 34

How do you avoid claims of data dredging when doing subgroup analyses

Answer 34

Pre-specify the groups used in subgroup analyses and have them be based on clinical rationale - avoids suspicion that you have done lots of tests and selected the strongest one.

Question 35

What does it mean that subgroup analyses are considered hypothesis generating

Answer 35

May be used to direct subsequent research

Question 36

How is multiplicity corrected

Answer 36

Splitting the type 1 error rate between analyses so the overall type 1 error doesn't exceed the planned level.

Question 37

What are the two ways of splitting type 1 error to control for multiplicity

Answer 37

Equally so that each test has an equal chance of significance, or so that less of the error is used on some analyses than others.

Question 38

What kind of trials usually split the alpha equally

Answer 38

Multiple endpoints, ,multiple treatments and subgroup analyses

Question 39

Which kind of trials usually split alpha unequally

Answer 39

Multiple time points. It is usually useful to use less error on interim analysis than final analysis - also means that only extremely certain results at interim analysis will stop a trial for efficacy.

Question 40

True/False: Correcting multiplicity has no effect on the power

Answer 40

False

Question 41

Give 3 corrections to control for type 1 error, when the hypotheses are of equal importance

Answer 41

Bonferroni, Hom, Hochberg

Question 42

What does the bonferroni correction do

Answer 42

split error equally over independent tests

Question 43

What is the disadvantage of the Bonferroni correction

Answer 43

Overly conservative.

Question 44

What does the Holm correction do

Answer 44

Sequentially rejective multiple test procedure with a stepwise nature based on bonferroni correction. Order p values smallest to largest. Compare (is p less than it) the most significant hypothesis against alpha/n - which is used in bonferroni - then compare second most significant to (alpha/n-1). Third significant to (alpha/ n-2) and so on. As soon as we fail to reject the null, stop and reject all remaining null hypotheses.

Question 45

True/False: The Holm correction is uniformly more powerful than the bonferroni correction

Answer 45

True

Question 46

Which of the Holm and Hocheberg is step up procedure and which is step dow

Answer 46

Holm is step up, Hochberg is step down.

Question 47

How does the Hochberg Correction work

Answer 47

order p values smallest to largest. Take the least significant (largest p) and compare to alpha/n. Take the 2nd largest p and compare to alpha/n-i+1. When p < specific alpha value, comparison stops and you conclude the hypothesis for that p and all other null hypothesis for p less than it are rejected.