Statistical Significance

Question 1

What is Chance/Random Error?

Answer 1

Error in measurement caused by factors which vary from one measurement to another

Question 2

What are the Differences in Measurement Due to?

Answer 2

Random Error

Question 3

What are the 2 Types of Random Error?

Answer 3

Sampling Error and Measurement Error

Question 4

Types of Random Error: What is Sampling Error?

Answer 4

These are differences between the samples used in each study. Not everybody is the same, and there may be differences in the way people respond as well as visible characteristics (age, etc.)

Question 5

Types of Random Error: What is Measurement Error?

Answer 5

Measuring the same parameter multiple times will not give the same result every time (e.g. weight will be slightly different each day)

Question 6

What Factors Influence Random Error?

Answer 6

Random error is influenced by the degree of variability (heterogeneity) between individuals and the sample size (the fewer people we look at, the less certain we are as to whether the observations we made are actually true)

Question 7

What is Statistical Significance?

Answer 7

Measures how likely any apparent differences in outcome between treatment and control groups are real and not due to random error/chance

Question 8

What is a Confidence Interval?

Answer 8

A confidence interval is the range in which you are confident that the true value lies

Question 9

What Percentage CI do we use?

Answer 9

95% CIs are normally used, indicating we are 95% sure that the true value lies in the specified range. In other words, if we were to repeat the study 100 times, 95 times out of those 100 the value will lie in that range

Question 10

What is better, Narrow or Wide CI?

Answer 10

Narrow CIs. They give more precise estimate for true effect

Question 11

How can CIs be narrowed?

Answer 11

Increasing sample size

Decreasing heterogeneity (e.g. give just to men, give drug just to age of X-Y)

Question 12

For a treatment to have a real difference (vs. its comparator), what would we like to be sure?

Answer 12

For a treatment to have a real difference (vs. its comparator), we would like to be sure (95% confident) that the true RR does not cross the ‘line of no effect’ (RR, OR or HR = 1) (ARR, RRR = 0)

Question 13

What is the Null Hypothesis?

Answer 13

In hypothesis testing, we make a statement called the null hypothesis, which we are trying to reject. The null hypothesis is usually the statement that there is no difference between the new and current treatments, so therefore we want to reject that to show that our new drug works. The study should be designed to find evidence that it is highly unlikely that there is no difference.

Question 14

What is P-Value?

Answer 14

The p-value is essentially a level of confidence. If the calculated p-value is less than 0.05 (5%), then we are 95% sure that differences between treatments are real and not down to random error. There is still a 5% chance that the difference that it is due to chance only

Question 15

If the p-value is >0.05, what does this indicate?

Answer 15

Indicates the result is not statistically significant. We are not confident that the differences are real, meaning there is a higher chance that any differences are due to random error

Question 16

How can P Values be Reduced?

Answer 16

Highly efficacious treatments, large sample sizes or low heterogeneity will reduce p-values and enhance statistical significance

Question 17

Is it possible for a result to be statistically significant but not clinically significant?

Answer 17

It is possible for a result to be statistically significant and for the differences to be real, but not clinically significant, meaning these differences don’t really matter in the real world and don’t justify a change in clinical practice

Question 18

For a result to be clinically significant, what must it do?

Answer 18

For a result to be clinically significant, it must be greater than the minimal clinically significant difference

Question 19

When is a result clinically and statistically significant?

Answer 19

This occurs when the CI does not cross 1 AND the CI exceeds the minimal clinically important difference (i.e. does not cross this line)

Question 20

Why is a result statistically significant but not clinically significant?

Answer 20

Usually due to large sample size

Question 21

Why is a result clinically significant but not statistically significant?

Answer 21

Usually due to small sample size

Question 22

What are Power Calculations?

Answer 22

Inform the sample size required for a trial

Powering studies appropriately (i.e. picking the right size) ensures the results are significant but not too costly or unethical

Question 23

Why don’t we want a very small sample size?

Answer 23

Results may not be statistically significant

Question 24

Why don’t we want a very small large size?

Answer 24

Would not be ethical and would be expensive

Question 25

What is a Superiority Trial?

Answer 25

Attempt to show that one treatment is better than another

Question 26

What is a Non-Inferiority Trial?

Answer 26

No worse than - want to show drug is working just as well as existing treatment

Question 27

What is an Equivalence Trial?

Answer 27

Not worse than, but not better than either