Lecture 14: Null Hypothesis Testing

Question 1

The alpha level

Answer 1

How low the p-value has to be for us to reject the null hypothesis (alpha)

Question 2

Conventionally what alpha level is used?

Answer 2

0.05

Question 3

The alpha level is usually 1 minus the _______ ____

Answer 3

Confidence level

Question 4

Null hypothesis

Answer 4

There is no effect

Question 5

Alternative hypothesis

Answer 5

There is an effect

Question 6

Typically the research expects/hopes that the null hypotheis is TRUE/FALSE

Answer 6

FALSE! So that the alternative hypothesis is true (we’d want to hope that the drug would work)

Question 7

Before we collect our data we want to set the _____ ______

Answer 7

Alpha level

Question 8

The alpha level defines how much ________ we need for our null hypothesis test

Answer 8

CONFIDENCE!

Question 9

After we get our data, the null hypothesis test is a 3 step process (3)

Answer 9

1. Compute the test statistics and degrees of freedom
2. Compute a p-value from the test statistic and df
3. Compare the p-value to a

Question 10

If p < a then we do we accept or reject the null hypothesis?

Answer 10

REJECT! There IS sufficient evidence of an effect!

Question 11

if p > a do we reject/accept the null hypothesis?

Answer 11

ACCEPT! There ISN’T ENOUGH sufficient evidence of an effect

Question 12

For mean comparisons, the null hypothesis test we typically do is called a _________

Answer 12

T-test

Question 13

There are 2 kinds of 2 tests!

Answer 13

1. Paired (dependent samples) t-tests
2. unpaired (independent samples) t tests

Question 14

If we use a pooled s we call it a ______ t-test

Answer 14

Student

Question 15

If we use an unspooled s we call it a ____ t-test

Answer 15

Welch’s

Question 16

How does the t-test work? (3)

Answer 16

1. Compute test statistics and df
2. Compute a p-value from the test statistic and df
3. Compare the p-value to the alpha level (a)

Question 17

What’s a short term for the test statistic?

Answer 17

T-statistic

Question 18

What’s a t-statistic? How do we compute this?

Answer 18

The sample mean difference OR AKA estimated effect size /estimated standard error

Question 19

What is the df based on?

Answer 19

Sample size

Question 20

For a paired comparison the df =?

Answer 20

Df = n-1 (n being the number of pairs)

Question 21

For unpaired comparisons the df =?

Answer 21

Df kinda = n-2

Question 22

For step two of the t-test it consists of a complex computation we’d never do by hand BUT basically the larger the t (in absolute value) then the smaller the _____

Answer 22

P

Question 23

If the p < a we say that the p-value and the sample mean difference that produced it are ______ __________

Answer 23

Statistically significant

Question 24

Why is the term “significant” critiqued?

Answer 24

It implies importance by even a small difference can be likely to be statistically significant if the sample size is large

Question 25

Saying that p < a is also equivalent to saying that the confidence interval (at the 1-a confidence level) doesn’t include ____

Answer 25

Zero

Question 26

Since 95% confidence intervals don’t contain the actual parameter in 5% of the studies we do how often will a p-value be less than 0.05 when the actual population mean difference is 0?

Answer 26

5% of studies when the population mean difference is zero

Question 27

For a true null hypothesis the p-value is _______ distributed between ___ and ____

Answer 27

Uniformly distrusted between 0 and 1 All values are equally likely

Question 28

For a normal distribution, the probably of p < 0.05 is ____% and so is the probability of p > 0.95 and so is the probability of p being in between 0.65 and 0.7

Answer 28

5%

Question 29

When the null hypothesis is false the chance of getting a p-value less than 0.05 are ______ which means that the skew is _____

Answer 29

Increased! Positive!

Question 30

What two things, if they were bigger, would skew the p-value distribution to the right? (2)

Answer 30

1. Standardized effect 2. Sample size

Question 31

For a true null hypothesis, the p-value is uniformly distributed between ___ and ___

Answer 31

0 and 1 = no value is more likely than another regardless of sample size

Question 32

For a false null hypothesis, the ___ p-values are more likely

Answer 32

low p-values = bigger the effect, bigger the sample size, lower p-value is likely to be

Question 33

Rejecting a true null hypothesis is which error?

Answer 33

Type 1 error (or aka type 1 error rate)

Question 34

Why not set the alpha level to be very low instead of at 0.05?

Answer 34

Then we would rarely reject the null hypothesis even when it's false...

Question 35

Not rejecting the null hypothesis when it's false is which error?

Answer 35

Type 2 error

Question 36

So a type 1 error is finding an effect that ____ there in the population

Answer 36

isn't realy there

Question 37

A type 2 error is basically ______ to find an effect that is really _____ in the population

Answer 37

failing to find an effect that is really there in the population

Question 38

The lower we set the alpha level the ____ the type 2 error rate if the null hypothesis is false

Answer 38

Higher

Question 39

How can we use an alpha level that's sufficiently low without risking an excessively high type 2 error rate?

Answer 39

LARGE SAMPLE!!!

Question 40

If we use a large sample we can set a ___ alpha level and still have a good chance of getting statistical significance is the null hypothesis is false

Answer 40

Low!

Question 41

So what protects us from type 1 errors if the null hypothesis is true?

Answer 41

LOW ALPHA LEVELS!

Question 42

So what protects us from type 2 errors if the null hypothesis is false?

Answer 42

LARGE SAMPLE SIZE!

Question 43

Statistical power (and the formula)

Answer 43

The probability of getting statistical significance when the null hypothesis is false 1 - Type 2 error rate

Question 44

What makes statistical power high?

Answer 44

1. Large standardized effect size 2. Large sample size