Lecture 14: Critical Thinking About Statistical Inferences

Question 1

Explain the 3 steps of the Fisher approach to significance testing

Answer 1

1. Formulate H0
2. Report the exact level of significance (p-value), without further discussion about accepting or rejecting hypotheses
3. Only do this if you know almost nothing about the subject (devil’s advocate)

Question 2

Explain the 3 steps of hypothesis testing according to the Neyman-Pearson approach

Answer 2

1. Formulate two statistical hypotheses, determine alpha, beta and N for the experiment
2. If the data falls in the H1 rejection region, assume H2 (this does not mean that you believe H2, only that you behave as if H2 is true)
3. Only use this procedure if there is a clear disjunction and if a cost-benefit assessment is possible

Question 3

Explain the 3 steps of the null ritual (what we do now)

Answer 3

1. Set up a statistical null hypothesis of “no mean difference” or “zero correlation”. Don’t specify predictions of your research hypothesis or of any alternative substantive hypotheses
2. Use 5% as a convention for rejecting the null. If significant, accept your research hypothesis
3. Always perform this procedure

Question 4

What are the differences between fisher and neyman-Pearson

Answer 4

Fisher did not talk about accepting/rejecting hypotheses, neyman-Pearson did
Fishers approach works only if you know almost nothing about the subject, neyman-Pearson only works if there is a clear disjunction and if a cost-benefit assessment is possible

Question 5

What are 4 fallacies of statistical inference

Answer 5

1. P-values equal the probability that the (null)hypothesis is true
2. Alpha equals the probability of making an error
3. Failing to reject H0 is evidence for H0
4. Power is irrelevant when results are significant

Question 6

What is subjective probability

Answer 6

Probability is the degree of belief that something is the case in the world, a degree of uncertainty

Question 7

What is objective probability

Answer 7

Probability is the extent to which something IS the case in the world (more about facts than about beliefs), it expresses the relative frequency in the long term

Question 8

Explain the two points of the first fallacy

Answer 8

Statements in which alpha, power and p-values occur relate to frequentist or objective probabilities.
Statements in which alpha, power and p-values occur relate to conditional probabilities.

Question 9

Explain how Bayes differs from frequentist probabilities

Answer 9

1. It talks about subjective probabilities (degrees of belief)
2. You can quantify the probability of a certain hypothesis

Question 10

Explain the term credibility in context of Bayes

Answer 10

It is the degree of belief that a certain possibility (hypothesis) is true

Question 11

What is the likelihood

Answer 11

How predictive the different hypotheses are of the data; how likely are the data given a certain hypothesis

Question 12

What component of the Bayes theorem formula P(A|B) = P(B|A)*P(A)/P(B) is the likelihood

Answer 12

P(B|A)

Question 13

What component of the Bayes theorem formula P(A|B) = P(B|A)*P(A)/P(B) is the prior

Answer 13

P(A)

Question 14

What component of the Bayes theorem formula P(A|B) = P(B|A)*P(A)/P(B) is the marginal probability

Answer 14

P(B)

Question 15

What is the prior

Answer 15

The degree to which we believe something to be the case before we start doing analysis (before we have seen the data)

Question 16

What is the marginal likelihood

Answer 16

How likely is the probability of the data irrespective of which hypothesis we are assuming/looking at (general prediction —> how much more predictive of the data is this assumption over any other assumptions we could make

Question 17

What does it mean when the prior distribution is flat? What does it mean for the degree of belief when the distribution gets more narrow?

Answer 17

When the distribution is flat, this means that the we believe that the probabilities of all our hypotheses are equally likely
When the distribution gets more narrow, that means that we have a stronger degree of belief that that is the population parameter

Question 18

What is the Bayes factor

Answer 18

It reflects a likelihood ratio, so how much more probable is the data given one parameter compared to the other —> this enables you to quantify evidence for or against a hypothesis which is useful when testing hypothesis

Question 19

Why is the power still relevant if we reject the H0 (when results are significant)

Answer 19

Because the rejection of the H0 is less informative when the power is low (higher chance of being wrong in rejecting the H0)