Hypothesis test

Question 1

Determine H & H0.

Are these outcomes equally likely?

Answer 1

H: are the counts /frq. of each category as expected?
H0: the probability of each group is exactly equal (or equal)

Question 2

Determine H & H0.

are groups different from each other?

Answer 2

H: are the means of X different in the two groups
H0: the mean of Xa equals the mean of Xb

Question 3

Determine H & H0.

did the outcome change?

Answer 3

H:did the mean of outcome X change across the two measurements?
H0: the mean of X1 equals the mean of X2

Question 4

Determine H & H0.

are these two outcomes correlated?

Answer 4

H: are they the same?
H0: X and Y are uncorrelated (orthogonal)

Question 5

Determine H & H0.

are these variables related

Answer 5

H: is there a linear relationship between X and Y
H0: the coefficient of alpha in an OLS regression of Y on X (and Z…) is zero

Question 6

Procedure for a statistical test

Answer 6

1-based on theory + H0 there is a stat of interest that you can calculate with sample
2-use theory to derive distribution of expected values (under H0). some assumption must be made
3- calculate the statistic actual value given sample
4-state likelihood of answer from sampling method, given that the H0 is true, if unlikely = reject H0

Question 7

What is the null hypothesis (H0)?

Answer 7

-what you are trying to disprove
- strong results, eliminate the possibility of the H0
- no difference, no change, small difference, no effect

Question 8

reject the null if ( statistical significance)

Answer 8

p-value is small or near zero

Question 9

type I error for binary decisions

Answer 9

rejecting the null hypothesis when it is true ( false positive)

Question 10

type ii error

Answer 10

failing to reject the null hypothesis when it is false (false negative)

Question 11

the probability of a type I error is determined by ____

Answer 11

significance level
- with a 99% significance threshold, type I error is less likely than with a 95% confidence

Question 12

the probability of a type ii can be computed for a particular test statistic (i.e. H0), if given______

Answer 12

population distribution (parameterization, mean and SD)
sample size (N) and alpha ( chance of type I error)

Question 13

hypothesis test power

Answer 13

1 - ( the probability of a type ii error)
AKA - the probability that we correctly reject the null, when the null is false

Question 14

frequentist statistics

Answer 14

developed before computers & calculators ( t-test, chi-square, F test)

Question 15

bayesian statistics

Answer 15

rapidly developing framework for using prior expectation + evidence to make bets

Question 16

measurement validity

Answer 16

how well your metric captures the underlying concept you are trying to measure

Question 17

internal validity

Answer 17

the degree to which the design of an experiment controls extraneous variables

Question 18

external validity

Answer 18

the degree to which effects found in an experiment generalize to other individuals, contexts, and outcomes
(ex: in sampling studies, are the times and places outside of the sampling frame represented accurately)

Question 19

Threats to external validity

interaction of selection and treatment

Answer 19

(unrepresentative responsiveness of treated pop.)

Question 20

Threats to external validity

interaction of setting and treatment

Answer 20

effect of the treatment may differ across geographic or institutional settings

Question 21

Threats to external validity

interaction of history and treatment

Answer 21

effect of the treatment may differ across time periods

Question 22

Threats to external validity

effect may not persist

Answer 22

-as individuals and institutions adapt over time to the treatment

Question 23

Threats to external validity

Partial- Equilibrium Effect

Answer 23

other components of the system also undergo related changes, reducing or eliminating the effect