Lecture 9

Question 1

Three common tools/indices to find ‘meaningfulness’ of statistical analysis?

Answer 1

• Statistical Significance (p-value)
• Confidence Intervals (95% or 99%)
• Magnitude of the effect (effect size)

Question 2

Tools/indices for meaningfulness also tell us:

Answer 2

How well SAMPLE statistics generalize to the larger TARGET population

Question 3

Inferential stats

Answer 3

Making inferences about larger population based on sample–we should see the same results with a different sample

Question 4

Statistical significance definition

Answer 4

The probability that a statistic from the sample represents a genuine phenomenon in the population–what we see in the sample we should see in the population

Question 5

Statistical significance elements

Answer 5

• Null Hypothesis Significance Testing
• Systematic and Unsystematic Variation
• Comparing signal to noise

Question 6

Null Hypothesis Significance Testing

Answer 6

We test null hypotheses… they are simpler. (H0) The question of interest is simplified into two competing claims (or hypotheses) between which we have a choice (between the null hypothesis and the alternative hypothesis). Special consideration is given to the null hypothesis. (e.g., H0 : there is no difference in symptoms for those receiving the new drug (Tx) compared to the current drug)

Question 7

Systematic Variation

Answer 7

Variation that is explained by the model (SIGNAL)

Question 8

Unsystematic Variation

Answer 8

Variation that cannot be explained by the model (NOISE)

Question 9

Comparing Signal:Noise

Answer 9

We want the Effect(signal)>Error(noise)

Question 10

What are the two possible conclusions of a hypothesis test with regard to the null hypothesis?

Answer 10

Reject

Fail to reject

Question 11

What are the four possible outcomes of a hypothesis test?

Answer 11

• REJECT NULL THAT IS TRUE (Type 1 error, Incorrect Decision)
• REJECT NULL THAT IS FALSE (Correct Decision)

*ACCEPT NULL THAT IS TRUE
(Correct Decision)

*ACCEPT NULL THAT IS FALSE
(Type 2 error, Incorrect Decision)

Question 12

Null Hypothesis

Answer 12

• H-0
• “Simpler” and given priority over a more “complicated” theory
• We either REJECT or FAIL TO REJECT

Question 13

Alternative Hypothesis

Answer 13

• H-1

* A statement of what a statistical hypothesis test is set up to establish

Question 14

Type I error

Answer 14

Rejecting the Null when it is true

• *Group differences were found when no actual differences exist
• *Alpha

Question 15

Which is the more serious error: Type I or Type II?

Answer 15

A Type I error is more serious and therefore more important to avoid.
*The test procedure is therefore adjusted so there is a guaranteed “low” probability of making a Type I error

Question 16

The probability of a Type I error can be precisely computed:

Answer 16

Probability of a Type I error = alpha = p-value

Question 17

The Probability Value (p-value)

Answer 17

The probability of getting a value of the test statistic as extreme as or more extreme than that observed by chance alone, if the null is true

Question 18

What do small p-values suggest?

Answer 18

The null is unlikely to be true

Question 19

Common values for significance

Answer 19

.05, .01, or .001

Question 20

What happens when you decrease the chance of a Type I error?

Answer 20

The chances of a Type II error increase!

Question 21

Type II error

Answer 21

Accepting the null when it is false

• *No group differences were found when group differences do actually exist
• *Beta

Question 22

What is a Type II error frequently due to?

Answer 22

Sample size being too small
**If we accept the null, it may still be false as the sample might not be big enough to detect the differences between groups

Question 23

What is the exact probability of a Type II error?

Answer 23

We don’t know!

Question 24

Power

Answer 24

The probability of correctly rejecting a false null

• *finding an effect when it exists
• *In other words, the probability of NOT committing a Type II error
• *aka: 1-beta

Question 25

Max and Min values of Power

Answer 25

Max: 1
Min: 0

Question 26

Reasons for low power

Answer 26

• Sample sizes too small

* Use of unreliable measures

Question 27

What is the Power cutoff social scientists often use?

Answer 27

.80; there should be at least an 80% chance of NOT making a Type II error

Question 28

Why is the Power more lenient than the .05 level used in significance testing?

Answer 28

Because greater care should be taken in asserting the relationship exists, rather than in failing to conclude that a relationship exists

Question 29

One-Tailed Test of Significance

Answer 29

Researcher has ruled out interest in one of the directions, and the test is the probability of getting a result as strong/stronger only in ONE direction

Question 30

Two-Tailed Test of Significance

Answer 30

Tests the probability of getting a result as strong/stronger than the observed result in either direction

Question 31

Sampling Distribution

Answer 31

A theoretical distribution of a sample statistic, used as a model of what would happen if the experiment was repeated infinitely.

Question 32

Standard Error of Measurement (SE)

Answer 32

• The standard deviation of the sampling distribution of a given statistic
• AKA: The measure of how much RANDOM variation between observed scores and expected scores

Question 33

Standard Error of the Mean

Answer 33

The average difference between the population mean and the individual sample mean

• *How much error can we expect
• *How confident the sample represents the population

Question 34

What characteristics must be examined for the Standard Error of the Mean?

Answer 34

• How large is the sample?

* The standard deviation of the sample

Question 35

How does sample size affect the Standard Error of the mean?

Answer 35

• Small sample size is related to Type II error (not big enough to detect differences)
• The larger the sample, the less error we should have in the estimate about the population (smaller standard error)

Question 36

How does standard deviation of the sample affect the standard error of the mean?

Answer 36

• If the scores in my sample are very diverse (i.e., a lot of variation, a large SD), we can assume the scores in the population are also diverse
• The larger the sample SD = the greater the assumed variation of scores in the population = the larger the standard error of the mean

Question 37

Small samples with large SDs produce large standard errors. Why?

Answer 37

These characteristics make it more difficult to have confidence that the sample accurately represents the population

*Conversely, a large sample with a small SD will produce a small standard error

Question 38

Effect size

Answer 38

A measure of strength or magnitude of experimental effect.

A way of expressing the difference between conditions using a common metric

Question 39

Why do we use effect size rather than other significance testing?

Answer 39

• When examining effects using small sample sizes, significance testing can be misleading because its subject to Type II errors.
• When examining effects using large samples, significant testing can be misleading because even small or trivial effects are likely to produce statistically significant results.

Question 40

Formulas for effect size

Answer 40

• Cohen’s d

* Confidence Intervals

Question 41

Cohen’s d

Answer 41

Mean difference divided by the pooled standard deviation

The effect size that expresses the difference between two means in standard deviation units

Question 42

Cohen’s d cut-offs

Answer 42

.2 = small effect
.5 = medium effect
.8 = large effect

Question 43

Confidence Interval

Answer 43

A range of values within which the true differences between groups are likely to be

Question 44

What p-values do a 95% CI and a 99% CI correspond to?

Answer 44

95% = .05
99% = .01

Question 45

Degrees of Freedom

Answer 45

The minimum amount of data needed to calculate a statistic

Determines that exact form of the probability distribution.

N-1