Flashcards in Lecture 9 Deck (45):
1
Three common tools/indices to find 'meaningfulness' of statistical analysis?
*Statistical Significance (p-value)
*Confidence Intervals (95% or 99%)
*Magnitude of the effect (effect size)
2
Tools/indices for meaningfulness also tell us:
How well SAMPLE statistics generalize to the larger TARGET population
3
Inferential stats
Making inferences about larger population based on sample--we should see the same results with a different sample
4
Statistical significance definition
The probability that a statistic from the sample represents a genuine phenomenon in the population--what we see in the sample we should see in the population
5
Statistical significance elements
*Null Hypothesis Significance Testing
*Systematic and Unsystematic Variation
*Comparing signal to noise
6
Null Hypothesis Significance Testing
We test null hypotheses… they are simpler. (H0) The question of interest is simplified into two competing claims (or hypotheses) between which we have a choice (between the null hypothesis and the alternative hypothesis). Special consideration is given to the null hypothesis. (e.g., H0 : there is no difference in symptoms for those receiving the new drug (Tx) compared to the current drug)
7
Systematic Variation
Variation that is explained by the model (SIGNAL)
8
Unsystematic Variation
Variation that cannot be explained by the model (NOISE)
9
Comparing Signal:Noise
We want the Effect(signal)>Error(noise)
10
What are the two possible conclusions of a hypothesis test with regard to the null hypothesis?
Reject
Fail to reject
11
What are the four possible outcomes of a hypothesis test?
*REJECT NULL THAT IS TRUE (Type 1 error, Incorrect Decision)
*REJECT NULL THAT IS FALSE (Correct Decision)
*ACCEPT NULL THAT IS TRUE
(Correct Decision)
*ACCEPT NULL THAT IS FALSE
(Type 2 error, Incorrect Decision)
12
Null Hypothesis
*H-0
*"Simpler" and given priority over a more "complicated" theory
*We either REJECT or FAIL TO REJECT
13
Alternative Hypothesis
*H-1
*A statement of what a statistical hypothesis test is set up to establish
14
Type I error
Rejecting the Null when it is true
**Group differences were found when no actual differences exist
**Alpha
15
Which is the more serious error: Type I or Type II?
A Type I error is more serious and therefore more important to avoid.
*The test procedure is therefore adjusted so there is a guaranteed "low" probability of making a Type I error
16
The probability of a Type I error can be precisely computed:
Probability of a Type I error = alpha = p-value
17
The Probability Value (p-value)
The probability of getting a value of the test statistic as extreme as or more extreme than that observed by chance alone, if the null is true
18
What do small p-values suggest?
The null is unlikely to be true
19
Common values for significance
.05, .01, or .001
20
What happens when you decrease the chance of a Type I error?
The chances of a Type II error increase!
21
Type II error
Accepting the null when it is false
**No group differences were found when group differences do actually exist
**Beta
22
What is a Type II error frequently due to?
Sample size being too small
**If we accept the null, it may still be false as the sample might not be big enough to detect the differences between groups
23
What is the exact probability of a Type II error?
We don't know!
24
Power
The probability of correctly rejecting a false null
**finding an effect when it exists
**In other words, the probability of NOT committing a Type II error
**aka: 1-beta
25
Max and Min values of Power
Max: 1
Min: 0
26
Reasons for low power
*Sample sizes too small
*Use of unreliable measures
27
What is the Power cutoff social scientists often use?
.80; there should be at least an 80% chance of NOT making a Type II error
28
Why is the Power more lenient than the .05 level used in significance testing?
Because greater care should be taken in asserting the relationship exists, rather than in failing to conclude that a relationship exists
29
One-Tailed Test of Significance
Researcher has ruled out interest in one of the directions, and the test is the probability of getting a result as strong/stronger only in ONE direction
30
Two-Tailed Test of Significance
Tests the probability of getting a result as strong/stronger than the observed result in either direction
31
Sampling Distribution
A theoretical distribution of a sample statistic, used as a model of what would happen if the experiment was repeated infinitely.
32
Standard Error of Measurement (SE)
*The standard deviation of the sampling distribution of a given statistic
*AKA: The measure of how much RANDOM variation between observed scores and expected scores
33
Standard Error of the Mean
The average difference between the population mean and the individual sample mean
**How much error can we expect
**How confident the sample represents the population
34
What characteristics must be examined for the Standard Error of the Mean?
*How large is the sample?
*The standard deviation of the sample
35
How does sample size affect the Standard Error of the mean?
*Small sample size is related to Type II error (not big enough to detect differences)
*The larger the sample, the less error we should have in the estimate about the population (smaller standard error)
36
How does standard deviation of the sample affect the standard error of the mean?
*If the scores in my sample are very diverse (i.e., a lot of variation, a large SD), we can assume the scores in the population are also diverse
*The larger the sample SD = the greater the assumed variation of scores in the population = the larger the standard error of the mean
37
Small samples with large SDs produce large standard errors. Why?
These characteristics make it more difficult to have confidence that the sample accurately represents the population
*Conversely, a large sample with a small SD will produce a small standard error
38
Effect size
A measure of strength or magnitude of experimental effect.
A way of expressing the difference between conditions using a common metric
39
Why do we use effect size rather than other significance testing?
*When examining effects using small sample sizes, significance testing can be misleading because its subject to Type II errors.
*When examining effects using large samples, significant testing can be misleading because even small or trivial effects are likely to produce statistically significant results.
40
Formulas for effect size
*Cohen's d
*Confidence Intervals
41
Cohen's d
Mean difference divided by the pooled standard deviation
The effect size that expresses the difference between two means in standard deviation units
42
Cohen's d cut-offs
.2 = small effect
.5 = medium effect
.8 = large effect
43
Confidence Interval
A range of values within which the true differences between groups are likely to be
44
What p-values do a 95% CI and a 99% CI correspond to?
95% = .05
99% = .01
45