Flashcards in Lecture 9 Deck (45):

1

## Three common tools/indices to find 'meaningfulness' of statistical analysis?

###
*Statistical Significance (p-value)

*Confidence Intervals (95% or 99%)

*Magnitude of the effect (effect size)

2

## Tools/indices for meaningfulness also tell us:

### How well SAMPLE statistics generalize to the larger TARGET population

3

## Inferential stats

### Making inferences about larger population based on sample--we should see the same results with a different sample

4

## Statistical significance definition

### The probability that a statistic from the sample represents a genuine phenomenon in the population--what we see in the sample we should see in the population

5

## Statistical significance elements

###
*Null Hypothesis Significance Testing

*Systematic and Unsystematic Variation

*Comparing signal to noise

6

## Null Hypothesis Significance Testing

### We test null hypotheses… they are simpler. (H0) The question of interest is simplified into two competing claims (or hypotheses) between which we have a choice (between the null hypothesis and the alternative hypothesis). Special consideration is given to the null hypothesis. (e.g., H0 : there is no difference in symptoms for those receiving the new drug (Tx) compared to the current drug)

7

## Systematic Variation

### Variation that is explained by the model (SIGNAL)

8

## Unsystematic Variation

### Variation that cannot be explained by the model (NOISE)

9

## Comparing Signal:Noise

### We want the Effect(signal)>Error(noise)

10

## What are the two possible conclusions of a hypothesis test with regard to the null hypothesis?

###
Reject

Fail to reject

11

## What are the four possible outcomes of a hypothesis test?

###
*REJECT NULL THAT IS TRUE (Type 1 error, Incorrect Decision)

*REJECT NULL THAT IS FALSE (Correct Decision)

*ACCEPT NULL THAT IS TRUE

(Correct Decision)

*ACCEPT NULL THAT IS FALSE

(Type 2 error, Incorrect Decision)

12

## Null Hypothesis

###
*H-0

*"Simpler" and given priority over a more "complicated" theory

*We either REJECT or FAIL TO REJECT

13

## Alternative Hypothesis

###
*H-1

*A statement of what a statistical hypothesis test is set up to establish

14

## Type I error

###
Rejecting the Null when it is true

**Group differences were found when no actual differences exist

**Alpha

15

## Which is the more serious error: Type I or Type II?

###
A Type I error is more serious and therefore more important to avoid.

*The test procedure is therefore adjusted so there is a guaranteed "low" probability of making a Type I error

16

## The probability of a Type I error can be precisely computed:

### Probability of a Type I error = alpha = p-value

17

## The Probability Value (p-value)

### The probability of getting a value of the test statistic as extreme as or more extreme than that observed by chance alone, if the null is true

18

## What do small p-values suggest?

### The null is unlikely to be true

19

## Common values for significance

### .05, .01, or .001

20

## What happens when you decrease the chance of a Type I error?

### The chances of a Type II error increase!

21

## Type II error

###
Accepting the null when it is false

**No group differences were found when group differences do actually exist

**Beta

22

## What is a Type II error frequently due to?

###
Sample size being too small

**If we accept the null, it may still be false as the sample might not be big enough to detect the differences between groups

23

## What is the exact probability of a Type II error?

### We don't know!

24

## Power

###
The probability of correctly rejecting a false null

**finding an effect when it exists

**In other words, the probability of NOT committing a Type II error

**aka: 1-beta

25

## Max and Min values of Power

###
Max: 1

Min: 0

26

## Reasons for low power

###
*Sample sizes too small

*Use of unreliable measures

27

## What is the Power cutoff social scientists often use?

### .80; there should be at least an 80% chance of NOT making a Type II error

28

## Why is the Power more lenient than the .05 level used in significance testing?

### Because greater care should be taken in asserting the relationship exists, rather than in failing to conclude that a relationship exists

29

## One-Tailed Test of Significance

### Researcher has ruled out interest in one of the directions, and the test is the probability of getting a result as strong/stronger only in ONE direction

30

## Two-Tailed Test of Significance

### Tests the probability of getting a result as strong/stronger than the observed result in either direction

31

## Sampling Distribution

### A theoretical distribution of a sample statistic, used as a model of what would happen if the experiment was repeated infinitely.

32

## Standard Error of Measurement (SE)

###
*The standard deviation of the sampling distribution of a given statistic

*AKA: The measure of how much RANDOM variation between observed scores and expected scores

33

## Standard Error of the Mean

###
The average difference between the population mean and the individual sample mean

**How much error can we expect

**How confident the sample represents the population

34

## What characteristics must be examined for the Standard Error of the Mean?

###
*How large is the sample?

*The standard deviation of the sample

35

## How does sample size affect the Standard Error of the mean?

###
*Small sample size is related to Type II error (not big enough to detect differences)

*The larger the sample, the less error we should have in the estimate about the population (smaller standard error)

36

## How does standard deviation of the sample affect the standard error of the mean?

###
*If the scores in my sample are very diverse (i.e., a lot of variation, a large SD), we can assume the scores in the population are also diverse

*The larger the sample SD = the greater the assumed variation of scores in the population = the larger the standard error of the mean

37

## Small samples with large SDs produce large standard errors. Why?

###
These characteristics make it more difficult to have confidence that the sample accurately represents the population

*Conversely, a large sample with a small SD will produce a small standard error

38

## Effect size

###
A measure of strength or magnitude of experimental effect.

A way of expressing the difference between conditions using a common metric

39

## Why do we use effect size rather than other significance testing?

###
*When examining effects using small sample sizes, significance testing can be misleading because its subject to Type II errors.

*When examining effects using large samples, significant testing can be misleading because even small or trivial effects are likely to produce statistically significant results.

40

## Formulas for effect size

###
*Cohen's d

*Confidence Intervals

41

## Cohen's d

###
Mean difference divided by the pooled standard deviation

The effect size that expresses the difference between two means in standard deviation units

42

## Cohen's d cut-offs

###
.2 = small effect

.5 = medium effect

.8 = large effect

43

## Confidence Interval

### A range of values within which the true differences between groups are likely to be

44

## What p-values do a 95% CI and a 99% CI correspond to?

###
95% = .05

99% = .01

45