Ch8 - Confidence Intervals, Effect Size, and Statistical Power

Question 1

What are the new statistics?

Answer 1

1. Effect sizes
   
   2. Confidence intervals
   3. Meta-analysis

Question 2

Point estimate

Confidence Intervals

Answer 2

• A summary statistic from a sample that is just one number used as an estimate of the population parameter - “best guess”
• The true population mean is unknown - and we take a sample from the population to estimate the population mean
• EX: In studies on gender differences in math performance - the mean for boys, the mean for girls, and the difference between them, are point estimates

Question 3

Interval estimate

Confidence Intervals

Answer 3

• Based on a sample statistic and provides a range of plausible values for the population parameter
• Frequently used by the media, often when reporting political polls, and are usually constructed by adding and subtracting a margin of error from a point estimate

Question 4

What is the interval estimate composed of (EQUATION)?

Confidence Intervals

Answer 4

interval estimate = percentage, + and - the margin of error

Question 5

Confidence intervals: we’re not saying that we’re confident that the population mean falls in the interval, but rather…

Confidence Intervals

Answer 5

we are merely saying that we expect to find the population mean within a certain interval a certain percentage of the time - usually 95% - when we conduct this same study with the same sample size

Question 6

Confidence level vs. interval:

Confidence Intervals

Answer 6

• Level - the %
• Interval - range between the two values that suround the sample mean

Question 7

Calculating confidence intervals with distributions

Confidence Intervals

Answer 7

1. Draw a normal curve that has the sample mean at its center (NOTE: different from curve drawn for z test, where we had population mean at the center)
2. Indicate the bounds of the confidence interval on the drawing
3. Determine the z statistics that fall at each line marking the middle of 95%
4. Turn the z statistics back into raw means
5. Check that the confidence interval makes sense

Question 8

Step 1 to calculating CI

Confidence Intervals

Answer 8

Draw a normal curve that has the sample mean at its center (NOTE: different from curve drawn for z test, where we had population mean at the center)

Question 9

Step 2 to calculating CI

Confidence Intervals

Answer 9

• ** 2: Indicate the bounds of the confidence interval on the drawing**
• Draw a vertical line from the mean to the top of the curve
• For a 95% confidence interval we also draw two small vertical lines to indicate the middle 95% of the normal curve (2.5% in each tail, for a total of 5%)
• The curve is symmetric, so half of the 95% falls above and half falls below the mean
• Half of 95% = 47.5%, represented in the segments on either sides of the mean

Question 10

Step 3 to calculating CI

Confidence Intervals

Answer 10

3. Determine the z statistics that fall at each line marking the middle of 95%

• To do so: turn back to the z table
• The % between the mean and each of the scores is 47.5% - when we look up this % in the z table, we find a statistic of 1.96
• Can now add the z statistics of -1.96 and 1.96 to the curve

Question 11

Step 4 to calculating CI

Confidence Intervals

Answer 11

4. Turn the z statistics back into raw means

• Need to identify appropriate mean and SD to use formula
• Two important points to remember:
• Center the interval around the sample mean (not the population mean), so use the sample mean in the calculation
• Because we have a sample mean (rather than an individual score), we use a distribution of means - so we calculate standard error as the measure of spread:

Question 12

Step 5 to calculating CI

Confidence Intervals

Answer 12

5. Check that the confidence interval makes sense
* The sample mean should fall exactly in the middle of the two ends of the interval

Question 13

Statistically significant doesn’t/does mean…

Answer 13

• Does NOT mean that the findings from a study represent a meaningful difference
• ONLY means that those findings are unlikely to occur, in fact, if the null hypothesis is true

Question 14

How does an increase in sample size affect SD and the test statistic? What dooes this cause?

The effect of sample size on statistical significance

Answer 14

• Each time we increased the sample size, the SD decreased and the test statistic increased
• Because of this, a small difference might not be statistically significant with a small sample but might be statistically significant with a large sample

Question 15

Why would a large sample allow us to reject the null hypothesis than a small sample? (EXAMPLE)

Answer 15

If we randomly selected 5 women and they had a mean score well above the OkCupid average, we might say “it could be chance”; but if we randomly selected 1000 women with a mean rating well above the OkCupid average, it’s very unlikely that we just happened to choose 1000 people with high scores

Question 16

Effect size

Answer 16

• Indicates the size of a difference and is unaffected by sample size
• Can tell us whether a statistically significant difference might also be an important difference
• Tells us how much two populations DO NOT overlap - the less overlap, the bigger the effect size
• DECREASING OVERLAP IS IDEAL!

Question 17

How can the amount of overlap between two distributions be decreased? TWO WAYS:

Answer 17

1: overlap decreases and effect size increases when means are farther apart (distance wise)
2: overlap decreases and effect size increases when variability within each distribution of scores is smaller (height of peak)

Question 18

How does effect size differ from statistical hypothesis testing?

Answer 18

Unlike statistical hypothesis testing, effect size is a standardized measure based on distributions of scores rather than distributions of means
* Rather than om = o/√N, effect sizes are based only on the variability in the distribution of scores and do not depend on sample size

Question 19

Since effect sizes are not dependent on sample size, what does this allow us to do?

Answer 19

This means we can compare the effect sizes of different studies with each other, even when the studies have different sample sizes

Question 20

When we conduct a z-test, the effect size is typically

Answer 20

Cohen’s D: a measure of effect size that expresses the difference between two means in terms of SD
* AKA, Cohen’s d is the standardized difference between two means

Question 21

Formula for Cohen’s d for a z statistic:

Answer 21

d = (M - u)/o
- Similar to z statistic (om -> o, um -> u)

Question 22

With the results, we can determine (from Cohen’s 3 guidelines)…

Small, Medium, Large Effects

Answer 22

• Small effects: 0.2 | 85% overlap
• Medium effects: 0.5 | 67% overlap
• Large effects: 0.8 | 53% overlap

Question 23

Does an effect need to be large to be meaningful?

Answer 23

Just because a statistically significant difference is small, that does not necessarily suggest no meaning; interpreting the meaningfulness of the effect sizes depends on the context

Question 24

Meta-analysis:

Meta-analysis

Answer 24

• a study that involves the calculation of a mean effect size from the individual effect sizes of more than one study

Question 25

How do meta-analysis improve statistical power?

Answer 25

By considering multiple studies simultaneously and helps to resolve debates fueled by contradictory research findings

Question 26

4 steps to calculating meta-analyses:

Answer 26

1: select the topic of interest and decide exactly how to proceed before beginning to track down studies 2: locate every study that has been conducted and meets criteria 3: calculate an effect size, often Cohen's d, for every study 4: calculate statistics - ideally, summary statistics, a hypothesis test, a confidence interval, and a visual display of the effect sizes

Question 27

Considerations to keep in mind: ## Footnote 1: select the topic of interest and decide exactly how to proceed before beginning to track down studies

Answer 27

* **Make sure the necessary statistical information is available**, either effect sizes of the summary stats necessary to calculate effect sizes * **Consider selecting only studies in which participants meet certain criteria**, such as age, gender, or geographic location * **Consider eliminating studies based on the research design** (EX: as they were not experimental in nature)

Question 28

Key part involves finding... ## Footnote 2: locate every study that has been conducted and meets criteria

Answer 28

...any studies that have been conducted but not published * Much of this "fugitive literature" or "gray literature" is unpublished simply **because studies did not find a significant difference; the overall effect size seems larger without accounting these studies** - AKA the **"file drawer problem"** * Can find by using other sources - like contacting researchers to find unpublished work

Question 29

"File drawer problem" - 2 solutions ## Footnote 2: locate every study that has been conducted and meets criteria

Answer 29

1: **File drawer analysis**: a statistical calculation, following a meta-analysis, of the number of studies with null results that would have to exist so that a mean effect size would no longer be statistically significant * If just a few studies could render a mean effect size nonsignificant (no longer statistically significantly different from zero) then the mean effect size should be viewed as likely to be an inflated estimate * If it would take several hundred studies in researchers' "file drawers" to render the effect non-significant, then it's safe to conclude that there really is a significant effect 2: Can work with replication to help draw more reliable conclusions

Question 30

What visual display can researchers include? ## Footnote 4: calculate statistics - ideally, summary statistics, a hypothesis test, a confidence interval, and a visual display of the effect sizes

Answer 30

**Forest plot**: type of graph which shoes the confidence interval for the effect size of every study

Question 31

Statistical power is...

Answer 31

...the likelihood of rejecting the null hypothesis WHEN WE SHOULD reject the null hypothesis

Question 32

What is the probability that researchers consider the MINIMUM for conducting a study ## Footnote Statistical power

Answer 32

0.80 - an 80% chance of rejecting the null if we should reject it * Thus, they perform power analysis prior to conducting a study: if they have an 80% chance of correctly rejecting the null, then it's appropriate to conduct the study

Question 33

When we conduct a statistical null hypothesis test, we make a decision to either reject or fail to reject the null hypothesis. One issue being that we don't have direct access to the truth about what we're studying - instead...

Answer 33

* We make inferences based on the data we collected; which could be a right or wrong decision * Overall a researcher's goal is to be correct as often as possible - 2 ways to be right, and 2 ways to be wrong

Question 34

What are 2 ways to be **WRONG** in rejecting/failing to reject the null hypothesis?

Answer 34

2 ways to be wrong - recap: **Type I and Type II errors**

Question 35

What are 2 ways to be **RIGHT** in rejecting/failing to reject the null hypothesis?

Answer 35

1 - **Correct decision:** if the null is true and we fail to reject the null, we have made the correct decision (**essentially leaving the null alone**) - In this case, we're saying that there's no effect, when in fact there is none 2 - **Correct decision (Power)**: if the null hypothesis is false, and we reject the null hypothesis, that's also a correct decision - A goal of research is to maximize statistical power

Question 36

Power is used by statisticians in a specific way - HOW?

Answer 36

* **Statistical power**: a measure of the likelihood that we will reject the null hypothesis, given that the null hypothesis is false * In other words - statistical power is the probability that we will reject the null hypothesis when we should reject the null hypothesis; **THE PROBABILITY THAT WE WILL NOT MAKE A TYPE II ERROR**

Question 37

The calculation of statistical power *ranges* from:

Answer 37

Probability of 0.00 to 1.00 (AKA 0% to 100%)

Question 38

Conceptual calculation for power

Answer 38

* Power = effect size x sample size * This means that we could achieve high power because the size of the effect is large - or we could achieve high power because the size of the effect is small, but it's a large sample

Question 39

The most practical way to increase statistical power for many behavioural studies is...

Answer 39

...to add more participants

Question 40

How can researchers **quantify** the statistical power of their studies? 2 WAYS

Answer 40

1: By referring to a published table 2: By using computing tools like G*Power

Question 41

G*Power

Answer 41

Used in 2 ways: **1: Can calculate power *AFTER* conducting a study from several pieces of information** - Because we are calculating power after conducting the study, G*Power refers to these calculations as **post hoc, meaning after the fact** **2: Can use in reverse, *BEFORE* conducting a study, so as to *identify the sample size necessary* to achieve a given level of power** - In this case, G*Power refers to calculations as a **priori, which means prior to**

Question 42

Of the two, which of post hoc vs priori power is more meaningful?

Answer 42

post hoc power is NOT as meaningful as a priori power calculation for sample size planning

Question 43

On a practical level, statistical power calculations tell researchers...

Answer 43

...how many participants are needed to conduct a study whose findings we can trust

Question 44

Five factors that affect statistical power:

Answer 44

1: Increase alpha 2: Turn a two-tailed hypothesis into a one-tailed hypothesis 3: Increase N/sample size 4: Exaggerate the mean difference between levels of the IV 5: Decrease SD

Question 45

1: Increase alpha ## Footnote Five factors that affect statistical power:

Answer 45

* Like changing the rules by widening the goal posts in football, **statistical power can increase when we increase an alpha level of 0.05** * This has the side effect of increasing the probability of a Type I error from 5% to 10%

Question 46

2: Turn a two-tailed hypothesis into a one-tailed hypothesis ## Footnote Five factors that affect statistical power:

Answer 46

* **One tailed tests provide more statistical power**, while two-tailed tests are more conservative * However, best to use two-tailed

Question 47

3: Increase N/sample size ## Footnote Five factors that affect statistical power:

Answer 47

* Increasing sample size leads to an increase in the test statistic, making it easier to reject the null hypothesis * Increase => distribution of means become more narrow and there is less overlap (larger sample size means smaller standard error)

Question 48

4: Exaggerate the mean difference between levels of the IV ## Footnote Five factors that affect statistical power:

Answer 48

The mean of population 2 is farther from the mean of population in part b) than it is in part a); difference in means is not easily changed, but can be done

Question 49

5: Decrease SD ## Footnote Five factors that affect statistical power:

Answer 49

When SD is smaller, standard error is smaller and the curves are narrower We can reduce SD in two ways: **1: by using *reliable measures* from the beginning of the study 2: by *sampling from a more homogenous group* in which participants' responses are more likely to be similar to begin with**

Question 50

LECTURES

Question 51

CHP8 concepts push beyond the limits of NHST

Answer 51

* Effect size * Confidence intervals * Power

Question 52

Effect size: ## Footnote CHP8 concepts push beyond the limits of NHST

Answer 52

* If the null is really false, **how big is that effect?** * Standardized numerical estimate of the population effect size using our sample data

Question 53

Confidence intervals: ## Footnote CHP8 concepts push beyond the limits of NHST

Answer 53

* Starting with sample mean, **compute a range of plausible values for the true population of the mean** * Helps us prepare for replications

Question 54

Power ## Footnote CHP8 concepts push beyond the limits of NHST

Answer 54

- If the null is really false, **how likely is it that we’re going to find a "significant effect" in our sample** - If the null is really false, how likely is it that we're going to avoid a type II error

Question 55

Effect size: after rejecting the null, we can conclude that...

Answer 55

we think we drew this sample from a different population with a different sampling distribution

Question 56

Effect size - How can we guess the population of the mean we drew from?

Answer 56

Calculating the tallest point in the distribution, most common score

Question 57

What does effect size look like, visually?

Answer 57

Distance from the highest peak of one distribution to the other (distance between group means)

Question 58

How/what does effect size help us estimate?

Answer 58

If we DID draw from a different population, how different is that new population's mean from the null mean?

Question 59

What are some different ways to estimate an effect size for different kinds of data?

Answer 59

* How far away is the true mean from the null hypothesis mean? * How far apart are the experimental and control conditions * Strength of correlation * How far from equal (50% each) is the distribution of proportions

Question 60

Which volume of effect is easiest to detect?

Answer 60

smaller effects are HARDER to detect from drawing a single sample; larger effects (thus, larger Cohen's d) are EASIER to detect

Question 61

What is one of the many "standardized" indicators of effect size?

Answer 61

* COHEN'S D * Estimates the population parameter - **δ** (delta)

Question 62

How many SD away from the comparison value is our sample group mean?

Answer 62

* d = (M-u)/o * NOTE: equation is in-between absolute value bars

Question 63

What can we calculate to answer: "***How likely is it*** that our class is a random sample from this general population? Or do we likely come from a different population?"

Answer 63

We can use a z test or we can use a confidence interval

Question 64

All CI's follow the same pattern…

Answer 64

Subtract margin of error to find lower bound (critical value x standard error), add margin of error to find upper point (critical value x standard error)

Question 65

Type I error

Answer 65

* H0 is reject * H0 is actually true

Question 66

Correct Decision POWER (1- β)

Answer 66

* H0 is reject * H0 is false

Question 67

Type II error

Answer 67

* Fail to reject H0 * H0 is actually false

Question 68

Correct Decision (1 + α)

Answer 68

* Retain H0 * H0 is true

Question 69

What does β mean?

Answer 69

If the null is really false (effect exists), β% of the time we're going to make a mistake and say there's no effect

Question 70

To identify power:

Answer 70

find the % of the curve of the H1 distribution that would lead us to correctly reject the null

Question 71

If effect size increases, what happens to type I error rate?

Answer 71

* NO CHANGE

Question 72

If effect size increases, what happens to type II error rate?

Answer 72

* DECREASES

Question 73

If effect size increases, what happens to power?

Answer 73

* INCREASES

Question 74

In priori power analysis, can ask two questions:

Answer 74

* "If I'm making the assumption that there is an effect to be found, HOW MANY PEOPLE DO I NEED IN MY STUDY?" * "If I'm limited to N participants, will I have enough power to reject the null hypothesis if I should do so?"