Chapter 11: Analysis Of Variance (ANOVA)

Question 1

ANOVA is appropriate for:

A: Between-group designs with three or more groups

B: Between-group designs with two or more groups

C: Within-group designs with two or more groups

Answer 1

B: Between-group designs with two or more groups

For example:
> You would start with a sample of participants. During the randomization procedure, you would then split them into two or more completely different groups (read research group, illustrate risks group, and control group).

> ANOVA is more flexible than say an independent-samples t-test and can handle any number of groups (whereas independent can only be used for a maximum of two groups) BUT between the two, they will provide identical p-values when both are applied to a two-group design

Question 2

ANOVA is appropriate for research scenarios with a:

A: Numerical independent variable and one categorical dependent variable

B: Categorical independent variable and one numeric dependent variable

C: Two or more independent variables and two or more numeric dependent variables

Answer 2

B: Categorical independent variable and one numeric dependent variable

> If you see that the independent variable is numerical and the dependent variable is categorical you should know that cannot use ANOVA anymore!

Question 3

The independent variable is the __________ and the dependent variable is the ____________.

A: Outcome, predictor

B: Predictor, outcome

C: Larger number, smaller number

Answer 3

B: Predictor, outcome

EXAMPLES:
Q: Do three treatments (therapy, drug, therapy + drug) differentially impact depression levels?
> Independent (Predictor) = Treatment condition (three groups).
> Dependent (Outcome) = Numeric depression scale

Q: Do Republicans, Democrats, and Independents differ with respect to their religiousness?
> Independent (Predictor) = Political affiliation (three groups)
> Dependent (Outcome) = Numeric religiousness scale

Q: Does memory training produce performance differences on a memory task relative to a control
group?
> Independent (Predictor) = Memory training (two groups)
> Dependent (Outcome) = Numeric number of memory errors

Question 4

We want to compare old and young people in
terms of their cognitive abilities. We can use:

A. ANOVA
B. Independent-sample t-test
C. ANOVA or Independent-sample t-test

Answer 4

C. ANOVA or Independent-sample t-test

> And, since there are only two groups you know that they will also both give you the same p-value!

Question 5

What is the null hypothesis for ANOVA?

A: The null hypothesis for between-group designs
targets group mean differences. The ANOVA null hypothesis states that all population means are the same ( i.e., “nothing going on”).

B: The null hypothesis in ANOVA suggests that each group’s mean is unique, and there are no commonalities in the population means.

C: For ANOVA, the null hypothesis assumes that the differences between group means are so large that they cannot be attributed to random variation within the samples.

Answer 5

A: The null hypothesis for between-group designs
targets group mean differences. The ANOVA null hypothesis states that all population means are the same ( i.e., “nothing going on”).

Written like this:
H0: μ1 = μ2 = μ3

Question 6

What is the alternate hypothesis for ANOVA?

A: The alternate hypothesis states that at least one
pair of groups in the population has different means (i.e., “something going on”).

B: The alternate hypothesis in ANOVA asserts that all group means in the population are identical, and any observed differences are due to sampling error.

C: In the alternate hypothesis for ANOVA, it is proposed that all groups have the exact same mean, indicating no significant variation between them.

Answer 6

A: The alternate hypothesis states that at least one
pair of groups in the population has different means (i.e., “something going on”).

Written like this:
Ha: μ1 ≠ μ2 OR μ1 ≠ μ3 OR μ2 ≠ μ3

** We will not be dealing with one-tailed ANOVA’s so this is all you need to know! Two-tailed only for this class **

Question 7

What is variance for ANOVA:

A: Variance in ANOVA is calculated as the absolute distance from individual scores to the mean, providing a measure of the overall spread of the data.

B: The variance for ANOVA is determined by taking the square root of the average distance between individual scores and the mean, giving a standardized measure of data dispersion.

C: The variance is the average SQUARED distance from
individual scores to the mean

Answer 7

C: The variance is the average SQUARED distance from
individual scores to the mean

NOTE: It’s just the standard deviation squared and the standard deviation is just the square root of the variance

> I would screenshot the formula on slide 22 and add it to your cheat sheet!!!

N-1
> The adjusted sample size in the denominator —
the degrees of freedom — gives a better estimate
of the population variance. If we don’t do this the value would be too small.

Question 8

Variance interpretation vs. standard deviation interpretation - ADD THIS TO YOUR CHEAT SHEET!!!

Answer 8

Variance interpretation:
> The average squared distance from the individual
scores to the sample mean is 1.29

Standard Deviation Interpretation:
> On a 6-point scale, the average distance from the individual scores to the sample mean is 1.14 points

Question 9

ANOVA views each person’s score as consisting of
two components:

A: In ANOVA, each person’s score is composed of a single factor representing the overall group effect, summarizing the impact of all interventions or treatments.

B: A between-group effect (e.g., the effect of the intervention) and a left-over part (residual).

C: ANOVA breaks down individual scores into three components: between-group effect, within-group effect, and the interaction effect, providing a comprehensive understanding of score composition.

Answer 9

B: A between-group effect (e.g., the effect of the intervention) and a left-over part (residual).

The between-group effect:
> Is variation due to the independent variable (Ho predicts this is zero!).

The left-over (residual):
> Is naturally occurring score variation unrelated to group membership. It’s unrelated to what you’re trying to measure.

> The ANOVA technique is based on variance (numerous variances).

Question 10

The group effect/residual effect is calculated as follows:

A: Total = Group 1 + Group 2 + Group 3, etc.

B: Total = Group Effect + Residual

C: Total = Variance + Standard Deviation

Answer 10

B: Total = Group Effect + Residual

PUT THIS ON YOUR CHEAT SHEET:

Group Effect:
> Group mean - sample mean (x̄2 - x̄)
x̄2 = 5.38 and x̄ = 5.08
Group effect = 5.38 - 5.08 = 0.3

Residual:
> Individual score - group mean (x - x̄2)
x = 6.00 and x̄2 = 5.38
Residual = 6.00 - 5.38 = 0.62

Total Distance:
> Group effect + residual
Group effect = 0.3 and residual = 0.62
Total distance = 0.3 + 0.62 = 0.92

ALTERNATIVELY, you can just do:
Individual score (x) - sample mean (x̄)
Individual score = 6.00 and Sample mean = 5.08
Total distance = 6.00 - 5.08 = 0.92

Question 11

You’ll need to look at slide 28 to answer this:

How large is the group effect?

A: 5.08 - 5.04
B: 5.04 - 5.08
C: 5.9 - 5.04

Answer 11

B: 5.04 - 5.08

Question 12

You’ll need to look at slide 28 to answer this:

How large is the residual?

A: 5.08 - 5.04
B: 5.04 - 5.08
C: 5.9 - 5.04

Answer 12

C: 5.9 - 5.04

Question 13

So what are the two sources of variability for ANOVA?

A: Between-group and residual (left-over)

B: Between-group and within-group

C: Variance and standard deviation

Answer 13

A: Between-group and residual (left-over)

Between-group variability:
> Shows up in Jamovi as the MEANS of all the interventions involved (mean differences - add image from slide 34)

Residual (left-over) variability:
> Shows up in Jamovi as the VARIANCE of all the interventions involved (natural score differences - add image from slide 35)

ANOVA divides score variation into these two sources: group differences and residuals. The sums of squares (group) and variances (residual) give the size of the partitions (add image from slide 36).

Question 14

With ANOVA, what is another name for variance?

A: Mean square

B: Standard deviation

C: Average distance

Answer 14

A: Mean square

Mean squares are ANOVA terms for variance
(an average squared distance)

The between-group mean square quantifies
group mean differences on a squared metric

The residual mean square quantifies naturally
occurred score differences within each group on
a squared metric

It’s ALWAYS on a squared metric!!!

PUT THIS ON YOUR CHEAT SHEET!!!!

Question 15

ADD THE IMAGE FROM SLIDE 38 TO YOUR CHEAT SHEET!!!

Answer 15

ADD THE IMAGE FROM SLIDE 38 TO YOUR CHEAT SHEET!!!

Question 16

So what is the between-group mean square (variance):

A: Between-group mean square (variance) captures the degree to which group means differ on a squared metric.

B: Between-group mean square in ANOVA is a measure of the average difference between individual scores within each group, indicating the extent of variability within groups.

C: The between-group mean square represents the average squared difference between the highest and lowest scores within each group, providing insights into the range of individual performance within groups.

Answer 16

A: Between-group mean square (variance) captures the degree to which group means differ on a squared metric.

This is the variation between groups due to the independent variable (treatment effect).

ADD IMAGE FROM SLIDE 39!!!

EXAMPLE:
> You’re looking at the “Intervention” line
> You’re taking the “mean square between groups” (sum of squares) ÷ “adjusted # units (df)” (in this case G-1 OR 3-1, since there are 3 groups)
> 19.3 ÷ 2 = 9.64
> MSbg = 9.64
> You cannot have a negative mean square

Question 17

So what is the residual mean square (variance):

A: The residual mean square in ANOVA reflects the average squared difference between group means, highlighting the inherent variability in group performances.

B: Residual mean square measures the squared distance between each individual score and the overall mean, providing a comprehensive assessment of the contribution of each score to the overall variance.

C: Residual mean square captures natural score variation having nothing to do with group membership on a squared metric.

Answer 17

C: Residual mean square captures natural score variation having nothing to do with group membership on a squared metric.

The residual mean square (also called the mean square error or mean square within) quantifies the variation within each group (within-group) that cannot be explained by the independent variable. It is the residual variation within groups that is not explained.

ADD IMAGE FROM SLIDE 40!!!

EXAMPLE:
> You’re looking at the “Residual” line
> You’re taking the “mean square of residual” (sum of squares) ÷ “adjusted # units (df)” (in this case N-G OR 341-3, since we have 341 participants and 3 groups)
> 420.1 ÷ 338 = 1.24
> MSresidual = 1.24
> You cannot have a negative mean square

Question 18

So what is the sum of squares of total:

A: The sum of squares of total in ANOVA is obtained by multiplying the Mean square between groups by the Mean square of residuals, providing a comprehensive measure of overall variance.

B: To calculate the sum of squares of total, you subtract the Mean square of residuals from the Mean square between groups, yielding a combined measure of group-specific and individual variation.

C: Add together the Mean square between groups (sum of squares) and the Mean square of residual (sum of squares) to get the total size of the pie.

Answer 18

C: Add together the Mean square between groups (sum of squares) and the Mean square of residual (sum of squares) to get the total size of the pie.

ADD IMAGE FROM SLIDE 40!!!

19.3 + 420.1 = 439.4 (the total size of the pie)

Question 19

The between-group mean square quantifies group mean differences as:

A: Variance
B: Standard deviation
C: Standard error

Answer 19

A: Variance

Question 20

What is a generic t-statistic:

A: The generic t-statistic measures the absolute difference between group means, without considering the impact of sampling error or random chance.

B: A t-statistic compares the observed effect from
the data to the size of the effect expected due
to sampling error (i.e., “random chance”)

C: The generic t-statistic is calculated by dividing the standard deviation of the sample by the mean difference, offering a direct measure of effect size in the absence of sampling error.

Answer 20

B: A t-statistic compares the observed effect from
the data to the size of the effect expected due
to sampling error (i.e., “random chance”)

INCLUDE THE FORMULA IMAGE FROM SLIDE 43 ON YOUR CHEAT SHEET!!!

THINGS TO KNOW:
> The denominator represents random chance
> Larger effects ⇨ larger t-statistics
> Larger t-statistic = reject the null hypothesis
> Represent in standard error units
> It’s the same idea from one-sample, paired-sample, and independent-sample
> It’s a ratio

Question 21

So then what is an F-statistic in ANOVA:

A: The F-statistic in ANOVA is a measure of the absolute difference between group means, ignoring the influence of naturally-occurring variance in the data.

B: ANOVA uses an F-statistic that compares the
variance in scores due to group mean differences
relative to naturally-occurring variance

C: ANOVA’s F-statistic compares the range of individual scores within each group to the overall mean, providing a direct measure of the spread of data across groups.

Answer 21

B: ANOVA uses an F-statistic that compares the
variance in scores due to group mean differences
relative to naturally-occurring variance

INCLUDE THE FORMULA IMAGE FROM SLIDE 44 ON YOUR CHEAT SHEET!!!

THINGS TO KNOW:
> It a ratio
> The numerator represents observed differences
> The denominator represents random chance (within-group)
> Larger effects ⇨ larger F ratios
> Large f ratio = reject the null hypothesis
> Larger f-value = smaller p-value
> You cannot have a negative f statistic
> Formula is between-group vs. within-group effect

Question 22

How do you interpret an F-statistic? PUT THIS ON YOUR CHEAT SHEET!!!

Answer 22

Interpretation:
> On a standardized metric, score variation due to the intervention group mean differences is 7.76 times as large as that due to naturally occurring differences (within-group)

THINGS TO KNOW:
> You can only call it a “naturally occurring difference”
> The numerator is 7.76 times larger than the denominator
> In Jamovi this is the intervention mean square ÷ the residual mean square and generates under “F”

Question 23

We want to compare old and young people in terms of their cognitive abilities. Will the t-test and F-test provide the same p-value?

A: Yes
B: No

Answer 23

A: Yes

A squared t-statistic will equal the F-statistic when you have only two groups and they will have the same p-value

Because of sampling error f-statistics will be different from one sample to another but together they will construct a sampling distribution called an f-distribution

Question 24

What is an F-distribution:

A: The F-distribution in ANOVA is symmetric, and its shape is independent of the number of groups and participants per group. It follows a normal distribution regardless of sample characteristics.

B: An F-distribution is characterized by a negatively skewed sampling distribution in ANOVA. The shape of the F-sampling distribution remains consistent regardless of the number of groups and participants per group.

C: Samples from a population with identical means
(the null is true) produce F-statistics that follow
a positively skewed sampling distribution. The shape of the F-sampling distribution (and its p-values) depends on the number of groups and participants per group.

Answer 24

C: Samples from a population with identical means
(the null is true) produce F-statistics that follow
a positively skewed sampling distribution. The shape of the F-sampling distribution (and its p-values) depends on the number of groups and participants per group.

> The expected F-value is 1 if the null hypothesis is true (non-significant).

> In an f-test everything is one-tailed

THIS IS CONFUSING SO I WOULD PROBABLY ADD THE IMAGES FROM SLIDES 50, 51, AND 52

Question 25

How do you interpret an F-statistic/F-distribution/p-value (probability)?

ADD THIS TO YOUR CHEAT SHEET!

Answer 25

If the group means are truly equal in the population, the
probability of drawing a sample with an F-statistic of 7.76 or larger is smaller than 0.001.

Look for answers that say things like:
> “Given” that ALL groups (however many groups)….
> Whatever the F-statistic is and then “or larger” since it’s one-tailed.

THIS IS CONFUSING SO I WOULD PROBABLY ADD THE IMAGE FROM SLIDE 52!

Question 26

You’ll need to look at slide 52 to answer this question:

What is your conclusion about the significance of
the results?

A. Significant
B. Nonsignificant

Answer 26

A. Significant

Question 27

What is the η2 (Eta-squared) effect size?

A: The η2 effect size is the proportion of variation due to the grouping variable (independent variable).

B: The η2 effect size in ANOVA is a measure of the absolute difference between group means, without considering the proportion of variation attributable to the independent variable.

C: η2 effect size represents the percentage of total variation in the data that is due to random error, providing insights into the stability of the results across different samples.

Answer 27

A: The η2 effect size is the proportion of variation due to the grouping variable (independent variable).

SHE MIGHT ALSO SAY:
> η2 is the percent of the variance “pie” due to group mean differences (THE PICTURE ON SLIDE 58 MIGHT BE MORE HELPFUL THAN THE PIC ON SLIDE 56 - YOU DECIDE - OR MAYBE YOU WANT A LITTLE OF BOTH - BECAUSE ONE ALSO HAS THE FORMULA WHICH MIGHT BE NICE TO SEE)

THINGS TO KNOW:
> This is considered a “practical significance,” like Cohen’s d and it’s less sensitive to sample size
> Denominator = size of the total pie (total sum square - intervention sum of squares + residual sum of squares)
> You can have a negative Cohen d and it can also be more than 1 BUT η2 cannot be negative and it cannot be more than 1 - it can only be 0 to 1

INTERPRETATION:
> The independent variable explains 4% of the total variation in attitudes (YOU CAN ADD THE IMAGE FROM SLIDE 56 IF YOU THINK IT WILL BE HELPFUL TO SEE IT)

Question 28

YOU NEED TO PUT THESE GUIDELINES ON YOUR CHEAT SHEET! JUST TAKE A SCREENSHOT!

Answer 28

YOU NEED TO PUT THESE GUIDELINES ON YOUR CHEAT SHEET! JUST TAKE A SCREENSHOT!

Question 29

You need to look at slide 58 to answer this question:

What is the practical conclusion that you can draw
from the test?

A. Negligible effect size
B. Small effect size
C. Moderate effect size
D. Large effect size

Answer 29

B. Small effect size

Question 30

True or false: We cannot compute a confidence interval?

A: True
B: False

Answer 30

A: True - That’s because ANOVA’s are only one-tailed

Question 31

What is a Pairwise (Post Hoc) Comparison:

A: Pairwise (Post Hoc) comparisons in ANOVA are conducted before the F-statistic test and involve comparing the means of each group with the overall mean to identify significant differences.

B: A significant F-statistic does not tell us which
pair or pairs of groups differ (Ha: μ1 ≠ μ2 OR μ1 ≠ μ3 OR μ2 ≠ μ3) so we need to do a pairwise (post hoc - “after the fact”) comparison. Pairwise comparisons are independent t-tests with a slight difference and are only necessary if you determine there is a significant (reject the null hypothesis) outcome.

C: After determining a significant F-statistic in ANOVA, pairwise comparisons are not needed as it is assumed that all group means are equal, and no further analysis is required.

Answer 31

A significant F-statistic does not tell us which
pair or pairs of groups differ (Ha: μ1 ≠ μ2 OR μ1 ≠ μ3 OR μ2 ≠ μ3) so we need to do a pairwise (post hoc - “after the fact”) comparison. Pairwise comparisons are independent t-tests with a slight difference and are only necessary if you determine there is a significant (reject the null hypothesis) outcome.

> You will only perform this comparison after conducting an f-test
You’ll determine this pot hoc comparison in Jamovi
This takes us back to Chapter 10 because we need to use an independent-samples t-test
Focus on comparing the “standard errors”, “t-statistics,” and “p-values.”

Question 32

What is your conclusion about the comparison
between the Control condition and Illustrate Risks condition?

A. Significant different means
B. Nonsignificant different means

Answer 32

A. Significant different means

I WOULD ADD THIS PIC (SLIDE 65) TO YOUR CHEAT SHEET WITH THIS DESCRIPTION

> Line one: Control - Research (not significant - fail to reject the null hypothesis)

> Line two: Control - Show Risks (significant -reject the null hypothesis)

> Line three: Research - Show Risks (significant - reject the null hypothesis)

CONCLUSION:
> The F-test indicated that at least one of the
experimental conditions produced a difference
mean.
> The Illustrate Risks condition was significantly higher
(look at the means) then both the Control and Read
Research conditions (you know this because the mean difference is a large negative number and the order that you subtracted the interventions tells you which one was a higher vs. lower number).
> We fail to find a difference between the Read
Research and Control conditions

PUT ALL OF THIS ON YOUR CHEAT SHEET!!! I WOULD PROBABLY ADD A SCREENSHOT OF SLIDE 68 AS WELL FOR THE APA-STYLE TABLE

Question 33

YOU NEED TO KNOW HOW TO READ AN APA-STYLE RESULTS SUMMARY - PUT THIS ON YOUR CHEAT SHEET!!!!

Answer 33

Line 1 - Research question and tests used:
> We performed a one-factor ANOVA analysis to
assess differences among the three experimental
conditions.

Line 2 - Descriptive statistics:
> Table 1 gives the means and standard
deviations for each condition.

Line 3 - Statistical significance:
> The ANOVA analysis revealed statistically significant differences among the group means, F(2,338) = 7.76, p < .001.

Line 4 - Practical significance:
> Further, the effect size indicated that group differences explained approximately 4% of the variability in vaccination attitude scores, which is a small effect by conventional standards.

(You don’t need the following section if the ANOVA result is nonsignificant and there are no follow-up tests)

Line 5 - Pairwise (Post Hoc) Comparisons:
> We next performed follow-up tests to examine all
possible pairwise group differences.

Line 6 - Follow-up test conclusion:
> These tests indicated that the Illustrate Risks condition was significantly higher than the control group (p < .001)
and the Read Research condition (p = .021). Finally, the Read Research condition was not significantly different from the Control condition (p = .102)

Question 34

Jamovi functions for:

> Treatment group mean square (mean differences) on variance metric (between-group)
Residual (naturally occurring) mean square (differences) on variance metric (within-group)
Sum of squares for each of the above
Degrees of freedom for each of the above
F-statistic
η2 effect size
p-value
Post hoc test (pairwise comparisons)

PUT THIS ON CHEAT SHEET!!!

Answer 34

Step 1: Analyses, ANOVA, ANOVA

Step 2: Move whatever the intervention is to the “Fixed Factors” side and whatever the scores are to the “Dependent Variable” side.

Step 3: Under “Effect Size” click η2

Step 4: Click the drop-down arrow next to “Post Hoc Tests

Step 5: Move the intervention over to the right

Step 6: Under “Correction” select “No correction” and unselect “Tukey.”