SCC Chapter 5 – Designs That Use Both Control Groups and Pretests

Question 1

Justification for use of pretest

Answer 1

• smaller differences on pretest = less likelihood of strong initial selection biases (though without random assignment we don’t know that confounds are unrelated to outcome
• helps with stat analyses, especially if measures are reliable

Question 2

Untreated Control Group Design With Pretest and Posttest Samples

Answer 2

• nonequivalent comparison group design

* most common of all quasi-experiments

Question 3

Design Map of Untreated Control Group Design With Pretest and Posttest Samples

Answer 3

Diagram = NR O1 X O2

NR O1 O2

Question 4

Selection-instrumentation threat

Answer 4

difficulty measuring certain points in a scale precisely, or having some items weighted more heavily than others): more acute if groups are unequal

Question 5

Selection-regression

Answer 5

groups selected are not equally matched due to performance (gifted children matched with non-gifted children in other group will bias results

Question 6

Selection-history

Answer 6

an event occurred between pre- and post- treatment that biases results

Question 7

Outcome 1: Both Groups Grow Apart in the Same Direction: selection-maturation

Answer 7

“fan-spread model” – standardizing scores makes “fan” disappear because the variance is equalized. These effects may be spurious if groups are unequal

Question 8

Outcome 2: No Change in the Control Group

Answer 8

treatment group may be older or other maturation issues

Question 9

Outcome 3: Initial Pretest Difference Favoring the Treatment Group That Diminish Over Time

Answer 9

superiority of tx group is diminished or eliminated at posttest

Question 10

Outcome 4: Initial Pretest Differences Favoring the Control Group Diminish Over Time

Answer 10

this is desirable, for instance, if a school implements a tx program for underachieving students. Outcome is subject to scaling and history threats

Question 11

Outcome 5: Outcomes that Cross Over in the Direction of Relationships

Answer 11

amenable to causal interpretation. Power to detect a statistically reliable interaction in this type of study is low. Should not rely on research design to obtain this type of result

Question 12

5 Outcome Patterns that are Plausible for Different Result Scenarios

Answer 12

Outcome 1: Both Groups Grow Apart in the Same Direction: selection-maturation
Outcome 2: No Change in the Control Group
Outcome 3: Initial Pretest Difference Favoring the Treatment Group That Diminish Over Time
Outcome 4: Initial Pretest Differences Favoring the Control Group Diminish Over Time
Outcome 5: Outcomes that Cross Over in the Direction of Relationships

Question 13

Ways to Improve the Untreated Control Group Design With Dependent Pretest and Posttest Samples

Answer 13

1. Using a Double Pretest
2. Using Switching Replications
3. Using a Reversed-Treatment Control Group
4. Matching Through Cohort Controls
5. Matching Through Cohort Controls by Adding Pretests
6. Improving Cohort Designs With a Nonequivalent DV
7. Combining Switching Replications With a Nonequivalent Control Group Design

Question 14

Using a Double Pretest

Answer 14

understand biases pre-treatment (from pretest1 to pretest 2). Also permits assessment of a selection-maturation threat on the assumption that the rates between the 2 pretests will continue between the second pretest and posttest. This is testable ONLY for the untreated group. It can be difficult to get a 2nd pretest together

Question 15

Using Switching Replications

Answer 15

Researcher administers tx at a later date to the group that initially served as a no treatment control. The second phase is not an exact replication. Obvious contextual differences between the groups (one receives treatment before the other, but they both receive O2 at same time). Problem: you can’t remove tx from the 1st group at O3

Question 16

Using a Reversed-Treatment Control Group

Answer 16

X+ intended to produce an effect in one direction, while X- is intended to produce the opposite. Assumes that little historical or motivational change is taking place.

Question 17

Matching Through Cohort Controls

Answer 17

Cohort = the successive groups that go through processes such as graduating from school, etc. cohorts can be useful as control groups if 1) one cohort experiences a given tx and ealier or later cohorts do no; 2) cohorts differ only in minor ways; 3) organizations insist that tx be given to everyone (making control impossible; 4) an org’s records can be used for constructing and comparing cohorts
Cohorts will never be as comparable as groups that are randomly assigned

Question 18

Improving Cohort Controls by Adding Pretests

Answer 18

Compare cohort pretest means to assess for nonequivalence; Pretest increases statistical power by allowing use of within-subject error terms. Enables assessment of maturation and regression and enters into better statistical adjustment for group nonequivalence. History is a salient internal validity threat in this design

Question 19

Improving Cohort Designs With a Nonequivalent DV

Answer 19

improve a study with a specific measure of DV

Question 20

Combining Switching Replications With a Nonequivalent Control Group Design

Answer 20

introduce tx to part of the original control group, with other controls remaining untreated over this later time period (requires more than 2 groups). Can also reintroduce tx a second time to some of the original tx group to evaluate the benefits of additional treatment