SCC Chapter 5 – Designs That Use Both Control Groups and Pretests Flashcards Preview

ECPY 767 (Final) > SCC Chapter 5 – Designs That Use Both Control Groups and Pretests > Flashcards

Flashcards in SCC Chapter 5 – Designs That Use Both Control Groups and Pretests Deck (20):
1

Justification for use of pretest

• smaller differences on pretest = less likelihood of strong initial selection biases (though without random assignment we don’t know that confounds are unrelated to outcome
• helps with stat analyses, especially if measures are reliable

2

Untreated Control Group Design With Pretest and Posttest Samples

• nonequivalent comparison group design
• most common of all quasi-experiments

3

Design Map of Untreated Control Group Design With Pretest and Posttest Samples

Diagram = NR O1 X O2
NR O1 O2

4

Selection-instrumentation threat

difficulty measuring certain points in a scale precisely, or having some items weighted more heavily than others): more acute if groups are unequal

5

Selection-regression

groups selected are not equally matched due to performance (gifted children matched with non-gifted children in other group will bias results

6

Selection-history

an event occurred between pre- and post- treatment that biases results

7

Outcome 1: Both Groups Grow Apart in the Same Direction: selection-maturation

“fan-spread model” – standardizing scores makes “fan” disappear because the variance is equalized. These effects may be spurious if groups are unequal

8

Outcome 2: No Change in the Control Group

treatment group may be older or other maturation issues

9

Outcome 3: Initial Pretest Difference Favoring the Treatment Group That Diminish Over Time

superiority of tx group is diminished or eliminated at posttest

10

Outcome 4: Initial Pretest Differences Favoring the Control Group Diminish Over Time

this is desirable, for instance, if a school implements a tx program for underachieving students. Outcome is subject to scaling and history threats

11

Outcome 5: Outcomes that Cross Over in the Direction of Relationships

amenable to causal interpretation. Power to detect a statistically reliable interaction in this type of study is low. Should not rely on research design to obtain this type of result

12

5 Outcome Patterns that are Plausible for Different Result Scenarios

Outcome 1: Both Groups Grow Apart in the Same Direction: selection-maturation
Outcome 2: No Change in the Control Group
Outcome 3: Initial Pretest Difference Favoring the Treatment Group That Diminish Over Time
Outcome 4: Initial Pretest Differences Favoring the Control Group Diminish Over Time
Outcome 5: Outcomes that Cross Over in the Direction of Relationships

13

Ways to Improve the Untreated Control Group Design With Dependent Pretest and Posttest Samples

1. Using a Double Pretest
2. Using Switching Replications
3. Using a Reversed-Treatment Control Group
4. Matching Through Cohort Controls
5. Matching Through Cohort Controls by Adding Pretests
6. Improving Cohort Designs With a Nonequivalent DV
7. Combining Switching Replications With a Nonequivalent Control Group Design

14

Using a Double Pretest

understand biases pre-treatment (from pretest1 to pretest 2). Also permits assessment of a selection-maturation threat on the assumption that the rates between the 2 pretests will continue between the second pretest and posttest. This is testable ONLY for the untreated group. It can be difficult to get a 2nd pretest together

15

Using Switching Replications

Researcher administers tx at a later date to the group that initially served as a no treatment control. The second phase is not an exact replication. Obvious contextual differences between the groups (one receives treatment before the other, but they both receive O2 at same time). Problem: you can’t remove tx from the 1st group at O3

16

Using a Reversed-Treatment Control Group

X+ intended to produce an effect in one direction, while X- is intended to produce the opposite. Assumes that little historical or motivational change is taking place.

17

Matching Through Cohort Controls

Cohort = the successive groups that go through processes such as graduating from school, etc. cohorts can be useful as control groups if 1) one cohort experiences a given tx and ealier or later cohorts do no; 2) cohorts differ only in minor ways; 3) organizations insist that tx be given to everyone (making control impossible; 4) an org’s records can be used for constructing and comparing cohorts
Cohorts will never be as comparable as groups that are randomly assigned

18

Improving Cohort Controls by Adding Pretests

Compare cohort pretest means to assess for nonequivalence; Pretest increases statistical power by allowing use of within-subject error terms. Enables assessment of maturation and regression and enters into better statistical adjustment for group nonequivalence. History is a salient internal validity threat in this design

19

Improving Cohort Designs With a Nonequivalent DV

improve a study with a specific measure of DV

20

Combining Switching Replications With a Nonequivalent Control Group Design

introduce tx to part of the original control group, with other controls remaining untreated over this later time period (requires more than 2 groups). Can also reintroduce tx a second time to some of the original tx group to evaluate the benefits of additional treatment