Lesson 11

Question 1

What is Evaluation Design?

Answer 1

AKA: Research Design. It is the plan established for data collection or hypothesis testing

Question 2

What is the Evaluation Design attempting to address?

Answer 2

Causality

Question 3

What are 3 design characteristics to improving causality:

Answer 3

1. Comparison/control group similar to the
   experimental/exposed group
2. Measurement of outcome variables before and after the intervention
3. Minimal threats to internal and external validity

Question 4

R- represents

Answer 4

Random assignment of an evaluation study participant (or unit) to a group

Question 5

E- represents

Answer 5

Experimental (intervention or treatment) group

 E1, E2, E3, …., En indicates planned exposure of the group to different intervention procedures

Question 6

C- represents

Answer 6

Control (equivalent) group established only by random assignment
 Indicates no exposure to an intervention or exposure to a minimum or standard intervention procedures

Question 7

C underlined -represents

Answer 7

Comparison group established through any method other than randomization

Question 8

X represents

Answer 8

Intervention procedures applied to an E (experimental) group

 X1, X2, X3,…., Xn indicates an intervention consisting of multiple, different procedures

Question 9

O represents

Answer 9

An observation or measurement to collect data. Including:
 Tests,  Interviews,  Visual or audio ratings, or  Record reviews
O1, O2, O3,…., On indicates multiple measurements at different times

Question 10

What is a Non-Experimental Design?

Answer 10

Includes one experimental group (E) but no control group (C) or comparison group (C). Weakest of the three research designs
 In regards to internal validity and causation

Question 11

What is a Quasi-Experimental Design?

Answer 11

Includes an experimental group (E) and a comparison group (C) created by methods other than random assignment
 Includes observations of both groups, both prior to and after application of intervention (X) procedures
 Multiple ‘waves’ of data collection  May yield interpretable and supportive evidence of behavioral impact  Asserts varying degrees of control over several biases
 But usually does not account for all biases of internal validity

Question 12

What is an Experimental Design?

Answer 12

Includes random assignment of experimental (E) and a control (C) group
 Observations of both groups, prior to and after application of the intervention (X) procedures
 Yields the most interpretable and defensible evidence of behavioral impact
 Strongest (Most rigorous, ‘Gold standard’)
 Asserts the greatest degree of control over biasing factors that compromise internal validity of results

Question 13

What are the most common Research and Evaluation Designs?

Answer 13

 One-Shot Case Study 
 Retrospective Pretest 
 One-group Pretest-Posttest Design 
 Time Series Design 
 Pretest-Posttest Control-Group Design 
 Case Study Design

Question 14

Explain a One-Shot Case Study

Answer 14

 X——— 0
Survey of program participants after participation in the program
 Absolute simplest form of outcome evaluation

Question 15

What are the Primary Weaknesses of a One-Shot Case Study?

Answer 15

Does not demonstrate change during or as a result of the program
Best if used to answer descriptive research questions  Ex: Effects of a tornado on a community
 Interviews will provide good information, but hard to establish observations are due to the disaster

Question 16

Explain a RETROSPECTIVE PRETEST

Answer 16

X——— 0
Survey of program participants after participation in the program
 However, asks respondents to recall information before and after the intervention

Question 17

What is a weakness of a Retrospective Pretest?

Answer 17

Recall bias

 Relies on memory

Question 18

Explain the One Group Pretest-Posttest Design

Answer 18

01—–X—–02
Basic in design
Evaluator collects data before and after the program is conducted

Question 19

What is the primary weakness of the One Group Pretest-Posttest Design?

Answer 19

Potential selection biases
 Attrition
Potential measurement bias
 Poor validity  Poor reliability

Question 20

What is INTERNAL VALIDITY?

Answer 20

 Validity is synonymous with accuracy

 Degree to which measured change can be attributed to the program

Question 21

What is EXTERNAL VALIDITY?

Answer 21

Generalizability.

Extent to which the program can produce similar effects in other populations

Question 22

What are two Non-program related sources of bias?

Answer 22

 Maturation  Historical Bias

Question 23

What are three Program related sources of bias?

Answer 23

 Attrition  Selection Bias  Measurement Bias

Question 24

How can we increase internal validity?

Answer 24

 Observe participants at additional times before and after a program (multiple waves)  Was change random or systematic?
 Observe additional natural groups who have no exposure to the program (control /comparison group)
 In other words, use a quasi-experimental or experimental design

Question 25

How to demonstrate causality?

Answer 25

1. The cause precedes the supposed effect in time 2. The cause covaries with the effect  More of the cause = More of the effect. Vice versa 3. No viable explanation of the effect can be found except for the assumed cause

Question 26

Explain the TIME SERIES DESIGN

Answer 26

Evaluator gathers data at multiple times before and after the program is delivered

Question 27

When is the Time Series Design useful?

Answer 27

``` Useful in examining the effects of a new health- related policy Minimum Legal Drinking Age Seatbelt law Raising cigarette taxes Mandatory to wear motorcycle helmets ```

Question 28

Explain the MULTIPLE TIME SERIES DESIGN

Answer 28

The addition of the C group may strengthen the program’s control over the primary three biases

Question 29

What are limitations of the multiple time series design?

Answer 29

 Need the same multiple observations of C group before, during, and after the program  A lot of people and a lot of observations  Time and $$ intensive

Question 30

NON-EQUIVALENT CONTROL-GROUP DESIGN

Answer 30

 Pretest & Posttest Comparison Group Design:  01-----X-----02  01----- -----02 Quasi Experimental Design ADD R and becomes Experimental

Question 31

What are several things to remember when implementing a Non-Equivalent Control Group Design?

Answer 31

 Standardized assessments at baseline and follow- up are conducted of all E and C / C group participants  Two groups should not be significantly different at baseline for any IVs or DVs that are predictors of impact  Assuming program fidelity, this design produces good control over three major biases

Question 32

Isolation of the effects of the program from other plausible interpretations the Non Equivalent Control Group Design by?

Answer 32

 Maturation: Groups tested at the same time (same amount of time to mature)  Historical Bias: Groups are equally exposed affected by historical events  Testing Effects: Both groups were tested twice (both become more familiar)  Attrition: Can compare to see if both rates are similar?