SCC CH. 3 Construct Validity and External Validity Flashcards Preview

ECPY 767 (Final) > SCC CH. 3 Construct Validity and External Validity > Flashcards

Flashcards in SCC CH. 3 Construct Validity and External Validity Deck (26):
1

Three requirements of a theory of constructs

1) Operationalize each construct several ways within and across studies.
2) Probing the pattern match between the multivariate characteristic of instances and the characteristics of the target category.
3) Acknowledge legitimate debate about the quality of that match given the socially constructed nature of operations and constructs.

2

14 threats to construct validity

1) Inadequate Explication of Constructs
2) Construct Confounding
3) Mono-Operation Bias
4) Mono-Method Bias
5) Confounding Constructs with Levels of Constructs
6) Treatment Sensitive Factorial Structure
7) Reactive Self-Report Changes
8) Reactivity to the Experimental Situation
9) Experimenter Expectancies
10) Novelty and Disruption Effects
11) Compensatory Equalization
12) Compensatory Rivalry
13) Resentful Demoralization
14) Treatment Diffusion

3

1) Inadequate Explication of Constructs

Failure to adequately explicate constructs may lead to incorrect inferences about the relationship between operation and construct.

4

Construct Confounding

Operations usually involve more than one construct, and failure to describe all the constructs may result in incomplete construct inferences.

5

Mono-Operation Bias

Any one operationalization of a construct both under represents the construct of interest and measures irrelevant constructs, complicating inference.

6

Mono-Method Bias

When all operationalizations use the same method (e.g. self-report), that method is part of the construct actually studied.

7

Confounding Constructs with Levels of Constructs

Inferences about the constructs that best represent study operations may fail to describe the limited levels of construct that were actually studied.

8

Treatment Sensitive Factorial Structure

The structure of a measure may change as a result of treatment, change that may be hidden in the same scoring is always used.

9

Reactive Self-Report Changes

Self-reports can be affected by the participant motivation to be in a treatment condition, motivation that can change after assignment is made.

10

Reactivity to the Experimental Situation

Participant responses reflect not just treatments and measures but also participants’ perceptions of the experimental situation, and those perceptions are part of the treatment construct actually tested

11

Experimenter Expectancies

The experimenter can influence participant responses by conveying expectations about desirable responses, and those expectations are part of the treatment construct as actually tested

12

Novelty and Disruption Effects

Participants may respond unusually well to a novel innovation or unusually poorly to one that disrupts their routine, a response that must then be included as part of the treatment construct description

13

Compensatory Equalization

When treatment provides desirable goods or services, administrators, staff or constituents may provide compensatory goods or services to those not receiving treatment, and this action must then be included as part of the treatment construct description

14

Compensatory Rivalry

Participants not receiving treatment may be motivated to show they can do as well as those receiving treatment, and this compensatory rivalry must then be included as part of the treatment construct description

15

Resentful Demoralization

Participants not receiving a desirable treatment may be so resentful or demoralized that they may respond more negatively than otherwise, and this resentful demoralization must then be included as part of the treatment construct description

16

Treatment Diffusion

Participants may receive services from a condition to which they were not assigned, making construct descriptions of both conditions more difficult.

17

What are 5 targets of generalization?

​- Narrow to Broad
​- Broad to Narrow
​- At a similar level
​- To a similar or different kind
​- Random sample to population members

18

What 2 external validity questions can be asked about whether a causal relationship holds?

1) Over variations in persons, setting, treatments, and outcomes that were in the experiment.
2) For persons, setting, treatments, and outcomes that were not in the experiment.

19

What is secondary analysis?

Reanalyzing data from the experiment to see what happens to the causal relationship as the variable in question is varied.

20

Threats to external validity deal with interaction with...?

o Units
o Treatments
o Outcomes
o Settings

21

What are 5 threats to external validity?

1) Interaction of the Causal Relationship with Units
2) Interaction of the Causal Relationship Over Treatment Variations
3) Interaction of the Causal Relationship with outcomes
4) Interactions of the Causal Relationship with Settings
5) Context-Dependent Mediation

22

Interaction of the Causal Relationship with Units

An effect found with certain kinds of units might not hold if other kinds of units had been studied

23

Interaction of the Causal Relationship Over Treatment Variations

An effect found with one treatment variation might not hold with other variations of that treatment, or when that treatment is combined with other treatments, or when only part of that treatment is used

24

Interaction of the Causal Relationship with outcomes

An effect found on one kind of outcome observation may not hold if other outcome observations were used.

25

Interactions of the Causal Relationship with Settings

An effect found in one kind of setting may not hold if other kinds of settings were to be used

26

Context-Dependent Mediation

An explanatory mediator of causal relationship in one context may not mediate in another context