Flashcards in SCC CH. 3 Construct Validity and External Validity Deck (26):
Three requirements of a theory of constructs
1) Operationalize each construct several ways within and across studies.
2) Probing the pattern match between the multivariate characteristic of instances and the characteristics of the target category.
3) Acknowledge legitimate debate about the quality of that match given the socially constructed nature of operations and constructs.
14 threats to construct validity
1) Inadequate Explication of Constructs
2) Construct Confounding
3) Mono-Operation Bias
4) Mono-Method Bias
5) Confounding Constructs with Levels of Constructs
6) Treatment Sensitive Factorial Structure
7) Reactive Self-Report Changes
8) Reactivity to the Experimental Situation
9) Experimenter Expectancies
10) Novelty and Disruption Effects
11) Compensatory Equalization
12) Compensatory Rivalry
13) Resentful Demoralization
14) Treatment Diffusion
1) Inadequate Explication of Constructs
Failure to adequately explicate constructs may lead to incorrect inferences about the relationship between operation and construct.
Operations usually involve more than one construct, and failure to describe all the constructs may result in incomplete construct inferences.
Any one operationalization of a construct both under represents the construct of interest and measures irrelevant constructs, complicating inference.
When all operationalizations use the same method (e.g. self-report), that method is part of the construct actually studied.
Confounding Constructs with Levels of Constructs
Inferences about the constructs that best represent study operations may fail to describe the limited levels of construct that were actually studied.
Treatment Sensitive Factorial Structure
The structure of a measure may change as a result of treatment, change that may be hidden in the same scoring is always used.
Reactive Self-Report Changes
Self-reports can be affected by the participant motivation to be in a treatment condition, motivation that can change after assignment is made.
Reactivity to the Experimental Situation
Participant responses reflect not just treatments and measures but also participants’ perceptions of the experimental situation, and those perceptions are part of the treatment construct actually tested
The experimenter can influence participant responses by conveying expectations about desirable responses, and those expectations are part of the treatment construct as actually tested
Novelty and Disruption Effects
Participants may respond unusually well to a novel innovation or unusually poorly to one that disrupts their routine, a response that must then be included as part of the treatment construct description
When treatment provides desirable goods or services, administrators, staff or constituents may provide compensatory goods or services to those not receiving treatment, and this action must then be included as part of the treatment construct description
Participants not receiving treatment may be motivated to show they can do as well as those receiving treatment, and this compensatory rivalry must then be included as part of the treatment construct description
Participants not receiving a desirable treatment may be so resentful or demoralized that they may respond more negatively than otherwise, and this resentful demoralization must then be included as part of the treatment construct description
Participants may receive services from a condition to which they were not assigned, making construct descriptions of both conditions more difficult.
What are 5 targets of generalization?
- Narrow to Broad
- Broad to Narrow
- At a similar level
- To a similar or different kind
- Random sample to population members
What 2 external validity questions can be asked about whether a causal relationship holds?
1) Over variations in persons, setting, treatments, and outcomes that were in the experiment.
2) For persons, setting, treatments, and outcomes that were not in the experiment.
What is secondary analysis?
Reanalyzing data from the experiment to see what happens to the causal relationship as the variable in question is varied.
Threats to external validity deal with interaction with...?
What are 5 threats to external validity?
1) Interaction of the Causal Relationship with Units
2) Interaction of the Causal Relationship Over Treatment Variations
3) Interaction of the Causal Relationship with outcomes
4) Interactions of the Causal Relationship with Settings
5) Context-Dependent Mediation
Interaction of the Causal Relationship with Units
An effect found with certain kinds of units might not hold if other kinds of units had been studied
Interaction of the Causal Relationship Over Treatment Variations
An effect found with one treatment variation might not hold with other variations of that treatment, or when that treatment is combined with other treatments, or when only part of that treatment is used
Interaction of the Causal Relationship with outcomes
An effect found on one kind of outcome observation may not hold if other outcome observations were used.
Interactions of the Causal Relationship with Settings
An effect found in one kind of setting may not hold if other kinds of settings were to be used