Threats to validity Flashcards
1
Q
Threats to Theoretical Validity
A
- the ‘problem’ not clearly formulated or articulated
- answering the wrong question
- answering a trivial question
- contribution to the literature not well established (ecological validity)
- rationale & logical reasoning inadequately explicated
- phenomena under investigation not clearly defined or explicated
- theory(ies) not delineated adequately (specification of central constructs and their interrelations; ‘atheoretical’ research, theorizing left implicit)
- statements, premises, or facts not supported adequately via references to empirical data or to theory
- equivocation of distinct constructs, terms, or relations
- inadequate test of theory or theorizing or not attempting to falsify theorizing
- introduction is methodology/statistically driven vs. theory driven
- inadequate conceptual integrity (theorizing does not incorporate all variables, constructs, and relations included in hypotheses and analyses)
- logical incoherence
2
Q
- the ‘problem’ not clearly formulated or articulated
- answering the wrong question
- answering a trivial question
- contribution to the literature not well established (ecological validity)
- rationale & logical reasoning inadequately explicated
- phenomena under investigation not clearly defined or explicated
- theory(ies) not delineated adequately (specification of central constructs and their interrelations; ‘atheoretical’ research, theorizing left implicit)
- statements, premises, or facts not supported adequately via references to empirical data or to theory
- equivocation of distinct constructs, terms, or relations
- inadequate test of theory or theorizing or not attempting to falsify theorizing
- introduction is methodology/statistically driven vs. theory driven
- inadequate conceptual integrity (theorizing does not incorporate all variables, constructs, and relations included in hypotheses and analyses)
- logical incoherence
A
Threats to Theoretical Validity
3
Q
Threats to Structural Validity
A
- mismatch of theorizing and hypotheses
- mismatch of a construct and its operational definitions
- mismatch of design-methods-procedures and analyses
- mismatch of population sampled with theorizing and hypotheses
- mismatch of sampling procedures with theorizing and hypotheses
4
Q
- mismatch of theorizing and hypotheses
- mismatch of a construct and its operational definitions
- mismatch of design-methods-procedures and analyses
- mismatch of population sampled with theorizing and hypotheses
- mismatch of sampling procedures with theorizing and hypotheses
A
Threats to Structural Validity
5
Q
Threats to Hypothesis Validity
A
- Inconsequential hypotheses (the extent to which hypotheses both corroborate one theory and falsify others)
- Ambiguous hypotheses (hypotheses are not specified, or if provided, the conditions under which hypotheses will fail or succeed are not delineated)
- Noncongruence of research and statistical hypotheses (incorrect statistical procedures or the statistical tests do not test the research hypotheses)
- Diffuse statistical hypotheses and tests (any combination of the following three)
- multiple statistical tests per hypothesis,
- using omnibus tests and subsequent follow-up or post hoc tests, or
- the statistical analyses include extraneous independent variables not specified in the hypotheses
6
Q
- Inconsequential hypotheses (the extent to which hypotheses both corroborate one theory and falsify others)
- Ambiguous hypotheses (hypotheses are not specified, or if provided, the conditions under which hypotheses will fail or succeed are not delineated)
- Noncongruence of research and statistical hypotheses (incorrect statistical procedures or the statistical tests do not test the research hypotheses)
- Diffuse statistical hypotheses and tests (any combination of the following three)
- multiple statistical tests per hypothesis,
- using omnibus tests and subsequent follow-up or post hoc tests, or
- the statistical analyses include extraneous independent variables not specified in the hypotheses
A
Threats to Hypothesis Validity
7
Q
Threats to Population Validity
A
- nonrandom sample
- inadequate sample description
- sample biases
- failure to use stratified sampling
- failure to test sample representativeness (e.g., respondents vs. nonrespondents)
- inadequate response rate
8
Q
- nonrandom sample
- inadequate sample description
- sample biases
- failure to use stratified sampling
- failure to test sample representativeness (e.g., respondents vs. nonrespondents)
- inadequate response rate
A
Threats to Population Validity
9
Q
Threats to Construct Validity
A
- inadequate explication of constructs
- inappropriate operationalization of construct
- mismatch of construct and operational definition (treatment, manipulation, measure)
- construct confounding and/or variable confounding
- inadequate operationalization of construct
- confounding constructs with restricted levels of a construct (e.g., restricted range)
- mono-method bias
- mono-operationalization bias
- reactivity to experimental situation (e.g., hypothesis guessing within treatments)
- evaluation apprehension
- experimenter expectancies (not blind)
- novelty and disruption effects
- restricted generalizability across constructs
- compensatory equalization of treatments
- rivalry by participants
- resentful demoralization
- diffusion of treatment
10
Q
- inadequate explication of constructs
- inappropriate operationalization of construct
- mismatch of construct and operational definition (treatment, manipulation, measure)
- construct confounding and/or variable confounding
- inadequate operationalization of construct
- confounding constructs with restricted levels of a construct (e.g., restricted range)
- mono-method bias
- mono-operationalization bias
- reactivity to experimental situation (e.g., hypothesis guessing within treatments)
- evaluation apprehension
- experimenter expectancies (not blind)
- novelty and disruption effects
- restricted generalizability across constructs
- compensatory equalization of treatments
- rivalry by participants
- resentful demoralization
- diffusion of treatment
A
Threats to Construct Validity
11
Q
Threats to Construct Validity – Measurement
A
- construct underrepresentation
- construct irrelevant variance
- content – evidence of content relevance, representativeness, & technical quality
- substantive – theoretical rationales for performance of assessment task and processes of assessment task
- structural – fidelity of scoring structure to structure of construct domain (structural fidelity)
- generalizability – of score properties and interpretations to and across groups, settings, & tasks & relationships, includes measurement error
- external – convergent & discriminant evidence, evidence of criterion relevance & applied utility
- consequential – value implications (social) of score interpretation, actual & potential consequences of test use, especially for invalidity related to bias, fairness, & distributive justice issues
12
Q
- construct underrepresentation
- construct irrelevant variance
- content – evidence of content relevance, representativeness, & technical quality
- substantive – theoretical rationales for performance of assessment task and processes of assessment task
- structural – fidelity of scoring structure to structure of construct domain (structural fidelity)
- generalizability – of score properties and interpretations to and across groups, settings, & tasks & relationships, includes measurement error
- external – convergent & discriminant evidence, evidence of criterion relevance & applied utility
- consequential – value implications (social) of score interpretation, actual & potential consequences of test use, especially for invalidity related to bias, fairness, & distributive justice issues
A
Threats to Construct Validity – Measurement
13
Q
Threats to Statistical Conclusion Validity
A
- failure to control adequately error rates
- inadequate statistical power (Type II error rate > .20)
- failure to perform an a priori statistical power analysis
- inflated experiment/study-wise Type II error rate
- inflated Type I error rate (> .10)
- inflated experiment/study-wise Type I error rates
- making “eye-balled” comparisons without performing statistical tests
- violation of assumptions or assumptions not tested for statistical procedures used
- non-normal data
- heterogeneity of variances (compound symmetry violated)
- auto-correlation, auto-regression
- nonindependence of data - observations (e.g., correlated error terms; some participants in more than one treatment condition)
- failure to define ‘meaningful’ effect size a priori
- inaccurate effect size estimates (e.g., unshrunken effect sizes)
- differential ceiling and floor effects (restricted range)
- irrelevancies in experimental setting
- confounded data
- nonrandomization (includes any of the following)
- nonrandomized administration of measures (sequence or order effects)
- nonrandom assignment of participants to groups, conditions, or treatments
- nonrandom assignment of experimenter-therapists to treatments (therapist effects)
- nonrandom assignment of treatments (e.g., as in multiple baseline designs)
- failing to test statistically the effectiveness of randomization procedures
- unreliability of treatment implementation
- measurement error (i.e., unreliability of measurement IVs and/or DVs)
14
Q
- failure to control adequately error rates
- inadequate statistical power (Type II error rate > .20)
- failure to perform an a priori statistical power analysis
- inflated experiment/study-wise Type II error rate
- inflated Type I error rate (> .10)
- inflated experiment/study-wise Type I error rates
- making “eye-balled” comparisons without performing statistical tests
- violation of assumptions or assumptions not tested for statistical procedures used
- non-normal data
- heterogeneity of variances (compound symmetry violated)
- auto-correlation, auto-regression
- nonindependence of data - observations (e.g., correlated error terms; some participants in more than one treatment condition)
- failure to define ‘meaningful’ effect size a priori
- inaccurate effect size estimates (e.g., unshrunken effect sizes)
- differential ceiling and floor effects (restricted range)
- irrelevancies in experimental setting
- confounded data
- nonrandomization (includes any of the following)
- nonrandomized administration of measures (sequence or order effects)
- nonrandom assignment of participants to groups, conditions, or treatments
- nonrandom assignment of experimenter-therapists to treatments (therapist effects)
- nonrandom assignment of treatments (e.g., as in multiple baseline designs)
- failing to test statistically the effectiveness of randomization procedures
- unreliability of treatment implementation
- measurement error (i.e., unreliability of measurement IVs and/or DVs)
A
Threats to Statistical Conclusion Validity
15
Q
Threats to Internal Validity
A
- ambiguity of causal direction / ambiguity of temporal precedence
- cohort effects (cross sectional data)
- inadequate comparison or control group(s)
- selection
- history
- maturation
- statistical regression
- differential attrition/mortality
- testing
- instrumentation
- additive and interactions effects of threats to internal validity
