Threats to validity Flashcards Preview

ECPY 767 (Final) > Threats to validity > Flashcards

Flashcards in Threats to validity Deck (18):
1

Threats to Theoretical Validity

* the ‘problem’ not clearly formulated or articulated
* answering the wrong question
* answering a trivial question
* contribution to the literature not well established (ecological validity)
* rationale & logical reasoning inadequately explicated
* phenomena under investigation not clearly defined or explicated
* theory(ies) not delineated adequately (specification of central constructs and their interrelations; ‘atheoretical’ research, theorizing left implicit)
* statements, premises, or facts not supported adequately via references to empirical data or to theory
* equivocation of distinct constructs, terms, or relations
* inadequate test of theory or theorizing or not attempting to falsify theorizing
* introduction is methodology/statistically driven vs. theory driven
* inadequate conceptual integrity (theorizing does not incorporate all variables, constructs, and relations included in hypotheses and analyses)
* logical incoherence

2

* the ‘problem’ not clearly formulated or articulated
* answering the wrong question
* answering a trivial question
* contribution to the literature not well established (ecological validity)
* rationale & logical reasoning inadequately explicated
* phenomena under investigation not clearly defined or explicated
* theory(ies) not delineated adequately (specification of central constructs and their interrelations; ‘atheoretical’ research, theorizing left implicit)
* statements, premises, or facts not supported adequately via references to empirical data or to theory
* equivocation of distinct constructs, terms, or relations
* inadequate test of theory or theorizing or not attempting to falsify theorizing
* introduction is methodology/statistically driven vs. theory driven
* inadequate conceptual integrity (theorizing does not incorporate all variables, constructs, and relations included in hypotheses and analyses)
* logical incoherence

Threats to Theoretical Validity

3

Threats to Structural Validity

* mismatch of theorizing and hypotheses
* mismatch of a construct and its operational definitions
* mismatch of design-methods-procedures and analyses
* mismatch of population sampled with theorizing and hypotheses
* mismatch of sampling procedures with theorizing and hypotheses

4

* mismatch of theorizing and hypotheses
* mismatch of a construct and its operational definitions
* mismatch of design-methods-procedures and analyses
* mismatch of population sampled with theorizing and hypotheses
* mismatch of sampling procedures with theorizing and hypotheses

Threats to Structural Validity

5

Threats to Hypothesis Validity

* Inconsequential hypotheses (the extent to which hypotheses both corroborate one theory and falsify others)
* Ambiguous hypotheses (hypotheses are not specified, or if provided, the conditions under which hypotheses will fail or succeed are not delineated)
* Noncongruence of research and statistical hypotheses (incorrect statistical procedures or the statistical tests do not test the research hypotheses)
* Diffuse statistical hypotheses and tests (any combination of the following three)
* multiple statistical tests per hypothesis,
* using omnibus tests and subsequent follow-up or post hoc tests, or
* the statistical analyses include extraneous independent variables not specified in the hypotheses

6

* Inconsequential hypotheses (the extent to which hypotheses both corroborate one theory and falsify others)
* Ambiguous hypotheses (hypotheses are not specified, or if provided, the conditions under which hypotheses will fail or succeed are not delineated)
* Noncongruence of research and statistical hypotheses (incorrect statistical procedures or the statistical tests do not test the research hypotheses)
* Diffuse statistical hypotheses and tests (any combination of the following three)
* multiple statistical tests per hypothesis,
* using omnibus tests and subsequent follow-up or post hoc tests, or
* the statistical analyses include extraneous independent variables not specified in the hypotheses

Threats to Hypothesis Validity

7

Threats to Population Validity

* nonrandom sample
* inadequate sample description
* sample biases
* failure to use stratified sampling
* failure to test sample representativeness (e.g., respondents vs. nonrespondents)
* inadequate response rate

8

* nonrandom sample
* inadequate sample description
* sample biases
* failure to use stratified sampling
* failure to test sample representativeness (e.g., respondents vs. nonrespondents)
* inadequate response rate

Threats to Population Validity

9

Threats to Construct Validity

* inadequate explication of constructs
* inappropriate operationalization of construct
* mismatch of construct and operational definition (treatment, manipulation, measure)
* construct confounding and/or variable confounding
* inadequate operationalization of construct
* confounding constructs with restricted levels of a construct (e.g., restricted range)
* mono-method bias
* mono-operationalization bias
* reactivity to experimental situation (e.g., hypothesis guessing within treatments)
* evaluation apprehension
* experimenter expectancies (not blind)
* novelty and disruption effects
* restricted generalizability across constructs
* compensatory equalization of treatments
* rivalry by participants
* resentful demoralization
* diffusion of treatment

10

* inadequate explication of constructs
* inappropriate operationalization of construct
* mismatch of construct and operational definition (treatment, manipulation, measure)
* construct confounding and/or variable confounding
* inadequate operationalization of construct
* confounding constructs with restricted levels of a construct (e.g., restricted range)
* mono-method bias
* mono-operationalization bias
* reactivity to experimental situation (e.g., hypothesis guessing within treatments)
* evaluation apprehension
* experimenter expectancies (not blind)
* novelty and disruption effects
* restricted generalizability across constructs
* compensatory equalization of treatments
* rivalry by participants
* resentful demoralization
* diffusion of treatment

Threats to Construct Validity

11

Threats to Construct Validity – Measurement

* construct underrepresentation
* construct irrelevant variance
* content – evidence of content relevance, representativeness, & technical quality
* substantive – theoretical rationales for performance of assessment task and processes of assessment task
* structural – fidelity of scoring structure to structure of construct domain (structural fidelity)
* generalizability – of score properties and interpretations to and across groups, settings, & tasks & relationships, includes measurement error
* external – convergent & discriminant evidence, evidence of criterion relevance & applied utility
* consequential – value implications (social) of score interpretation, actual & potential consequences of test use, especially for invalidity related to bias, fairness, & distributive justice issues

12

* construct underrepresentation
* construct irrelevant variance
* content – evidence of content relevance, representativeness, & technical quality
* substantive – theoretical rationales for performance of assessment task and processes of assessment task
* structural – fidelity of scoring structure to structure of construct domain (structural fidelity)
* generalizability – of score properties and interpretations to and across groups, settings, & tasks & relationships, includes measurement error
* external – convergent & discriminant evidence, evidence of criterion relevance & applied utility
* consequential – value implications (social) of score interpretation, actual & potential consequences of test use, especially for invalidity related to bias, fairness, & distributive justice issues

Threats to Construct Validity – Measurement

13

Threats to Statistical Conclusion Validity

* failure to control adequately error rates
* inadequate statistical power (Type II error rate > .20)
* failure to perform an a priori statistical power analysis
* inflated experiment/study-wise Type II error rate
* inflated Type I error rate (> .10)
* inflated experiment/study-wise Type I error rates
* making “eye-balled” comparisons without performing statistical tests
* violation of assumptions or assumptions not tested for statistical procedures used
* non-normal data
* heterogeneity of variances (compound symmetry violated)
* auto-correlation, auto-regression
* nonindependence of data - observations (e.g., correlated error terms; some participants in more than one treatment condition)
* failure to define 'meaningful' effect size a priori
* inaccurate effect size estimates (e.g., unshrunken effect sizes)
* differential ceiling and floor effects (restricted range)
* irrelevancies in experimental setting
* confounded data
* nonrandomization (includes any of the following)
* nonrandomized administration of measures (sequence or order effects)
* nonrandom assignment of participants to groups, conditions, or treatments
* nonrandom assignment of experimenter-therapists to treatments (therapist effects)
* nonrandom assignment of treatments (e.g., as in multiple baseline designs)
* failing to test statistically the effectiveness of randomization procedures
* unreliability of treatment implementation
* measurement error (i.e., unreliability of measurement IVs and/or DVs)

14

* failure to control adequately error rates
* inadequate statistical power (Type II error rate > .20)
* failure to perform an a priori statistical power analysis
* inflated experiment/study-wise Type II error rate
* inflated Type I error rate (> .10)
* inflated experiment/study-wise Type I error rates
* making “eye-balled” comparisons without performing statistical tests
* violation of assumptions or assumptions not tested for statistical procedures used
* non-normal data
* heterogeneity of variances (compound symmetry violated)
* auto-correlation, auto-regression
* nonindependence of data - observations (e.g., correlated error terms; some participants in more than one treatment condition)
* failure to define 'meaningful' effect size a priori
* inaccurate effect size estimates (e.g., unshrunken effect sizes)
* differential ceiling and floor effects (restricted range)
* irrelevancies in experimental setting
* confounded data
* nonrandomization (includes any of the following)
* nonrandomized administration of measures (sequence or order effects)
* nonrandom assignment of participants to groups, conditions, or treatments
* nonrandom assignment of experimenter-therapists to treatments (therapist effects)
* nonrandom assignment of treatments (e.g., as in multiple baseline designs)
* failing to test statistically the effectiveness of randomization procedures
* unreliability of treatment implementation
* measurement error (i.e., unreliability of measurement IVs and/or DVs)

Threats to Statistical Conclusion Validity

15

Threats to Internal Validity

* ambiguity of causal direction / ambiguity of temporal precedence
* cohort effects (cross sectional data)
* inadequate comparison or control group(s)
* selection
* history
* maturation
* statistical regression
* differential attrition/mortality
* testing
* instrumentation
* additive and interactions effects of threats to internal validity

16

* ambiguity of causal direction / ambiguity of temporal precedence
* cohort effects (cross sectional data)
* inadequate comparison or control group(s)
* selection
* history
* maturation
* statistical regression
* differential attrition/mortality
* testing
* instrumentation
* additive and interactions effects of threats to internal validity

Threats to Internal Validity

17

Threats to External Validity

* interaction of theorizing and observed relations (see Forsyth & Strong, 1986)
* interaction of causal relations with units (selection or participants)
* interaction of causal relations with treatment variations
* interaction of causal relations with outcomes
* interaction of causal relations with setting
* context dependent mediation or moderation
* degree of analogue (i.e., degree procedures-methods-setting are removed from phenomena of interest)

18

* interaction of theorizing and observed relations (see Forsyth & Strong, 1986)
* interaction of causal relations with units (selection or participants)
* interaction of causal relations with treatment variations
* interaction of causal relations with outcomes
* interaction of causal relations with setting
* context dependent mediation or moderation
* degree of analogue (i.e., degree procedures-methods-setting are removed from phenomena of interest)

Threats to External Validity