Flashcards in Threats to validity Deck (18):

1

## Threats to Theoretical Validity

###
* the ‘problem’ not clearly formulated or articulated

* answering the wrong question

* answering a trivial question

* contribution to the literature not well established (ecological validity)

* rationale & logical reasoning inadequately explicated

* phenomena under investigation not clearly defined or explicated

* theory(ies) not delineated adequately (specification of central constructs and their interrelations; ‘atheoretical’ research, theorizing left implicit)

* statements, premises, or facts not supported adequately via references to empirical data or to theory

* equivocation of distinct constructs, terms, or relations

* inadequate test of theory or theorizing or not attempting to falsify theorizing

* introduction is methodology/statistically driven vs. theory driven

* inadequate conceptual integrity (theorizing does not incorporate all variables, constructs, and relations included in hypotheses and analyses)

* logical incoherence

2

##
* the ‘problem’ not clearly formulated or articulated

* answering the wrong question

* answering a trivial question

* contribution to the literature not well established (ecological validity)

* rationale & logical reasoning inadequately explicated

* phenomena under investigation not clearly defined or explicated

* theory(ies) not delineated adequately (specification of central constructs and their interrelations; ‘atheoretical’ research, theorizing left implicit)

* statements, premises, or facts not supported adequately via references to empirical data or to theory

* equivocation of distinct constructs, terms, or relations

* inadequate test of theory or theorizing or not attempting to falsify theorizing

* introduction is methodology/statistically driven vs. theory driven

* inadequate conceptual integrity (theorizing does not incorporate all variables, constructs, and relations included in hypotheses and analyses)

* logical incoherence

### Threats to Theoretical Validity

3

## Threats to Structural Validity

###
* mismatch of theorizing and hypotheses

* mismatch of a construct and its operational definitions

* mismatch of design-methods-procedures and analyses

* mismatch of population sampled with theorizing and hypotheses

* mismatch of sampling procedures with theorizing and hypotheses

4

##
* mismatch of theorizing and hypotheses

* mismatch of a construct and its operational definitions

* mismatch of design-methods-procedures and analyses

* mismatch of population sampled with theorizing and hypotheses

* mismatch of sampling procedures with theorizing and hypotheses

### Threats to Structural Validity

5

## Threats to Hypothesis Validity

###
* Inconsequential hypotheses (the extent to which hypotheses both corroborate one theory and falsify others)

* Ambiguous hypotheses (hypotheses are not specified, or if provided, the conditions under which hypotheses will fail or succeed are not delineated)

* Noncongruence of research and statistical hypotheses (incorrect statistical procedures or the statistical tests do not test the research hypotheses)

* Diffuse statistical hypotheses and tests (any combination of the following three)

* multiple statistical tests per hypothesis,

* using omnibus tests and subsequent follow-up or post hoc tests, or

* the statistical analyses include extraneous independent variables not specified in the hypotheses

6

##
* Inconsequential hypotheses (the extent to which hypotheses both corroborate one theory and falsify others)

* Ambiguous hypotheses (hypotheses are not specified, or if provided, the conditions under which hypotheses will fail or succeed are not delineated)

* Noncongruence of research and statistical hypotheses (incorrect statistical procedures or the statistical tests do not test the research hypotheses)

* Diffuse statistical hypotheses and tests (any combination of the following three)

* multiple statistical tests per hypothesis,

* using omnibus tests and subsequent follow-up or post hoc tests, or

* the statistical analyses include extraneous independent variables not specified in the hypotheses

### Threats to Hypothesis Validity

7

## Threats to Population Validity

###
* nonrandom sample

* inadequate sample description

* sample biases

* failure to use stratified sampling

* failure to test sample representativeness (e.g., respondents vs. nonrespondents)

* inadequate response rate

8

##
* nonrandom sample

* inadequate sample description

* sample biases

* failure to use stratified sampling

* failure to test sample representativeness (e.g., respondents vs. nonrespondents)

* inadequate response rate

### Threats to Population Validity

9

## Threats to Construct Validity

###
* inadequate explication of constructs

* inappropriate operationalization of construct

* mismatch of construct and operational definition (treatment, manipulation, measure)

* construct confounding and/or variable confounding

* inadequate operationalization of construct

* confounding constructs with restricted levels of a construct (e.g., restricted range)

* mono-method bias

* mono-operationalization bias

* reactivity to experimental situation (e.g., hypothesis guessing within treatments)

* evaluation apprehension

* experimenter expectancies (not blind)

* novelty and disruption effects

* restricted generalizability across constructs

* compensatory equalization of treatments

* rivalry by participants

* resentful demoralization

* diffusion of treatment

10

##
* inadequate explication of constructs

* inappropriate operationalization of construct

* mismatch of construct and operational definition (treatment, manipulation, measure)

* construct confounding and/or variable confounding

* inadequate operationalization of construct

* confounding constructs with restricted levels of a construct (e.g., restricted range)

* mono-method bias

* mono-operationalization bias

* reactivity to experimental situation (e.g., hypothesis guessing within treatments)

* evaluation apprehension

* experimenter expectancies (not blind)

* novelty and disruption effects

* restricted generalizability across constructs

* compensatory equalization of treatments

* rivalry by participants

* resentful demoralization

* diffusion of treatment

### Threats to Construct Validity

11

## Threats to Construct Validity – Measurement

###
* construct underrepresentation

* construct irrelevant variance

* content – evidence of content relevance, representativeness, & technical quality

* substantive – theoretical rationales for performance of assessment task and processes of assessment task

* structural – fidelity of scoring structure to structure of construct domain (structural fidelity)

* generalizability – of score properties and interpretations to and across groups, settings, & tasks & relationships, includes measurement error

* external – convergent & discriminant evidence, evidence of criterion relevance & applied utility

* consequential – value implications (social) of score interpretation, actual & potential consequences of test use, especially for invalidity related to bias, fairness, & distributive justice issues

12

##
* construct underrepresentation

* construct irrelevant variance

* content – evidence of content relevance, representativeness, & technical quality

* substantive – theoretical rationales for performance of assessment task and processes of assessment task

* structural – fidelity of scoring structure to structure of construct domain (structural fidelity)

* generalizability – of score properties and interpretations to and across groups, settings, & tasks & relationships, includes measurement error

* external – convergent & discriminant evidence, evidence of criterion relevance & applied utility

* consequential – value implications (social) of score interpretation, actual & potential consequences of test use, especially for invalidity related to bias, fairness, & distributive justice issues

### Threats to Construct Validity – Measurement

13

## Threats to Statistical Conclusion Validity

###
* failure to control adequately error rates

* inadequate statistical power (Type II error rate > .20)

* failure to perform an a priori statistical power analysis

* inflated experiment/study-wise Type II error rate

* inflated Type I error rate (> .10)

* inflated experiment/study-wise Type I error rates

* making “eye-balled” comparisons without performing statistical tests

* violation of assumptions or assumptions not tested for statistical procedures used

* non-normal data

* heterogeneity of variances (compound symmetry violated)

* auto-correlation, auto-regression

* nonindependence of data - observations (e.g., correlated error terms; some participants in more than one treatment condition)

* failure to define 'meaningful' effect size a priori

* inaccurate effect size estimates (e.g., unshrunken effect sizes)

* differential ceiling and floor effects (restricted range)

* irrelevancies in experimental setting

* confounded data

* nonrandomization (includes any of the following)

* nonrandomized administration of measures (sequence or order effects)

* nonrandom assignment of participants to groups, conditions, or treatments

* nonrandom assignment of experimenter-therapists to treatments (therapist effects)

* nonrandom assignment of treatments (e.g., as in multiple baseline designs)

* failing to test statistically the effectiveness of randomization procedures

* unreliability of treatment implementation

* measurement error (i.e., unreliability of measurement IVs and/or DVs)

14

##
* failure to control adequately error rates

* inadequate statistical power (Type II error rate > .20)

* failure to perform an a priori statistical power analysis

* inflated experiment/study-wise Type II error rate

* inflated Type I error rate (> .10)

* inflated experiment/study-wise Type I error rates

* making “eye-balled” comparisons without performing statistical tests

* violation of assumptions or assumptions not tested for statistical procedures used

* non-normal data

* heterogeneity of variances (compound symmetry violated)

* auto-correlation, auto-regression

* nonindependence of data - observations (e.g., correlated error terms; some participants in more than one treatment condition)

* failure to define 'meaningful' effect size a priori

* inaccurate effect size estimates (e.g., unshrunken effect sizes)

* differential ceiling and floor effects (restricted range)

* irrelevancies in experimental setting

* confounded data

* nonrandomization (includes any of the following)

* nonrandomized administration of measures (sequence or order effects)

* nonrandom assignment of participants to groups, conditions, or treatments

* nonrandom assignment of experimenter-therapists to treatments (therapist effects)

* nonrandom assignment of treatments (e.g., as in multiple baseline designs)

* failing to test statistically the effectiveness of randomization procedures

* unreliability of treatment implementation

* measurement error (i.e., unreliability of measurement IVs and/or DVs)

### Threats to Statistical Conclusion Validity

15

## Threats to Internal Validity

###
* ambiguity of causal direction / ambiguity of temporal precedence

* cohort effects (cross sectional data)

* inadequate comparison or control group(s)

* selection

* history

* maturation

* statistical regression

* differential attrition/mortality

* testing

* instrumentation

* additive and interactions effects of threats to internal validity

16

##
* ambiguity of causal direction / ambiguity of temporal precedence

* cohort effects (cross sectional data)

* inadequate comparison or control group(s)

* selection

* history

* maturation

* statistical regression

* differential attrition/mortality

* testing

* instrumentation

* additive and interactions effects of threats to internal validity

### Threats to Internal Validity

17

## Threats to External Validity

###
* interaction of theorizing and observed relations (see Forsyth & Strong, 1986)

* interaction of causal relations with units (selection or participants)

* interaction of causal relations with treatment variations

* interaction of causal relations with outcomes

* interaction of causal relations with setting

* context dependent mediation or moderation

* degree of analogue (i.e., degree procedures-methods-setting are removed from phenomena of interest)

18