Validity Flashcards
(19 cards)
Does a questionnaire actually measure what it’s proponents claim it does?
Unscientific to presume validity
Face validity can deceive
Can’t rely on authority
Precondition validity
Are preconditions for validity met?
Discriminations, reliability, structure
Preconditions necessary but not sufficient for validity
Construct validity
Corresponse
Perfect correlation between ideal population score and measured score on your test
Sources of invalidity
Systematic error
Random error
Systematic error
Potentially knowable bias pushing scores one way or another
Directional confound
Random error
Lots of unknown miscellaneous influences, pushing scores every which way
Jittery noise
1) item design
2) item analysis
3) reliability analysis
4) factor analysis
5) scale validation
Item design
Open ended questions
Qualitative info
Can be quantitatively coded after
Labor-intensive
Bottom-up
Close ended: respondent quantifies something, relatively efficient, top-down
Items should be simple and non-biased, and try cover fully the construct validity
Oxford Capacity Analysis = bad example, personality test
Scaling = put number on an item, convert psychological content to a number, labels on every number point to standardise meaning,
Neutral/uncertain response option, can increase accuracy/laziness
Numbers of forward and recovers score itllas
Item analysis
Discrimination is good
Levels for analysis = 6
Reliability analysis
Internal consistency
Internal consistency
Consistency between items
Internal consistency is overall form of reliability
Test-retest reliability
Consistency over time
Why does scale reliability matter?
It’s a precondition for validity (must be consistent to be true)BUT consistent story can also be false
Assume two fully reliable scales, assume constructs correlate at p=0.5, samples estimate r=0.5, this is an unbiased estimate, but if sample estimates r=0.25, this is clearly an underestimate
How should items relate to one another?
Related = on same team
Distinct = in different positions
Should intercorrelate well but not perfectly (bloated specific), coverage (content validity) matters
Scale level index
Overall internal consistency(alpha)
Average of all possible split-half correlations
Lower-bound estimate: could be higher
Increases with number of items
Lower alphas reduce possible correlations
A>0.6 is minimal, a>0.7 is poor, a>0.8 is average?? Etc
Factor analysis
Factor is a single underlying dimension (suggested but not guaranteed by high scale reliability)
Possible to have two or more factors underlying a highly reliable scales
To find fa, ask “where do the correlations clump?”
FA factor extraction method
Principal Axes Factoring (PAF),
Maximum Likelihood (ML)
Factor extraction = use scree plot gap to infer number of factors, eigenvalue must exceed 1
Factor rotation
Orthological rotation (varimax) (assumes factors independent (unlikely), solution more interpretable
Use oblique rotation (direct oblimin)
