Validity

Question 1

Why is it not strictly accurate to talk about the validity of a test?

Answer 1

A particular test might be valid in one context but not in another; it’s about the extent to which it’s valid in this context

Question 2

What are constructs?

Answer 2

Unobservable, underlying hypothetical traits or characteristics that we can try and measure indirectly using tests (e.g. intelligence, anxiety, speeding propensity)

Question 3

What is construct under-representation?

Answer 3

The variation that the measure doesn’t capture in the underlying construct it’s supposed to be measuring (e.g. behaviour a participant won’t admit to in self-report)

Question 4

What is construct irrelevant variance?

Answer 4

Score variation not related to the construct (e.g. misinterpretation of question wording, dishonesty or biased responses)

Question 5

How should the test and construct overlap in order to get a measure with high validity?

Answer 5

With a large overlap so valid measurement is large, with minimal construct under-representation and construct irrelevant variance

Question 6

What’s the difference between content and face validity?

Answer 6

Face validity is how valid a test appears to be (usually from the perspective of the test-taker); Content validity is often based on expert opinion (still not enough without empirical validity)

Question 7

Why is face validity not directly important from a psychometric perspective, and when can it be useful?

Answer 7

Because a test can have good face validity without any actual validity, or poor face validity while still having good actual validity; can be useful from a public relations point of view, so laypeople accept or choose the test because they think it’s appropriate and take it seriously, etc

Question 8

How could you go about evaluating content validity?

Answer 8

Getting experts to rate each question in a uni examination for its relevance to course content; getting those experts to make judgments on whether particular lectures were over or under-represented

Question 9

How could I create an exam that had great empirical validity but poor content validity?

Answer 9

By predicting students’ understanding of the course (e.g. GPA, hours spent studying, number of lectures attended, etc) with a high degree of accuracy, and without including any content from the course (although this wouldn’t be appropriate)

Question 10

What is the general process involved in testing empirical validity?

Answer 10

Create hypotheses regarding how your measure ought to perform if it is valid, then design and run studies to test these hypotheses

Question 11

What are the questions we might ask when designing an empirical validation study?

Answer 11

Does it map onto criteria as expected?; are there group differences as expected?; does it correlate with other things?; does it not correlate with dissimilar things?; are developmental changes, experimental effects, and internal structure as expected?

Question 12

Give two examples of things that might restrict the range of scores in a test, and indicate what influence this could have on the validity coefficient.

Answer 12

Non-random attrition (certain types of people dropping out of a longitudinal study); and Self-selection (only certain people are in the sample or will volunteer in the first place); both will lead to a smaller validity correlation coefficient

Question 13

How big does the validity coefficient have to be?

Answer 13

Magnitude depends entirely on the context; e.g. if validating a driving behaviour measure with accident involvement, we’d want a low validity coefficient (.10); if comparing a new and old intelligence test, we’d expect it to be much larger (at least .80)

Question 14

What is criterion validity?

Answer 14

A judgment regarding how adequately a score on a test can predict an individual’s most probable standing on some measure of interest (the criterion); the standard against which the test is evaluated

Question 15

Give 3 examples of criterion validity

Answer 15

1. University admissions test > GPA at end of the 1st year
2. Depression inventory > Clinicians’ ratings of severity of depression
3. Clerical aptitude test > supervisor’s ratings of job performance

Question 16

What is the Method of Contrasted Groups?

Answer 16

An approach to criterion validity, to determine whether test scores of groups of people vary as expected (e.g. clinical groups vs. non-clinical controls; experts vs. novices for a skill or ability test)

Question 17

What is criterion contamination?

Answer 17

It’s where the criterion being used to assess the test’s validity is pre-determined by the test, thereby undermining the logic of criterion validity

Question 18

Give two examples of criterion contamination, explaining what is contaminated in each case

Answer 18

Discovering that your validating test to distinguish between people with or without schizophrenia, was used for the original diagnosis (test is circular); or validating the Zuckerman sensation seeking scale by comparing scores with a risk-taking behaviour scale (the criterion) – they’re both asking about the same specific behaviours so the relationship is artificially inflated

Question 19

Does it matter if a criterion used to validate a test is not that reliable? Why/why not?

Answer 19

Yes, because the reliability of both the test and its criterion limits the size of how big the validity coefficient can be; both need to be reliable, valid and relevant

Question 20

What’s the relationship between the reliability of the test/criterion and the validity coefficient?

Answer 20

The validity coefficient is always less than or equal to the square root of the test’s reliability coefficient, multiplied by the square root of the criterion’s reliability coefficient

Question 21

What is concurrent validity?

Answer 21

A subset of criterion validity which determines whether test scores correlate with other measures of the same or similar thing (e.g. measure of depression correlating highly with previous validated measures of depression)

Question 22

What is predictive validity?

Answer 22

A subset of criterion validity where the test is trying to predict what the criterion will be at some future time

Question 23

Describe incremental validity

Answer 23

Using a number of different predictors, and seeing how much each individual predictor adds to predicting the criterion in addition to the effect of other predictors (can be predictive or concurrent)

Question 24

If predicting whether someone will bungee jump, how could we go about obtaining incremental validity?

Answer 24

Measure sensation seeking, risk-taking behaviour in the past, susceptibility to peer pressure, and fear of heights, etc; while any of these alone might not affect bungee jumping (poor criterion validity), used in combination, they might (using multiple regression)

Question 25

What if two predictors are highly correlated when using an incremental validity measure?

Answer 25

Then one may not add much predictive power beyond the other, even if they can both predict the criterion on its own (poor incremental validity), so you might not want to include both

Question 26

Describe discriminant/divergent validity

Answer 26

It’s when test scores don’t correlate highly with measures you’d expect them not to correlate with (e.g. depression measure doesn’t correlate highly with validated anxiety measures; shows you’re specifically measuring depression, not just a general measure of maladjustment)

Question 27

Describe the main features of the WISC IV intelligence test

Answer 27

10 core subtests are arranged into 4 groups: verbal comprehension index; perceptual reasoning index; working memory index; processing speed index; These can be combined to form a measure of general intellectual functioning (full scale IQ)

Question 28

What age is the WISC IV intelligence test designed for?

Answer 28

Children aged 6-16 years

Question 29

What are the factors about the WISC-IV that show content and empirical validity?

Answer 29

Content: scales judged to cover theoretical aspects of IQ; Empirical: correlated with other tests; predicts academic achievement; predicts differences between normative and special groups

Question 30

Theoretically, what are we asking when examining whether the internal structure is as expected?

Answer 30

Is the construct we’re measuring supposed to be multi-faceted (multiple subscales/heterogeneous) or not? Either way, if the test is a valid reflection of the construct, we’d expect the items to behave in the same way

Question 31

How can we investigate the predictions about whether the internal structure of our test maps onto the internal structure of our construct?

Answer 31

By using Factor Analysis

Question 32

Explain what factor analysis does

Answer 32

It uses mathematical techniques to group items into clusters (factors or components) on the basis of how much they correlate with one another; it picks up the distinct clusters of items in our data based on their inter-correlations

Question 33

What is construct validity?

Answer 33

It’s how well the scores on the test reflect the construct (trait or characteristic) that the test is supposed to be measuring; it’s essentially an umbrella term for all other types of validity evidence (except face validity)