Validity

Question 1

What is the distinction between validity and reliability?

Answer 1

Validity = measurement of the test is valid when it measures what it is supposed to.
Reliability = consistency of measurement

Question 2

Why is it imprecise to say that a test is valid?

Answer 2

• Because validity refers to the measurement that is attained by use of the test

Question 3

Construct underrepresentation:

Answer 3

Construct underrepresentation: aspects of the construct not assessed by the measure (maybe due to poor construction of the measure or changing knowledge of the construct)

Question 4

Construct irrelevant variance:

Answer 4

variance in test scores related to things unrelated to the construct (e.g., random and systematic error). Reduces valid measurement.

Question 5

Valid measurement:

Answer 5

overlap where measurement is measuring what it is supposed to

Question 6

How does increased or decreased validity play into the relationship between these?

Answer 6

How does increased or decreased validity play into the relationship between these?
Decreased validity = more construct underrepresentation and/or more construct irrelevant variance.
Increased validity = less of those.

Question 7

What is validity of measurement?

Already discussed above… maybe look into in more detail in the additional readings section.

Answer 7

overlap where measurement is measuring what it is supposed to… maybe look into in more detail in the additional readings section.

Question 8

What are content and construct validity? How do these differ?

Answer 8

• Content validity: examination of aspects of the test itself. Confirmatory, non-statistical/non-quantifiable process of determining how well a measure overlaps with the construct.
• Construct validity: examination of relationship between test scores & scores of other measures

• Content validity only examines aspects of the test itself, whereas construct validity compares the test scores with scores on another measure of the same construct

Question 9

Construct validity

Answer 9

Construct validity: examination of relationship between test scores & scores of other measures

Question 10

Content validity

Answer 10

Content validity: examination of aspects of the test itself. Confirmatory, non-statistical/non-quantifiable process of determining how well a measure overlaps with the construct.
Content validity only examines aspects of the test itself, whereas construct validity compares the test scores with scores on another measure of the same construct

Question 11

Determine the areas of the content domain that are measured by each test item

Answer 11

Boundary = identification of all sub-constructs that exist within that construct.
• Structure = relative weight or importance of each sub-construct within the larger construct (determined by examining research)

Question 12

Determine the areas of the content domain that are measured by each test item

Answer 12

Making sure all sub-constructs are represented by items:
o To account for weight: adjust # of items or adjust values in scoring
o This avoids construct irrelevant variance

Question 13

Compare the structure of the test with the structure of the content domain

Answer 13

Need to ensure your measurement reflects construct

o Increases content validity, decreases construct irrelevant variance & underrepresentation

Question 14

Why is an assessment of content validity seen less in psych measurement than educational testing?

Answer 14

It is harder to assess content validity with psychological constructs. Our knowledge and conceptualization of the constructs change, making it difficult to identify boundary and structure.

Question 15

What is face validity?

Answer 15

The degree to which a measure appears to measure the target construct.

Question 16

Why is it good to have a measure that appears to be valid, but not at the expense of empirical validity?

Answer 16

High face validity can increase participants’ likelihood of continuing to respond. People hate wasting their time.
However, you may need low face validity to get honest answers for some constructs (e.g. racist attitudes).

Question 17

What are the methods of determining construct validation?

Answer 17

1. Correlational study: Correlations between measures of certain behaviors (either related or unrelated to our construct) and our test.
   • Likelihood of 2 measures failing to tap into the same aspect of a construct is small.
   • Easiest and cheapest option
2. Factor analysis: analyzing which groups of items “hang” together.
   • Compares factor structure that results from a measure with the assumed factor structure of the construct.
   • Gives a determination of which items conceptually intercorrelate/covary
   • Hard to do and requires huge sample size
3. Experimental manipulation: manipulate the construct of interest (e.g., induce fear) and see if it relates to scores on your test.
   • Best way EXCEPT problems:
   o ethics (e.g., unethical to induce PTSD)
   o only works with stable constructs
   o more expensive and time-consuming

Question 18

What is the difference between convergent and discriminant validity? What would you expect from each in terms of the resulting correlation analysis?

Answer 18

Convergent validity: we want significant correlations between test & measures of behaviors related to construct
Discriminant validity: we want non-significant correlations between our test & behaviors unrelated to construct

We expect a significant positive correlation when things are related (convergent validity).
We expect a nonsignificant correlation when things are not related (discriminant validity).

Question 19

What types of information can be gained from use of the multitrait-multimethod matrix approach?
Note: You will need to determine and interpret this information from a matrix on the exam (see slides).

Answer 19

MTMM allows you to determine 3 things:

1. Convergent validity
2. Discriminant validity
3. Check for method bias

Question 20

What is method bias?

Answer 20

If there is a significant relationship (one or a couple) within the same method, it may indicate a training issue or a conceptual overlap between two traits.

Question 21

What is a criterion?

Answer 21

A measure that could be used to determine the accuracy of a decision. Outcome measure.

Question 22

What is criterion-related validity? (aka decision-making validity)

Answer 22

• Quantitative way to evaluate how successful decisions made from the test’s results are.
• We compare scores on our measure with scores on a criterion assessment (an outcome measure that helps you determine the accuracy of decisions).
• There is high criterion-related validity when there are significant correlations between our test and the criterion assessment.
• Assessed using either prediction validation strategies or concurrent validation strategies.

Question 23

Why is a predictive validation strategy considered the ideal approach? Why isn’t it always used?

Answer 23

• Maximizes potential for covariance (avoids truncated range)
• BUT not always ethical or cost effective

Question 24

How does one conduct a predictive validation procedure?

Answer 24

You screen everyone and then accept everyone (so you can see the full possible range in later assessment).

1. Obtain test scores
2. Obtain performance measures
3. Correlate these with the test scores

Question 25

How does this differ in a concurrent validation procedure?

Answer 25

Timing: Test scores and criterion scores from a preselected population are obtained at the same time (after decision has been made) and then a correlation is conducted between the two.

Question 26

Know the strengths and weaknesses of the predictive and concurrent validation approaches:

Answer 26

Predictive Validation: • Strengths: maximizes potential for covariance (avoids truncated range), screens out failures. • Weaknesses: not always ethical or cost effective Concurrent Validation: • Strengths: much more practical, easier, validity coefficients are often similar to predictive anyway • Weaknesses: only get info from people already accepted, range is more restricted (reduces correlation) Range restriction not that important unless it’s highly restricted already

Question 27

What is range restriction? How can this impact our validity for decisions?

Answer 27

• Limited variance (if you limit who makes it through, you only know how good your screen is for top performing individuals). Reduces correlation between predictor (test score) and criterion (outcome measure). Less predictable.

Question 28

How do you determine the variance in the criterion that can be explained by the predictor measure?

Answer 28

Square the validity coefficient. Ex. Using a test of IQ to predict job performance results in a validity coefficient of 0.5 so we can say that 25% of the variance in job performance (criterion) is accounted for by IQ (predictor measure)

Question 29

Know the differences between and how to calculate base rate and selection ratio.

Answer 29

* Base rate: level of performance on criterion in the general population (e.g., if 75% of the population would be successful, the base rate = .75). Easy task = higher base rate. Hard task = lower base rate. * Selection ratio: ratio of positions to applicants (e.g., if 30 people apply for 3 jobs, the selection ratio is .10 or 10%)

Question 30

Know the possible decision outcomes. Which of these outcomes are we looking to maximize?

Answer 30

```  LOOK IN SLIDES AT CHART   Accept Reject Actual Performance Success True Positive BR x SR False Negative BR – (BR x SR) Failure False Positive SR – (BR x SR) True Negative 1 – all of those (they add to 100%) ```

Question 31

What is the impact of a large base rate?

Answer 31

High true positives and high false negatives

Question 32

What is the impact of a small base rate?

Answer 32

High false positives and high true negatives

Question 33

What level of base rate is ideal in terms of accurate decision making?

Answer 33

.50 which is moderate

Question 34

What level of selection ratio has the biggest impact on correct decision making?

Answer 34

Low level of selection ratio. • High selection ratio means nothing matters because you’re accepting everyone. • Low selection ratio is much more stringent.

Question 35

Do know the formulas that I presented in class regarding how to calculate the likelihood of obtaining each of the possible decision outcomes… both with and without the use of a test as a predictor.

Answer 35

``` P(TP)=BRxSR+r Sqroot of {BR(1-BR)}{SR(1-SR)} Where P(TP) = probability of true positive BR = base rate SR = selection ratio Then related cells can be determined the same was as before (see table above). ```