Ch 6 Validity

Question 1

Validity

Answer 1

• Extent to which a test measures what it claims to measure in a specific context
• Are the inferences appropriate?

Question 2

Tests are not universally valid but can still measure…

Answer 2

• particular purpose
• particular population of people
• particular time

Question 3

Validity is only valid in…

Answer 3

particular circumstances

Question 4

Validation

Answer 4

Gathering and evaluating validity evidence

Question 5

Whose responsibility to check for validity?

Answer 5

• Test developers
• Test users
• Local validation studies (Necessary when test is altered in some way). (Also necessary when used on a different population)

Question 6

Face Validity (FV)

Answer 6

How valid a test looks to those involved

• Judgement concerning item relevance

Judgment comes from test taker and not test user

• Lack of FV may lead to lack of confidence in the test’s effectiveness
• FV is not correlated to psychometric soundness

Question 7

Content

Answer 7

The actual items that make up the test (specific questions)

Question 8

Content Validity

Answer 8

• How adequately a test samples behavior it is designed to sample
• Like FV except you are attempting to cover every possible aspect of what you are trying to measure
• Content validity heavily stresses comprehensiveness

Ex: Does a comprehensive final cover topics presented during a course?

Question 9

Content Validity

Answer 9

• Expert opinion
• Test develper creates many potential items
• Experts rate each items

– essential
– useful, but not neccessary
– not necessary

• Remaining items are ranked and ordered

Question 10

Content Validity Ratio

Answer 10

• Percentage of items judges agree are essential

Question 11

Relativity of content validity

Answer 11

• A measure that is content valid in one culture may not have content validity in another culture

Ex: A measure of content validity for depression in Western culture should be different from a measure of content validity for depression in Eastern cultures

Question 12

Criterion

Answer 12

Standard against which a test or score is evaluated

Ex: Mental disorder diagnosis of depression, anxiety, etc.

Question 13

Criterion - Related Validity

Answer 13

• How adequately a score can be used to infer an individual’s standing on some criterion
• Measured with validity coefficient

– Correlation between test score and score on criterion measure

Ex: Correlation between depression score and DSM-5 depression diagnosis

Question 14

Criterion - Related Validity Two Types [Placeholder]

Answer 14

.

Question 15

Concurrent

Answer 15

Test scores & criterion measure at same time

ex:
- New diagnostic tool * Existing Diagnosis

• Test A * Test B

Question 16

Predictive

Answer 16

• Test scores, then criterion measure taken in future
• Ex: Act score to predict freshman GPA (ACT was taken in the past, GPA is the criterion)
• A test with high predictive validity useful

Question 17

Incremental Validity (A type of predictive validity)

Answer 17

• Degree to which an additional predictor explains something else about the criterion

Ex: High School GPA added to the above example

Question 18

Criterion Characteristics [placeholder]

Answer 18

.

Question 19

Uncontaminated [skip]

Answer 19

.

Question 20

Criterion Contamination

Answer 20

• The criterion is based on the predictor (I.e., the criterion is on the test)
• Ex: Develop a depression measure based on BDI (Beck Depression Inventory) and then test it on patient diagnoses that were based on BDI
• Relevant
• Valid (ratings/tests used as criteria must be valid)

Question 21

Standard Error of Estimate (SEE)

Answer 21

• Margin of error expected in the predicted criterion score
• Related to the correlation between the test score and criterion

Question 22

Standard Error of Measurement (SEM)

Answer 22

For reliability

Question 23

Standard Error of Estimate (SEE)

Answer 23

For Validity

• When the measure tries to predict the criterion

Question 24

Construct

Answer 24

informed, scientific idea hypothesized to describe or explain behavior

Ex: intelligence, leadership

Question 25

Construct Validity

Answer 25

- Appropriateness of inferences drawn from test scores regarding standing on a construct - Referred to as "Umbrella Validity"

Question 26

Evidence of Construct Validity

Answer 26

- Evidence of homogeneity - How uniform a test is in measuring a single concept - Evidence of changes with age - Some constructs are expected to change over time (Reading rate) (vs. Marital satisfaction) - Evidence of pretest/posttest changes - test scores change as a result of some experience between a pretest and a posttest (therapy) - Evidence from distinct groups - Scores on a test vary in a predictable way as a function of membership in some group (scores on the psychopathy checklist for prisoners vs. civilians)

Question 27

To test for construct validity, look to the theory

Answer 27

- No one way to test, solely based on your construct and the theory behind your construct - Does it theorize that your construct is a single construct? - Does it theorize it should look differently between people who have been treated and not treated? - Does it theorize it should change over time?

Question 28

Other Forms of Evidence

Answer 28

Your test scores should correlate with other "tried and true" tests of the same thing

Question 29

Convergent Validity

Answer 29

A test that correlates highly with other tests of the same construct

Question 30

Should your tests correlate with scores of other non-related variables?

Answer 30

No

Question 31

Discriminant (Divergent) Validity

Answer 31

A test does not correlate with other tests of different constructs

Question 32

Factor

Answer 32

Characteristics, dimensions, or attributes that people differ on

Question 33

Factor Analysis

Answer 33

- A family of methods that classifies several items into groups of related factors or latent variables - Done by finding similarities among the items - Most often used in survey research to see if a long series of questions can be grouped into smaller sets of questions

Question 34

Two types of Factor Analysis [placeholder]

Answer 34

.

Question 35

Exploratory

Answer 35

You do not have a hypothetical model and let data guide the factors

Question 36

Confirmatory

Answer 36

You have a hypothetical model and test that

Question 37

Reliability - Validity Relationship

Answer 37

Without strong reliability a test cannot be valid; however, a test can be reliable without strong validity - Attenuation is the weakening of validity due to poor reliability