Chapter 6

Question 1

What is validity in psychological measurement?

Answer 1

The meaningfulness of a test score. What the test score actually means.

How well a test captures what it purports to capture

Question 2

Inference

Answer 2

A logical result or deduction

Question 3

Is there such thing as a universally valid test?

Answer 3

No. There are boundaries

Question 4

Local validation studies

Answer 4

Test users make their own validation studies for their purpose (eg. altering the test, different population, etc.)

Question 5

Trinitarian view of validity

Answer 5

Content, Criterion-related, and Construct Validity

Question 6

Ecological Validity

Answer 6

How generalizable things are to real world

Question 7

Face validity

Answer 7

What a test appears to measure to the person being tested

Question 8

Content Validity

Answer 8

How well a test samples behavior representative of the universe of behavior that it’s designed to sample

Question 9

How to ensure content validity in educational assessments?

Answer 9

Make sure test approximates proportion of material covered in the course

Question 10

Test blueprint

Answer 10

A plan for what information should be covered by items, how much of each area, organization on test, etc.

Question 11

How might (content) validity be relative?

Answer 11

Different cultures/religions/political parties have different historical interpretations. This means a test can be valid in one but not another region

Question 12

Criterion-Related Validity

Answer 12

How well a test score infers most probable standing on some measure of interest

Question 13

Concurrent validity

Answer 13

How much test score is related to some measure at the same time (concurrently)

Question 14

Predictive validity

Answer 14

How well a test score predicts some criterion measure

Question 15

Criterion (definition in testing evaluation)

Answer 15

The standard to which a test or score is being evaluated

Question 16

An adequate criterion must be:

Answer 16

1) also Valid

2) Uncontaminated (a criterion measure cannot be based on its predictor)

Question 17

How do you statistically find criterion contamination or correct it?

Answer 17

Can’t

Question 18

Base rate vs. hit rate vs. miss rate

Answer 18

base rate: proportion that actually has something

hit rate: proportion accurately identified by a test to have something

miss rate: proportion that a test fails to identify to have something

Question 19

2 statistics used for criterion-related validity

Answer 19

1) validity coefficient:
- relationship between test scores and scores on the criterion measure (eg. Pearson, Spearman)

2) expectancy data:

Question 20

What happens to validity coefficient when attrition of subjects happens over time?

Answer 20

Lower validity? (adverse effect)

Question 21

Incremental validity

Answer 21

How much an additional predictor explains something about criterion measure that isn’t already explained by the existing predictors

Question 22

Time spent studying (1) and time in library (2) likely has high or low incremental validity?

Answer 22

Low. Because they’re probably highly correlated

Question 23

Incremental validity of emotional intelligence

Answer 23

Has been doubted because of how they correlate strongly with intelligence and personality

Question 24

Construct Validity

Answer 24

how appropriate inferences drawn from a test are

Question 25

Construct (definition)

Answer 25

scientific idea that describes/explains behavior. unobservable/underlying

Question 26

Homogeneity of a test

Answer 26

Whether a test measures a specific concept

Question 27

If a concept meaning says the score is supposed to improve with age (eg. reading ability), measuring scores across ages would be evidence for what type of validity?

Answer 27

construct validity

Question 28

Pre-test and post-test of marriage thing after marriage therapy is evidence for which validity?

Answer 28

construct validity

Question 29

Method of contrasted groups

Answer 29

Show that scores differ between groups (one that should score higher vs. shoudn't)

Question 30

Convergent evidence

Answer 30

eg. new anxiety test correlate with old

Question 31

Divergent evidence

Answer 31

evidence that something doesn't correlate with something that it isn't supposed to correlate with

Question 32

multitrait-multimethod matrix what is it used for?

Answer 32

A matrix/table resulted from correlating traits within/between methods used for convergent/divergent validity evidence

Question 33

Method variance

Answer 33

Similarity in scores due to the same method

Question 34

Factor analysis

Answer 34

Procedure to identify factors of attributes/characters/dimensions

Question 35

Factor loading high or low for convergent/discriminant validity?

Answer 35

Determines extent to which a factor determines the test score. high = convergent

Question 36

Intercept bias vs. Slope bias

Answer 36

intercept bias: consistent under/overprediction of a score of specific group slope bias: a predictor has a weaker correlation with an outcome of specific groups

Question 37

Rating error

Answer 37

Misuse of a rating scale (eg. leniency error = grading too easy)

Question 38

Severity error

Answer 38

too harsh (opposite of leniency error)

Question 39

Central tendency error

Answer 39

Rater tends to grade average rather than positive/negative

Question 40

What might be a way to overcome rating errors?

Answer 40

Use ranking rather than scores. Forces a rank between individuals

Question 41

Fairness in psychometric context

Answer 41

How equitable, just, impartial a test is

Question 42

Does no members of a specific group being part of test make it unvalid for a test user in the group?

Answer 42

Not in itself.

Question 43

According to Borsboom article, what is validity? (when is a test valid?)

Answer 43

When the attribute exists and if variations in the attribute causes variations in outcomes

Question 44

Ontology vs. Epistemology (Borsboom)

Answer 44

Ontology: Existence of attributes Epistemology: Process of knowing or finding out about the attributes

Question 45

Importance of causality in validity (Borsboom)

Answer 45

Validity is tied to causal relationships (changes in attribute --> changes in test score). Currently, correlational views of validity are misconceptions. Tests aren't valid because of correlations.

Question 46

What does Borsboom propose in test construction research?

Answer 46

To focus on causal processes between attributes and test outcomes rather than optimizing predictive properties (multicollinearity issues)

Question 47

How does correlational approaches not provide good evidence for validity?

Answer 47

They can lead to stupid conclusions, like thunder and lightning perfectly correlating, but you can't say they're the same thing.