Ch. 6 - Validity

Question 1

validity

Answer 1

judgment of how well a test measures what it purports to measure in a particular context; judgment based on evidence about the appropriateness of inferences drawn from test scores

Question 2

inference

Answer 2

logical result or deduction

Question 3

a valid test has been shown to be valid for

Answer 3

a particular use with a particular population of testtakers at a particular time

Question 4

no test is ____ valid

Answer 4

universally valid for all times, all uses, and with all populations

Question 5

a test is valid within ____

Answer 5

“reasonable boundaries” of a contemplated usage

Question 6

validation

Answer 6

the process of gathering and evaluating evidence about validity

Question 7

validation studies can be done with ____

Answer 7

a group of testtakers, to provide insights regarding a particular group of testtakers as compared to a norming sample (local validation)

Question 8

what are the three categories of validity?

Answer 8

content, criterion-related, and construct (umbrella)

Question 9

content validity

Answer 9

scrutinizing the test’s content

Question 10

criterion-related validity

Answer 10

relating scores obtained on the test to other test scores or other measures

Question 11

construct validity

Answer 11

umbrella validity; all others fall under it. a comprehensive analysis.
analysis of how test scores relate to other measures and how scores can be understood within some theoretical framework. (maybe your hypothesis about what’s different about high and low test scorers)

Question 12

face validity

Answer 12

not one of the three C’s

what a test appears to measure or how relevant the test items look to the testtaker

Question 13

why does face validity matter?

Answer 13

testtakers may not put forth good effort; parents may complain about their kids taking a non-face-valid test; lawsuits may be filed

Question 14

content validity

Answer 14

judgment of how adequately a test samples behavior representative of the whole universe of behavior that the test was designed to sample.

e. g. assertiveness test assesses behavior on the job, in social situations, etc
e. g. test samples all chapters

Question 15

how can we judge content validity?

Answer 15

get a panel of judges or experts - if more than half indicate that an item is essential, that item has some content validity. more people agree, more content validiy

Question 16

what’s a problem with establishing content validity?

Answer 16

we frequently don’t know all of the items in the theoretical domain of possible items

Question 17

criterion-related validity

Answer 17

a judgment of how adequately a test score can be used to infer an individual’s standing on a criterion being measured (3 types: concurrent, predictive, incremental)

Question 18

concurrent validity

Answer 18

a judgment of how adequately a test score can be used to infer an individual’s present standing on a criterion (ex: diagnosing someone from a test when you already know they have the thing - perhaps from a diff validated test. the test might be an easier way to reach the diagnosis)

Question 19

predictive validity

Answer 19

measures of the relationship between the test scores and a criterion measure obtained at a future time (ex: using GRE scores to predict graduate course passing)

Question 20

criterion

Answer 20

standard against which a test score is measured; can be almost anything (behavior, diagnosis)

Question 21

a good criterion is

Answer 21

relevant (pertinent to matter at hand); valid (if X is being used to predict Y, then we need to know X is valid); uncontaminated (not based on a predictor measure (if X is used to predict Y, and Y is in part based on X, then X is contaminiated)

Question 22

what are three types of criterion-related validity?

Answer 22

concurrent validity
predictive validity
incrimental validity

Question 23

base rate

Answer 23

extent to which a particular trait, behavior, etc exists in the popluation (proportion)

Question 24

hit rate

Answer 24

proportion of people that a test accurately identifies as having a specific trait

Question 25

miss rate

Answer 25

proportion of people a test failes to identify as having a trait. an inaccurate prediction

Question 26

false positive

Answer 26

test identifies a testtaker as having the trait when they don't

Question 27

false negative

Answer 27

test does not identify a testtaker as having the trait when they do

Question 28

validity coefficient

Answer 28

correlation coefficient; provides a measure of the relationship between test scores and the scores on the criterion measure; no rules to determine a minimum accepted size; affected by restriction of range

Question 29

restriction of range

Answer 29

self-selection, testing firefighting skills on only firefighters, not general population

Question 30

incremental validity

Answer 30

kind of part of predictive; degree to which an additional predictor variable explains something about the criterion measure that's not already explained by predictors already in use (e.g.: how much sleep, how much time spent in library, how much time spent studying should both help predict GPA. if library is overlapping with studying, it doesn't have great incremental validity)

Question 31

one measure of a test's value is

Answer 31

the extent to which it improves on the hit rate for a trait that existed before the test was used

Question 32

construct validity is not mutually exclusive with ____

Answer 32

criterion

Question 33

construct validity is shown when...

Answer 33

(1) test is homoegenous (2) test scores change over time (3) post-test scores vary as predicted (from some intervention) (4) test scores from people of different groups vary as predicted (AKA method of contrasted groups) (5) test scores correlate with scores on other tests as predicted (BDI correlates with another depression index)

Question 34

example of using the method of contrasted groups

Answer 34

psych patients are more depressed than random Wal-Mart shoppers

Question 35

convergent evidence

Answer 35

if scores on the new test correlate highly in the predicted direction with scores on an older, more established, and already validated test that's testing the same thing

Question 36

divergent evidence

Answer 36

if scores on the new test don't correlate with scores on a test that you didn't theorize they would correlate with

Question 37

a valid test can be used...

Answer 37

fairly or unfairly

Question 38

test bias

Answer 38

a factor inherent in a test that systematically prevents accurate, impartial measurement. systematic = not due to change. can be identified and remedied; ex: weighted coin toss

Question 39

what's a type of test bias?

Answer 39

rating error

Question 40

rating error

Answer 40

a judgment resulting from the intentional or unintentional misuse of a raiting scale

Question 41

examples of rating error

Answer 41

leniency/generosity error (lenient on grading) severity errors - always rate bad central tendency errors - all ratings at the middle halo effect - tendency of a rater to give a ratee a higher rating than they deserve for everything (e.g., Lady Gaga speech never going to be bad no matter the topic if the rater is the president of her fan club)

Question 42

test fairness

Answer 42

the extent to which a test is used in an impartial, just, and equitable way; has to do with values and opposing points of view

Question 43

test ____ can be seen as a statistical problem, test ____ cannot

Answer 43

test bias can be seen as a statistical problem, test fairness cannot