Prelim 2 prep

Question 1

What are the differences between True Score Theory, Generalizability Theory, and Item Response Theory?

Question 2

What is the standard error of measurement?

Question 3

What are confidence intervals and what do they tell us?

Question 4

When confidence intervals increase in terms of percentage (i.e., 90% vs 95%), what does that do
to the range of scores it comprises?

Question 5

What does it mean if a test is valid?

Question 6

What are the three main categories of validity?

Question 7

What is: content validity, criterion-related validity, construct validity, ecological validity, external
validity, face validity?

Question 8

What is the content validity ratio and how is it used to determine content validity of test items?

Answer 8

If more than half the panelists of experts say an item is essential, has content validity, CVR 0 is half, negative if fewer than half

Question 9

Name the three characteristics of a criterion

Answer 9

Relevant, valid, uncontaminated

Question 10

What does it mean for a criterion to be uncontaminated?

Answer 10

Independent- independent group of raters decides who is good and who isn’t, then correlate that with test scores

Question 11

Define concurrent validity and predictive validity

Answer 11

Concurrent- degree to which a test score is related to some criterion measure obtained at the same time
Predictive- degree to which a test score predicts a criterion measure

Question 12

What are false negatives, false positives, specificity, and sensitivity?

Answer 12

False negative- test predicts someone doesn’t possess a trait and they do
False positive- test says someone has a trait and they don’t
Specificity- perfect wouldn’t mistakenly identify as someone having a trait when they don’t
Sensitivity- perfect identify all people who have the trait

Question 13

What is incremental validity and what would be proof of its existence?

Answer 13

Extent to which adding a second or third predictor gives more information about a criterion
proof??

Question 14

What is construct validity?

Answer 14

Extent to which a test measures a construct we are examining

Question 15

Name a describe the several ways in which you can find evidence for construct validity.

Answer 15

• Homogenous
• Evidence changes w age
• Test scores change w experience
• Distinct groups score differently
• Convergent evidence between two tests measuring the same construct

Question 16

What is the difference between convergent and concurrent validity?

Question 17

What is a factor analysis and how does an exploratory factor analysis differ from a confirmatory
one?

Answer 17

?
Exploratory- estimating or extracting factors, deciding how many to retain, rotating to an interpretable orientation ????
Confirmatory- degree to which a hypothetical model fits the data

Question 18

Name and define the different types of rating error that can occur

Answer 18

Leniency error- arises from tendency on part of rater to be lenient
Severity- opposite
Central tendency- rater doesn’t use extreme ends of scale
Halo effect- seeing people well no matter what

Question 19

What is test utility?

Answer 19

Usefulness or practical value of testing to improve efficiency, use in a particular situation helps us make better decisions

Question 20

What are some of the costs of administering a test, and what are some costs of NOT
administering one?

Answer 20

Administering:
- buying
- supply of blank test protocols
- computer program to score the test
- paying to score the test
- hiring people to administer the test
- costs of doing business

Not administering:
- loss of confidence as an ultimate cost of the company???
- missing a child abuser
- failing to diagnose when someone underreports on an interview???

Question 21

Keep in mind the real-life example I discussed about how to think about the cost of testing when
doing evaluations.

Answer 21

????

Question 22

What are the Taylor Russel tables used for and what three variables are considered when using
them to decide if giving a test is “worth it.”

Answer 22

• COME BACK

Question 23

Be able to name a few other tables (i.e., Naylor-Shine) and have a basic sense of how they work
(they could be multiple-choice option for instance)

Answer 23

COME BACK

Question 24

Name some different ways cut scores are determined.

Answer 24

COME BACK

Question 25

What’s the difference between a fixed and relative cut score?

Answer 25

Relative- actual score you need to meet a criteria changes
Fixed- always the same

Question 26

What is pilot work and why is it used?

Answer 26

Preliminary research surrounding creation of prototype of test, experiment with test items

Question 27

Name some ways scales are graded?

Answer 27

Age based
Grade based
Unidimensional v multidimensional??
Categorical v dimensional

Question 28

What are some scaling methods – and remember, they can overlap – so a categorical scale can
be graded “summatively,” etc.

Answer 28

Rating
Summative
Paired comparisons
Sorting tasks
Categorical scaling
Guttman scale
????

Question 29

What is the empirical vs analytical way of writing test items?

Answer 29

Analytical- write test questions you think will measure the qualities you want to measure
Empirical- find people with a problem, ask different types of questions, see how they respond

Question 30

Why would we want to find seemingly arbitrary items for use of distinguishing one group for
another (in other words, non-face-valid ones)

Answer 30

?

Question 31

Name some different ways items can be formatted.

Answer 31

Selected response
Constructed response
Computerized adaptive testing?

Question 32

What is computerized adaptive testing, and how does item branching work?

Answer 32

?
Adds or deletes a branch depending on performance (meaning what)?

Question 33

Name and describe a few different ways in which items can be scored

Answer 33

Cumulative
Class/category scoring
Ipsative scoring- comparing one score on one scale to another scale within the same test ?

Question 34

What are the following: item-difficulty index, item endorsement index, item reliability index,
item discrimination index

Answer 34

?

Question 35

How are item characteristic curves useful?

Answer 35

?

Question 36

Extra paper

Answer 36

almost done