Test Construction

Question 1

Reliability

Answer 1

Amt of consistency, repeatability, and dependability in scores on a given test

Question 2

Classical Test Theory

Answer 2

Obtained score in a combo of truth & error

X=T + E

Question 3

True score variability vs Error Variability

Answer 3

due to real diffs in ability or knowledge in test takers;

caused by chance or random factors

Question 4

Range of reliability coefficient for a test

Answer 4

0. 00 to 1.00

* Interp directly as the percentage of true score variability

Question 5

Minimum acceptable Reliability

Answer 5

0.80

Question 6

Sources of Error in Test Impacting Reliability

Answer 6

1. Content Sampling- items that by chance tap into test taker’s knowledge, or don’t
2. Time Sampling-impact of giving tests at 2 diff points in time
3. Test Heterogeneity

Question 7

Factors Affecting Reliability

Answer 7

# of items (more items--> higher rel)
Homogeneity of items (items tap similar content areas--> higher rel)
Range of Scores (unrestricted range maximizes reliability)
Ability to guess (easier to guess, lower reliability b/c no longer assessing true knowledge)

Question 8

4 Estimates of Reliability

Answer 8

Test-Retest
Parallel Forms
Internal consistency
Interrater Reliability

Question 9

Coefficient of stability

Answer 9

for test-retest reliability

Major source of error is time sampling (time interval btwn administrations)

Question 10

Coefficient of equivalence

Answer 10

for parallel forms reliability

major sources of error include time sampling & content sampling

Question 11

Spearman Brown Prophecy Formula

Answer 11

used for split half reliability computation to compensate for impact of reducing # of items on each half of the test

Question 12

Split half reliability is inappropriate for ___ and ___ tests

Answer 12

Speeded (all items easy, but about how many comp in allotted time)
Power (subjects need to complete all items of varying difficulty, like EPPP)

Question 13

Ways to measure internal consistency reliability

Answer 13

Split Half

Kuder-Richardson (used when items are scored dichotomously) & Cronbach’s Alpha (used when items are likert scale)

Question 14

Standard Error of Measurement

Answer 14

standard deviation of a theoretically normal distribution of test scores obtained by one ind on equivalent tests

Question 15

Range of values for Standard Error of the Measurement

Answer 15

0.0 to standard deviation of the test (when test is totally unreliable, SEM equal the standard deviation of the test)

Question 16

Due to error in measurement, subjects scores often reported using ___ ___

Answer 16

confidence bands (standard error of measurement corresponds to z scores)
68% btwn -1 and +1 z score
95% -2 and + 2
99% -3 and + 3

Question 17

Validity of a test

Answer 17

meaningfulness, usefulness, or accuracy of a test

Question 18

3 types of validity

Answer 18

Content
Criterion related
Construct

Question 19

Content validity

Answer 19

how adequately does a test sample a particular content area
-quantified by asking a panel of experts
no numerical validity coefficient

Question 20

Criterion related validity

Answer 20

how adequately a test score can be used to predict criterion outcome
ex- how well do SAT scores predict college GPA

Question 21

How is criterion related validity calculated?

Answer 21

Pearson r; coefficient range of -1.0 to + 1.0
as low as .20 freq considered acceptable
*Correlation squared tells how much variability in the criterion is accounted for by performance on the predictor

Question 22

2 subtypes of criterion related validity

Answer 22

Concurrent-predictor & criterion measured at about the same time
Predictive-delay btwn meas of predictor & criterion

Question 23

Standard Error of the Mean vs
Standard Error of Measurement vs
Standard Error of Estimate

Answer 23

Mean-the means of many many samples should be normally distributed, the amt of variation/deviation in these group means is standard error of the mean (umbrella of inferential stats)

Measurement-under umbrella of reliability; standard deviation of theoretically normal distribution of test scores obtained by ONE Person on equivalent tests

Estimate- under umbrella of validity; standard deviation of a theoretically normal distribution of criterion scores obtained by ONE person measured repeatedly

Question 24

Standard Error of Estimate

Answer 24

standard deviation of a theoretically normal distribution of criterion scores obtained by ONE person measured repeatedly
min value of 0.0, max value is SD of the criterion
if perfect predictor SEE is 0.0; if no predictive value, SEE=standard deviation of the criterion

Question 25

3 Applications of Criterion Related Validity Coefficient

Answer 25

Expectancy Tables Taylor Russell Tables Decision Making Theory

Question 26

Expectancy Tables

Answer 26

List the probability that a person's criterion score will fall in a specified range based on the range in which a person's predictor score fell

Question 27

Taylor Russell Tables

Answer 27

Show how much more accurate selection decisions are when using a particular predictor test as opposed to no predictor test Speaks to INCREMENTAL VALIDITY (amt of improvement in success rate resulting from use of a predictor test)

Question 28

3 variables affecting incremental validity (and used in Taylor Russell Tables)

Answer 28

``` Criterion related validity coefficient of the predictor test Company's base rate (rate of selecting succ employees w/o a predictor test) Selection Ratio (proportion of available openings to available applicants) ```

Question 29

Item Response Curve

Answer 29

provides one to 3 pieces of info re: test item: it's difficulty (rep by position of the curve-left vs. right) it's ability to distinguish between high & low scorers (the slope of the curve) probability of getting item right by guessing (y intercept)

Question 30

Item Response Theory

Answer 30

Based on complex math assumptions, including invariance of item parameters, which holds that charac of items should be same for all theoretically equivalent groups of subjects chosen from the same population For example, items should not have diff charac for minority & non minority groups *used to dev comp programs that tailor tests to the individual's ability level

Question 31

Other assumptions of Item Response Theory

Answer 31

assumes test items measure a latent trait | usually has little sig unl;ess working w/very large samples

Question 32

Incremental Validity is greatest when:

Answer 32

base rate is moderate, validity coefficient is high, & selection ratio is low