Test Construction

Question 1

true score variability

Answer 1

variability due to real differences in ability or knowledge in the test-takers

Question 2

error variability

Answer 2

variability caused by chance or random factors

Question 3

classical test theory

Answer 3

test scores = true score variability + error variability

Question 4

reliability

Answer 4

the amount of consistency, repeatability, and dependability in scores obtained on a test

Question 5

reliability coefficient

Answer 5

• represented as ‘r’
• ranges from 0.00-1.00
• minimum acceptability is 0.8
• two factors that affect the size are: range of test scores and the homogeneity of test content

Question 6

sources of errors in tests

Answer 6

• content sampling
• time sampling (ex - forgetting over time)
• test heterogeneity

Question 7

factors that effect reliability

Answer 7

• number of items (the more the better)
• homogeneity (the more similar the items are, the better)
• range of scores (the greater the range, the better)
• ability to guess (true/false = the least reliable)

Question 8

test-retest reliability
(or coefficient of stability)

Answer 8

• correlating pairs of scores from the same sample of people who are administered the identical test at two points in time

Question 9

parallel forms reliability
(or coefficient of equivalence)

Answer 9

• correlating the scores obtained by the same group of people on two roughly equivalent but not identical forms of the same test administered at two different points in time

Question 10

internal consistency reliability

Answer 10

• looks at the consistency of the scores within the test
• 2 ways: Kuder-Richardson or Cronbach’s coefficient alpha

Question 11

split half reliability

Answer 11

• splitting the test in half (ex - odd vs even numbered questions) and then correlating the scores based on half the number of items

Question 12

Spearman-Brown prophecy formula

Answer 12

• a type of split-half reliability
• tells us how much more reliable the test would be if it were longer

*inappropriate for speeded tests

Question 13

Kuder-Richardson and Cronbach’s coeffcient alpha

Answer 13

• involve analysis of the correlation of each item with every other item in the test (reliability/internal consistency)
• KR-20 is used when items are scored dichotomously (ex - right or wrong)
• Cronbach’s is used when items are scored non-dichotomously (ex - Likert scale)

Question 14

interrater reliability

Answer 14

• the degree of agreement between two or more scorers when a test is subjectively scored
• best way to improve = provide opportunity for group discussion, practice exercises, and feedback

Question 15

validity

Answer 15

• the meaningfulness, usefulness, or accuracy of a measure

Question 16

3 basic types of validity

Answer 16

• content
• criterion
• construct

Question 17

face validity

Answer 17

• the degree to which a test subjectively appears to measure what it says it measures

Question 18

content validity

Answer 18

• how adequately a test samples a particular content area

Question 19

true positive

Answer 19

• test takers who are accurately identified as possessing what is being measured

*correct prediction

Question 20

false positive

Answer 20

• test takers who are inaccurately identified as possessing what is being measured

*incorrect prediction

Question 21

true negative

Answer 21

• test takers who are accurately identified as not possessing what is being measured

*correct prediction

Question 22

false negatives

Answer 22

• test takers who are inaccurately identified as not possessing what is being measured

*incorrect prediction

Question 23

item difficulty

Answer 23

• represented by ‘p’
• can range in value from 0 - 1+ (0 easy, 1 very difficult)
• difficulty level = number of people who got it right
• items should have an average difficulty level of 0.50, and a range of 0.30 and 0.80

example - if ‘p’ is 0.10, that means only 10% of people got the item right (therefore it was difficult)

Question 24

diagnostic validity

Answer 24

• focuses on who DOES NOT have the disorder
• an ideal situation would result in high sensitivity, specificity, hit rate, and predictive values with few false positives and few false negatives

Question 25

convergent validity

Answer 25

the degree to which scores have high correlations with scores on another measure that assesses the same thing

Question 26

standard error of measurement

Answer 26

used to construct a confidence interval

Question 27

confidence intervals

Answer 27

68% = 1 standard error of measurement
95% = 2 standard error of measurement
99% = 3 standard error of measurement

Question 28

content validity

Answer 28

Does the content accurately measure all of the topics/items it’s intending to measure?

Example: In a stats exam, content validity would be very low if it only asked you how to calculate the mean. Content validity would increase if it asked additional questions about other things within the stats world as well.

Question 29

criterion validity

Answer 29

Does the content accurately reflect a set of abilities in a current or future setting?

2 kinds:
Concurrent criterion = does this test accurately assess my student’s current level of ability?
Predictive criterion = does this test accurately assess how my students will do in the future?

Question 30

construct validity

Answer 30

does your test measure the construct it claims to measure (ex - aggression)

construct = a group of interrelated variables that you care about

Question 31

Increase item difficulty

Answer 31

0= very difficult
1= very easy
So to increase you want to add items that have p values as close to 0 as possible

Question 32

Sensitivity vs specificity

Answer 32

Sensitivity: senses people WITH diagnosis = positive predictive value true positives/ (TP+FN)

Specificity: specifies people who DONT have diagnosis. Negative predictive value = True negatives/ (TN+FN)

Question 33

Factor matrix & commonality

Answer 33

Tool that helps us see how different things (variables) are connected or share common traits.

Commonality: outlines proportion of variability

We rotate a matrix to obtain a factor matrix that is easier to interpret

Question 34

Size of Standard error of mean increases as

Answer 34

Population SD INCREASES and sample size DECREASES

Question 35

Raising test’s cut off score. Will have which effects?

Answer 35

Raising cut off score: will result in fewer applications being hired.

DECREASE: number of false positives
INCREASE: number of true negatives

Question 36

When test scores represent an interval or ratio scale and the distribution is skewed. The best measure of central tendency is what?

Answer 36

Median

Question 37

Item characteristic curve intercepts Y (vertical axis) and provides information about which of the following?

Answer 37

Probability of answering item correctly by guessing

Question 38

Banding (statistical banding)

Answer 38

Put test scores into groups based on a range of possible errors. People with scores in the same group are considered equally good at something (assuming small core differences don’t matter for job performance)

Question 39

Attenuation formula

Answer 39

Used to estimate what the maximum criterion related validity coefficient would be if the predictor/criterion had a reliability coefficient of 1

Question 40

When the prevalence of disorder increases how are its magnitude effects of a test’s positive and negative predictive values impacted?

Answer 40

When prevalence INCREASES the POSITIVE predictive value INCREASES and NEGATIVE predictive value DECREASES

Question 41

How to calculate standard error of estimate

Answer 41

A 68% confidence interval is constructed by adding and subtracting one standard error of estimate to and from the person’s predicted criterion score, a 95% confidence interval is constructed by adding and subtracting two standard errors, and a 99% confidence interval is constructed by adding and subtracting three standard errors.