Test Construction Quiz Questions

Question 1

A screening test for a disorder that has a very low base rate in the population is known to have an overall accuracy rate of 98%. When using this test to identify individuals in the general population who have the disorder, it’s important to keep in mind that the test will produce:

Answer 1

a larger number of false positives than false negatives

Question 2

In a factor matrix, the factor loading for Test A and Factor II is .70. This means that:

Answer 2

49% of variability in Test A is accounted for by Factor II

Question 3

When using the multitrait-multimethod matrix to evaluate the construct validity of a newly developed test, a __________ coefficient provides evidence of the test’s divergent (discriminant) validity.

Answer 3

small heterotrait-monomethod

Question 4

When item response theory has been used as the basis for test construction, an examinee’s score on the test provides information about his/her:

Answer 4

future status on an external criterion

Question 5

To construct the 68% confidence interval for an examinee’s obtained test score, you would need the examinee’s score and:

Answer 5

the standard error of measurement

Question 6

When a test’s reliability coefficient is equal to 0, the standard error of measurement for the test is:

Answer 6

equal to the tests standard deviation.

Question 7

A test developer uses a multitrait-multimethod matrix to organize the data she has collected in a validation study of her newly developed self-report measure of self-esteem. The matrix indicates that the correlation between her self-report measure of self-esteem and an established (previously validated) teacher rating of self-esteem is .91. This correlation coefficient suggests that the self-report measure of self-esteem has:

Answer 7

adequate convergent validity.

Question 8

A measure of test anxiety is administered to a sample of 50 psychologists who are studying for the licensing exam, and a split-half reliability coefficient of .80 is calculated from their scores. The test is then administered to another group of 50 psychologists who are more heterogeneous with regard to level of test anxiety. The split-half reliability coefficient for the second group is most likely to be:

Answer 8

Larger than .80

Question 9

You would use which of the following to estimate what a predictor’s criterion-related validity coefficient would be if the predictor and/or criterion had a reliability coefficient of 1.0?

Answer 9

Correction for attenuation formula

Question 10

Scores on a predictor that will be used to estimate job performance rating range from 0 to 200. If the predictor’s cutoff score is raised from 130 to 150, this will have which of the following effects?

Answer 10

Decrease the number of false positives

Question 11

To assess the internal consistency reliability of a test that contains 50 items that are each scored as either “correct” or “incorrect,” you would use which of the following?

Answer 11

KR-20

Question 12

The kappa statistic for a test is .95. This means that the test has:

Answer 12

Adequate inter-rater reliability

Question 13

For a newly developed test of cognitive flexibility, coefficient alpha is .55. Which of the following would be useful for increasing the size of this coefficient?

Answer 13

Adding more items that are similar in terms of content and quality

Question 14

A student receives a score of 450 on a college aptitude test that has a mean of 500 and standard error of measurement of 50. The 68% confidence interval for the student’s score is:

Answer 14

400 to 500

Question 15

Consensual observer drift tends to:

Answer 15

produce an overestimate of a test’s inter-rater reliability.

Question 16

To determine a test’s internal consistency reliability by calculating coefficient alpha, you would:

Answer 16

administer the test to a single sample of examinees one time.

Question 17

A problem with using percent agreement as a measure of inter-rater reliability is that it doesn’t take into account the effects of:

Answer 17

chance agreement among raters.

Question 18

According to classical test theory, total variability in obtained test scores is composed of:

Answer 18

true score variability plus random error.

Question 19

Which of the following methods for evaluating reliability is most appropriate for speed tests?

Answer 19

Coefficient of equivalence

Question 20

Your newly developed measure of integrity correlates highly with a well-known and widely used measure of integrity. This correlation provides evidence of your measure’s ________ validity.

Answer 20

Convergent

Question 21

In a multitrait-multimethod matrix, a test’s construct validity would by confirmed when:

Answer 21

monotrait-heteromethod coefficients are high and heterotrait-monomethod coefficients are low.

High Mono-trait, Low Hetero-trait HMLH
ham l’hell

Question 22

Which of the following best defines the relationship between a predictor’s reliability coefficient and its criterion-related validity coefficient?

Answer 22

Validity is no greater than the square root of reliability

V before R because of VR- no greater than sq root

Question 23

The results of a factor analysis indicate that Test A has a factor loading of .70 for Factor I and a factor loading of .20 for Factor II. Assuming that only two factors were extracted and that the factors are orthogonal, you can conclude that the communality for Test A scores is:

Answer 23

0.53

Question 24

When conducting a factor analysis, you would choose an oblique rotation of the factors if:

Answer 24

you believe the constructs measured by the tests included in the analysis are correlated.

Question 25

The standard error of estimate is used to:

Answer 25

estimate the difference between an examinee's predicted criterion score and his or her true criterion score.

Question 26

In a scatterplot constructed from data collected in a concurrent validity study, the number of "false negatives" is likely to increase if:

Answer 26

the predictor cutoff score is raised and/or the criterion cutoff score is lowered.

Question 27

Validity is best described as:

Answer 27

Accuracy

Question 28

A test developer uses a sample of 50 current employees to identify items for and then validate a new selection test (predictor). When she correlates scores on the test with scores on a measure of job performance (criterion) for this sample, she obtains a criterion-related validity coefficient of .63. When the test developer administers the test and the measure of job performance to a new sample of 50 employees, she will most likely obtain a validity coefficient that is:

Answer 28

Less than .63 (shrinkage)

Question 29

When determining a predictor's incremental validity, the positive hit rate is calculated by:

Answer 29

dividing the number of true positives by the total number of positives.

Question 30

Test’s sensitivity measures

Answer 30

True positives

Question 31

Test’s specificity measures

Answer 31

True negatives “How much no” | specific = pacific = bad (bc California)

Question 32

A test's content validity is established primarily by which of the following?

Answer 32

Having subject matter experts systematically review the test's items

Question 33

To ascertain if the test you have developed is valid as a screening test for determining whether a person has an anxiety or affective disorder, you would be most interested in evaluating the test's:

Answer 33

Concurrent validity (type of criterion-related validity)

Question 34

To evaluate the concurrent validity of a new selection test for computer programmers, you would:

Answer 34

administer the test to current computer programmers and correlate their test scores with recently assigned job performance ratings.

Question 35

____________ refers to the percent of examinees who have the condition being assessed by a predictor who are identified by the predictor as having the condition.

Answer 35

Sensitivity “How much yes”

Question 36

Assuming a normal distribution, which of the following represents the highest score? - z-score of 1.5 - T score of 70 - WAIS FSIQ of 120 - Percentile rank of 88

Answer 36

T score of 70

Question 37

Eigenvalues are associated with:

Answer 37

principal component analysis.

Question 38

: Which of the following scores is NOT a norm-referenced score? - Percentile Rank - T-Score - Pass or Fail - Grade Equivalent Score

Answer 38

Pass or fail

Question 39

A psychologist develops a diagnostic test to identify people who have injection phobia. In this situation, the test's ________ refers to how good the test is at identifying people who have injection phobia from the pool of people who actually have injection phobia.

Answer 39

sensitivity

Question 40

To maximize the ability of a test to discriminate among test takers, a test developer will want to include test items that vary in terms of difficulty. If the test developer wants to add more difficult items to her test, she will include items that have an item difficulty index of:

Answer 40

.10 (scale goes from 0 to 1, 0 is hardest, .10 means 10% of people will answer correctly)

Question 41

The correction for attenuation formula is used to measure the impact of increasing:

Answer 41

a test's reliability on its validity

Question 42

When a test has been constructed on the basis of item response theory, an examinee's total test score provides information about his/he

Answer 42

status on a latent trait or ability

Question 43

The primary advantage in using a percentile rank, z-score, or T-score is that these scores:

Answer 43

are easy to interpret because they reference an individual's test performance to the performance of other examinees

Question 44

You would use a multitrait-multimethod matrix in order to:

Answer 44

determine if a test has adequate convergent and discriminant validity

Question 45

An advantage of using the kappa statistic rather than percent agreement when assessing a test's inter-rater reliability is that the former:

Answer 45

corrects for chance agreement

Question 46

Which type of reliability would be most appropriate for estimating the reliability of a multiple-choice speed test?

Answer 46

Alternate forms

Question 47

When using principal component analysis:

Answer 47

the first principal component represents the largest share of the total variance.

Question 48

Stella obtains a score of 50 on a test that has a standard deviation of 10 and a standard error of measurement of 5. The 95% confidence interval for Stella's score is approximately:

Answer 48

5*1.96= ~10, correct answer is 40-60

Question 49

In a normal distribution, which of the following represents the lowest score? - percentile rank of 20 -z-score of -1.0 -T-score of 25 -FSIQ of 70

Answer 49

T-score of 25 (2.5 SD below mean)

Question 50

In terms of magnitude, the standard error of measurement can be:

Answer 50

no greater than the standard deviation of the test scores

Question 51

To obtain a "coefficient of stability," you would:

Answer 51

administer the same test twice to the same group of examinees on two separate occasions and correlate the two sets of scores.

Question 52

The best way to control consensual observer drift is to:

Answer 52

Altnerate raters

Question 53

A reliability coefficient is best defined as a measure of:

Answer 53

Consistency

Question 54

A test designed to measure knowledge of clinical psychology is likely to have the highest reliability coefficient when:

Answer 54

High number of items, heterogeneous sample

Question 55

In factor analysis, a factor loading indicates the correlation between:

Answer 55

a test and an identified factor

Question 56

In factor analysis, when two factors are "orthogonal," this means that:

Answer 56

the factors are uncorrelated.

Question 57

Which of the following types of validity would you be most interested in when designing a selection test that will be used to predict the future job performance ratings of job applicants?

Answer 57

Criterion-related

Question 58

Which of the following best describes the relationship between validity and reliability?

Answer 58

A valid test is also a reliable test.

Question 59

In a multitrait-multimethod matrix, the coefficient that indicates a test's reliability is the _____________ coefficient.

Answer 59

monotrait-monomethod

Question 60

The heterotrait-monomethod coefficient is

Answer 60

a measure of divergent validity.

Question 61

The monotrait-heteromethod coefficient is

Answer 61

a measure of convergent validity.

Question 62

The heterotrait-heteromethod coefficient is

Answer 62

a measure of divergent validity.

Question 63

Cronbach's alpha is an appropriate method for evaluating reliability when:

Answer 63

all test items are designed to measure the same underlying characteristic.

Question 64

Incremental validity is a measure of:

Answer 64

decision-making accuracy

Question 65

In factor analysis, communality refers to:

Answer 65

the proportion of variance accounted for in a single variable by all of the identified factors.

Question 66

Criterion contamination has which effect

Answer 66

It artificially increases the predictor's criterion-related validity coefficient.

Question 67

To evaluate the concurrent validity of a new selection test for clerical workers, you would:

Answer 67

administer the test to a sample of current clerical workers and correlate their scores on the test with their recently assigned performance ratings

Question 68

A test developer would use the Kuder-Richardson Formula (KR-20) in order to:

Answer 68

evaluate a test's internal consistency reliability

Question 69

To maximize the inter-rater reliability of a behavioral observation scale, you should make sure that coding categories:

Answer 69

are mutually exclusive

Question 70

The minimum and maximum values of the standard error of estimate are:

Answer 70

0 and the standard deviation of the criterion

Question 71

When the kappa statistic for a measure is .90, this indicates that the measure:

Answer 71

has adequate inter-rater reliability

Question 72

A personnel director hires all job applicants who obtain a high score on a job selection test but, after using the test for six months, realizes that many of the new employees are obtaining low-performance ratings. Assuming that the selection test has adequate criterion-related validity, the personnel director can reduce the number of unsatisfactory workers that she hires using the test by:

Answer 72

raising the selection test cutoff score

Question 73

All other things being equal, which of the following tests is likely to have the largest reliability coefficient?

Answer 73

A multiple-choice test that consists of items that each have five answer options

Question 74

According to classical test theory, total variability in test scores is due to:

Answer 74

true score variability plus random error

Question 75

The assumption underlying convergent validity is that:

Answer 75

a measure of a characteristic should correlate highly with a different type of measure that is already known to assess the same characteristic.

Question 76

Content sampling is not a potential source of measurement error for which of the following methods for evaluating a test's reliability?

Answer 76

Test-retest only

Question 77

After reviewing the data collected on a new selection test during the course of a criterion-related validity study on new hires, a psychologist decides to lower the selection test cutoff score. Apparently the psychologist is hoping to do which of the following?

Answer 77

Increase the number of true positives (it will also increase the number of false positives, but that is not what he wants)

Question 78

The standard error of measurement is used to:

Answer 78

calculate the range within which an examinee's true test score is likely to fall given her obtained score

Question 79

The distribution of percentile ranks is always:

Answer 79

rectangular (flat) regardless of the shape of the distribution of raw scores

Question 80

In factor analysis, the original factor matrix is usually rotated in order to:

Answer 80

facilitate interpretation of the identified factors

Question 81

When a test user uses a correction for guessing formula that involves subtracting points from each examinee's scores, the resulting distribution of scores will have a ____________________ than the original (non-corrected) distribution.

Answer 81

lower mean and larger standard deviation

Question 82

The item discrimination index (D) ranges in value from:

Answer 82

-1.0 to +1.0

Question 83

Which of the following is used to estimate the effects of shortening or lengthening a test on the test's reliability coefficient?

Answer 83

Spearman-Brown formula

Question 84

A test developer would construct an expectancy table to:

Answer 84

facilitate criterion-referenced interpretation of test scores

Question 85

To evaluate the validity of a newly developed selection test for clerical workers, a test developer will correlate scores obtained on the test by newly hired clerical workers with the job performance ratings they receive after being on-the-job for six months. The resulting correlation coefficient will provide information on the test's:

Answer 85

predictive validity

Question 86

The point at which an item characteristic curve intercepts the vertical (Y) axis provides information on which of the following?

Answer 86

The probability of answering the item correctly by guessing

Question 87

‘hen the heterotrait-monomethod coefficient is large, this indicates:

Answer 87

Lack of discriminant validity

Question 88

It would be most important to assess the test-retest reliability of a measure that:

Answer 88

Measures a stable trait

Question 89

In terms of item response theory, the slope (steepness) of the item characteristic curve (ICC) indicates the item's:

Answer 89

ability to discriminate between examinees