Validity Flashcards
Question ID #11802: Your newly developed measure of integrity correlates highly with a well-known and widely used measure of integrity. This correlation provides evidence of your measure’s ________ validity.
Select one:
A.
incremental
B.
internal
C.
discriminant
D.
convergent
The correct answer is D.
A high correlation between a new and an established measure of the same construct provides evidence of the new measure’s convergent validity.
Answer A: Incremental validity is a measure of decision-making accuracy and is associated with criterion-related validity.
Answer B: Internal validity is one of the standards used to evaluate research designs and is not relevant to the situation described in this question.
Answer C: Discriminant validity (also known as divergent validity) refers to the extent to which a test does not correlate with measures of different constructs.
Question ID #12400: In a multitrait-multimethod matrix, a test’s construct validity would by confirmed when:
Select one:
A.
monotrait-monomethod coefficients are low and heterotrait-heteromethod coefficients are high.
B.
monotrait-heteromethod coefficients are high and heterotrait-monomethod coefficients are low.
C.
monotrait-monomethod coefficients are high and monotrait-heteromethod coefficients are low.
D.
heterotrait-monomethod coefficients and heterotrait-heteromethod coefficients are low.
The correct answer is B.
When monotrait-heteromethod (same trait-different methods) coefficients are large, this provides evidence of the test’s convergent validity; that is, it shows that the test is measuring the trait it was designed to measure. Conversely, when heterotrait-monomethod (different traits-same method) coefficients are small, this provides evidence of the test’s discriminant validity; that is, it shows that the test is not measuring a different trait. Of the answer choices, this answer includes evidence of both convergent and discriminant validity, which best contributes to a test’s construct validity.
Answer A: Low monotrait-monomethod (same trait-same method) coefficients reflect low reliability coefficients. The coefficients are not directly relevant to a test’s convergent and discriminant validity, but they need to be large in order for the matrix to provide useful information. High heterotrait-heteromethod (different traits-different methods) coefficients indicate the test does not have evidence of discriminant validity.
Answer C: Monotrait-monomethod coefficients are reliability coefficients and need to be large in order for the matrix to provide useful information. Heterotrait-heteromethod coefficients provide evidence of discriminant validity when they are small.
Answer D: Heterotrait-monomethod and heterotrait-heteromethod coefficients that are low indicate that a test has discriminant validity.
Question ID #12401: Which of the following best defines the relationship between a predictor’s reliability coefficient and its criterion-related validity coefficient?
Select one:
A.
A test’s validity coefficient cannot exceed its reliability coefficient.
B.
A test’s validity coefficient cannot exceed the square root of its reliability coefficient.
C.
A test’s validity coefficient can exceed the square root of its reliability coefficient.
D.
A test’s reliability coefficient cannot exceed its validity coefficient.
The correct answer is B.
This answer describes the formula that defines the relationship between reliability and validity–i.e., a test’s validity coefficient cannot be greater than the square root of its reliability coefficient. Reliability places a ceiling on validity.
Answers A, C, and D: These responses provide incorrect information - a test’s validity coefficient cannot exceed the square root of its reliability coefficient.
Question ID #12402: The results of a factor analysis indicate that Test A has a factor loading of .70 for Factor I and a factor loading of .20 for Factor II. Assuming that only two factors were extracted and that the factors are orthogonal, you can conclude that the communality for Test A scores is:
Select one:
A.
0.9
B.
0.53
C.
0.49
D.
0.04
The correct answer is B.
Factor loadings are interpreted like correlation coefficients between two or more variables and are squared to obtain a measure of shared variability. When the factors are orthogonal (uncorrelated), the squared factor loadings can be added to obtain the communality. The factor loading for Factor I is .70 and the factor loading for Factor II is .20: .70 squared is 49% and .20 squared is 4%, so the communality is 49% plus 4%, which equals 53%. This means that the total amount of variability in Test A scores explained by the factor analysis is 53%.
Answers A, C, and D: These responses provide incorrect data - the total amount of variability in Test A scores explained by the factor analysis is 53%.
Question ID #12403: When conducting a factor analysis, you would choose an oblique rotation of the factors if:
Select one:
A.
you are assessing the construct validity of a test designed to measure a single trait.
B.
you believe that each test included in the analysis measures a different construct.
C.
you believe the constructs measured by the tests included in the analysis are correlated.
D.
you want to determine if a test has an adequate level of incremental validity.
The correct answer is C.
In factor analysis, orthogonal means uncorrelated, while oblique means correlated. Therefore, you would conduct an oblique rotation if you believe the test you are validating measures constructs that correlate with the constructs measured by other tests conducted in the analysis.
Answer A: A test is shown to have construct validity when it has high correlations with factor(s) it is expected to correlate with and low correlations with the factor(s) is not expected to correlate with. Although you may use factor analysis to determine if a test has construct validity, this answer does not directly address the use of oblique rotation of factors.
Answer B: You would conduct an orthogonal rotation if you believe that each test included in the analysis measures a different construct.
Answer D: Incremental validity refers to the extent to which a predictor increases decision-making accuracy, and it is calculated by subtracting the base rate from the positive hit rate.
Question ID #12404: The standard error of estimate is used to:
Select one:
A.
estimate the difference between an examinee’s obtained test score and his or her true test score.
B.
estimate the difference between an examinee’s predicted criterion score and his or her true criterion score.
C.
estimate the maximum a predictor’s validity coefficient can be given the reliabilities of the predictor and the criterion.
D.
estimate the probability that an examinee will obtain a particular score on a criterion.
The correct answer is B.
The standard of estimate is used to estimate the range within which an examinee’s true criterion score is likely to fall given his or her predicted score on the criterion.
Answer A: The standard error of measurement is used to estimate the difference between an examinee’s obtained test score and his or her true test score.
Answer C: The correction for attenuation formula is used to estimate the predictor’s validity coefficient if the predictor and/or criterion were perfectly reliable.
Answer D: An expectancy table makes it possible to use an examinee’s predictor (test) score to estimate the probability that he or she will attain particular scores on a criterion.
Question ID #12405: In a scatterplot constructed from data collected in a concurrent validity study, the number of “false negatives” is likely to increase if:
Select one:
A.
the predictor and criterion cutoff scores are both raised.
B.
the predictor and criterion cutoff scores are both lowered.
C.
the predictor cutoff score is raised and/or the criterion cutoff score is lowered.
D.
the predictor cutoff score is lowered and/or the criterion cutoff score is raised.
The correct answer is C.
The number of false negatives increases as the predictor cutoff score is raised (moved to the right in a scatterplot) and when the criterion score is lowered (moved toward the bottom of the scatterplot).
Answer A: True positives increase when the predictor and criterion cutoff scores are both raised.
Answer B: True negatives increase when the predictor and criterion cutoff scores are both lowered.
Answer D: False positives increase when the predictor cutoff score is lowered and/or the criterion cutoff score is raised.
Question ID #12406: Validity is best described as:
Select one:
A.
consistency.
B.
accuracy.
C.
distinctiveness.
D.
stability.
The correct answer is B.
When a test is valid, it accurately measures the attribute(s) it was designed to measure.
Answer A: Reliability is a measure of consistency.
Answer C: Distinctiveness is not a description of validity.
Answer D: Stability is not a description of validity.
Question ID #12407: A test developer uses a sample of 50 current employees to identify items for and then validate a new selection test (predictor). When she correlates scores on the test with scores on a measure of job performance (criterion) for this sample, she obtains a criterion-related validity coefficient of .63. When the test developer administers the test and the measure of job performance to a new sample of 50 employees, she will most likely obtain a validity coefficient that is:
Select one:
A.
greater than .63.
B.
less than .63.
C.
about .63.
D.
negative in value.
The correct answer is B.
The validity coefficient tends to shrink on the second sample because the test was tailor-made for the initial sample. Chance factors that contributed to the validity coefficient in the initial sample will not all be present in the second sample. This question is asking about “shrinkage,” which occurs when a test is cross-validated on another sample.
Answer A: The validity coefficient will not likely increase on the second sample.
Answer C: The validity coefficient will not likely be the same on the second sample.
Answer D: The validity coefficient will likely be smaller on the second sample, but will not change from a positive to negative correlation.
Question ID #12408: When determining a predictor’s incremental validity, the positive hit rate is calculated by:
Select one:
A.
dividing the number of true positives by the total number of positives.
B.
dividing the total number of positives by the number of people in the sample.
C.
dividing the base rate by the number of true positives.
D.
dividing the total number of positives by the base rate.
The correct answer is A.
The positive hit rate is calculated by dividing the number of true positives by the total number of positives.
Answers B, C, and D: When determining a predictor’s incremental validity, the positive hit rate is calculated by dividing the number of true positives by the total number of positives.
Question ID #12409: A test’s specificity refers to the number of __________ that were identified by the test.
Select one:
A.
true positives
B.
false positives
C.
true negatives
D.
false negatives
The correct answer is C.
Specificity refers to the identification of true negatives (the percent of cases in the validation sample who do not have the disorder and were accurately classified by the test as not having the disorder).
Answer A: True positives are the people who are predicted to succeed by the predictor and who are actually successful on the criterion. A test’s sensitivity is the percent of people in the validation sample who have the disorder and were accurately identified by the predictor as having the disorder.
Answer B: False positives are the people who are predicted to succeed by the predictor by who are not successful on the criterion.
Answer D: False negatives are the people who are predicted to be unsuccessful on the predictor but who are actually successful on the criterion.
Question ID #12415: A test’s content validity is established primarily by which of the following?
Select one:
A.
Conducting a factor analysis
B.
Assessing the test’s convergent and divergent validity
C.
Having subject matter experts systematically review the test’s items
D.
Determining if changes in test scores reflect expected developmental changes
The correct answer is C.
Content validity refers to the degree to which test items are an adequate sample of the content domain and is determined primarily by the judgment of subject matter experts. Answers A, B, & D are incorrect, as these methods are used to establish a test’s construct validity.
Question ID #12618: To ascertain if the test you have developed is valid as a screening test for determining whether a person has an anxiety or affective disorder, you would be most interested in evaluating the test’s:
Select one:
A.
content validity.
B.
external validity.
C.
concurrent validity.
D.
differential validity.
The correct answer is C.
Concurrent validity is a type of criterion-related validity. It is used to establish validity when the purpose of the test is to estimate current status on a criterion. In this case, the criterion would be some method of diagnosis that is known to be accurate. This situation is analogous to using a predictor to estimate current performance on a criterion. The predictor, in this case, is the screening test, while the criterion is the accuracy of the diagnosis.
Answer A: Content validity would be of interest when a test is designed to be a sample of a particular content domain.
Answer B: External validity refers to the generalizability of research results and does not apply to this situation.
Answer D: A test has differential validity when it has different validity coefficients for different groups. Differential validity exists when the validity coefficient of a predictor is significantly different for one subgroup than for another subgroup.
Question ID #13228: To evaluate the concurrent validity of a new selection test for computer programmers, you would:
Select one:
A.
use factor analysis to determine if the test measures the abilities it was designed to measure.
B.
have subject matter experts review test items to ensure they are relevant to success as a computer programmer.
C.
administer the test to current computer programmers and correlate their test scores with recently assigned job performance ratings.
D.
administer the test to applicants for computer programmer jobs, hire all applicants regardless of their scores on the test, and correlate their test scores with job performance ratings they receive six months later.
The correct answer is C.
Concurrent validity is a type of criterion-related validity and involves correlating scores on the predictor and criterion when both measures have been administered to examinees at about the same time.
Answer A: Factor analysis is one method used to establish a test’s construct validity.
Answer B: Content validity refers to the degree to which test items are an adequate sample of the content domain and is determined primarily by the judgment of subject matter experts.
Answer D: Correlating programmers’ test scores with job performance ratings they receive six months later is an example of predictive validity. Predictive validity is a type of criterion-related validity and refers to the measure of a criterion some time after the predictor has been administered.
Question ID #13229: ____________ refers to the percent of examinees who have the condition being assessed by a predictor who are identified by the predictor as having the condition.
Select one:
A.
Specificity
B.
Sensitivity
C.
Positive predictive value
D.
Negative predictive value
The correct answer is B.
Sensitivity refers to the probability that a predictor will correctly identify people with the disorder from the pool of people with the disorder. It is calculated by dividing the number of true positives by the number of true positives plus false negatives.
Answer A: Specificity refers to the probability that a predictor will correctly identify people who do not have the disorder. It is calculated by dividing the number of true negatives by the number of true negatives plus false positives.
Answer C: The positive predictive value (PPV) indicates the probability that people who test positive have the disorder. It is calculated by dividing the number of true positives by the number of true and false positives.
Answer D: The negative predictive value (NPV) indicates the probability that people who test negative do not have the disorder. It is calculated by dividing the number of true negatives by the number of true and false negatives.
