Stats, Research Design, Test Construction Flashcards

Question

Test Construction: Factors Affecting Reliability

Answer 1

-Number of items: more items = more reliable -Homogeneity of items: more homogeneous = more reliable -Range of scores: full spread of scores (unrestricted range) maximizes reliability - range of scores obtained is related to the heterogeneity of the subtests - more heterogeneous subjects = greater range of scores -Ability to guess: easier to guess = lower reliability (e.g., true/false is less reliable than multiple choice)

Answer 2

1) Test-Retest Reliability (coefficient of stability) -test and re-test with identical instrument -correlate people's scores at two points in time -are the scores stable when you measure people and measure again using the same instrument -main source of error in test-retest is time - longer time between test and retest - lower reliability 2) Parallel Forms (Alternate Forms, Equivalent Forms) Reliability -calculated by correlating the scores obtained by the same group of people on two roughly equivalent but not identical forms of the same test administered at two different points in time -can be costly and time-consuming -minimizes practice effects -major sources of error: time and content sampling 3) Internal Consistency Reliability -consistency of the scores within the test -test is administered only once to one group of people -two ways to assess internal consistency a) Split-Half Reliability -splitting the test in half and then correlating the scores obtained on each half by each person. Correlate those sets of scores -when the test is split in half, the correlation is based on half the number of items = lower reliability --> underestimates the true reliability -so, spearman brown (SP for split half and Spearman Brown) is used. Spearman Brown Prophecy Formula tells us how much more reliable the test would be if it were longer -source of error is item or content sampling a1) Speeded Tests - Split-Half reliability is inappropriate for speeded tests - like coding on the WAIS -because subjects typically get all completed items correct, there should be a nearly perfect correlation between halved of the test --> reliability will be artificially inflated a2) Power Tests - items that are of varying difficulty level -subjects provided ample time -scores expressed in terms of percentage correct b) Kuder-Richardson & Cronbach's Coefficient Alpha -involves analysis of the correlation of each item with every other item in the test -calculated by taking the mean of the correlation coefficients of for every possible split-half -average of every possible way of splitting the test in half -Kuder-Richardson = dichotomous -Cronbach's Alpha = non-dichotomous 4) Inter-Rater Reliability -when what you are rating is subjectively scored -degree of agreement between two or more scores between raters -best way to improve = group discussion, practice exercises, feedback -measured by: percent agreement, pearson r, Kappa statistic, Yule's Y

Answer 3

-one way to get some idea of the average amount of measurement in each person's score would be to construct a theoretical distribution. This distribution would consist of one person's scores if they were tested hundreds of times with alternate or equivalent forms of the test. This parallels what is done with the standard error of the mean -the standard deviation of this distribution would indicate the average amount of measurement error, or the standard error of measurement -standard error of measurement is the average amount of measurement error when we use a test and we measure anybody -assumption is that for a given test, the amount of measurement error would be consistent across all persons -formula on page 50 Standard Error of Measurement -range of measurement error = 0.00 to a maximum value of the SD of the test -if a test is totally unreliable, the standard error of measurement would be equal to the standard deviation of the test -because there is error in measurement, a subject's score on a test can never be reported as the true score. Rather, we report scores using confidence bands or confidence intervals -to calculate a confidence interval, create a bell-curve. plot the score in the middle. add standard error to the obtained score. You need the score and you need the standard error. add and subtract error to the score for either side of the bell. -possible confidence bands are 68%, 95%, 99%

Answer 4

-defined as the meaningfulness, usefulness, or accuracy of a measure -can tell us how well a test is measuring what it is supposed to be measuring or how well a test can be used to infer criterion performance

Answer 5

-addresses how adequately a test samples a particular content area -is our test measuring the content, knowledge, info that it's supposed to? -quantified by asking a panel of experts -applies to tests that require knowledge of a particular domain or skills

Answer 6

-can we use our test to predict something? -use pearson r to correlate test scores (predictor scores) with criterion scores (outcome scores) -test scores = x; outcome scores = y -rxy ranges from -1.0 to 1.0 -validity as low as .20 considered acceptable -higher coefficient = more valid predictor -negative validity coefficient indicates inverse relationship -when there is a correlation, can construct a regression equation -to determine how much of the outcome can be accounted for by the predictor, the criterion-related validity coefficient must be squared Two Subtypes 1) Concurrent (same time e.g., few weeks apart) -predictor and criterion are measured about the same point in time 2) Predictive Validity (e.g., few years apart) -delay between the measurement of the predictor and criterion

Answer 7

-error having to do with estimating (predicting) -how much error there is in estimating what we are trying to estimate -range of standard error is from a minimum value of 0.0 to a maximum value of the standard deviation of the criterion

Answer 8

-relevant for purposes of prediction -three applications: 1) Expectancy Tables -likelihood that criterion scores will fall in a range given the range of scores that a predictor falls 2) Taylor-Russell Tables -tells you how much of an improvements you will make in your selection or hiring decisions when you use a predictor test -base rate: rate of successful employees without using a predictor test at all -selection ratio: proportion of openings to applicants -openings divided by applicants -incremental validity: the amount of improvement in the success rate when you use a predictor test compared to no test at all -variables that affect incremental validity: criterion-related validity of the instrument (rxy); base rate (!); selection ratio (!) -incremental validity is optimized when the base rate is moderate (.5) and the selection ratio is low (.1) 3) Decision-Making Theory -looks at our predictions based on using our predictor tests and compares them with actual results -four possible outcome: 1) true positives; 2) false positives; 3) true negatives; 4) false negatives -classification of positive and negative is based on whether the person falls above or below the predictor cutoff (to the right hand side is positive) -false positives are more problematic because they have been hired and you've wasted your money on them -changing the cutoff affects the number of people in each category. it is less desirable to change the criterion cutoff relative to predictor.

Answer 9

1) conceptualization - decide on test's objective, administration, overall format 2) test construction - decide item format and write items- always create many more items than what you end up with 3) test tryout - trying out the items on the available sample 4) Item analysis - analyze items regarding: a) item difficulty - how many people get it right? proportion - .9 = easy item. want range to be .3-.8 with an average of .5 b) item discrimination - discriminates between high and low scorers c) item validity - correlation between item score and test score - higher correlation = higher validity d) item-characteristic curve - plot of the relationship between item performance and total score - basis of IRT - item response theory aka latent trait theory -IRT - calculate the extent to which the specific item correlates with the underlying construct we think we are measuring. Used to develop individually tailored adaptive tests - answer to one question in a domain area determines whether another question in that area will be asked next 5) Test Revision - subset of items is kept. Criterion-related validity coefficient is calculated. Then, the test is cross-validated - administered to a new sample. When we cross-validate, we end up with shrinkage - validity coefficient shrinks

Answer 10

Range of Scores -unrestricted range of scores - we want a broad range of scores Reliability of the Predictor -reliability of the predictor affects the validity -reliability (xx) puts a ceiling on validity - gives the upper limit of how valid it can be Reliability of the Predictor and Criterion -imperfect reliability of predictor and criterion creates measurement error when predictor and criterion are correlated -correlation is less than what it would have been if the predictor and criterion had been perfectly reliable -correction for attenuation - calculates how much higher validity would be if the predictor and criterion were perfectly reliable Criterion Contamination -occurs with subjectively-scored criterion outcomes when the rater is informed of the predictor scores before assigning rating -typically results in an inflated or spuriously high criterion-related validity coefficient

Answer 11

-asks if my instrument is measuring the trait that I think it's measuring Multi-Trait, Multi-Method Matrix -has information about convergent and divergent (discriminant) validity -need both for construct validity Convergent Validity -correlation of scores on the new test with other available measures of the same trait -correlation should be moderate-high e.g., .56; monotrait-heteromethod -always about measuring the same trait and getting a decent correlation Divergent Validity -idea that if you are trying to create a test to measure aggression, you want that to diverge from measurement of a different construct e.g., hyperactivity -correlation should thus be low e.g., ,07 -heterotrait-monomethod

Stats, Research Design, Test Construction Flashcards

(37 cards)