Knowledge Questions Part 2 - Modern Test Theory

Question 1

Give reasons why modern psychometrics came to be developed in addition to classic
psychometrics (Tips: level of measurement, parallel items, tailored testing)

Answer 1

In modern psychometrics, the assumption of parallel items is abolished. Now, they are allowed to vary in degree of difficulty and discrimination characteristic, which is more realistic. Additionally, classical test theory makes the questionable assumption, that the level of measurement is at interval. For modern psychometric models, this assumption is more validated.

Question 2

What does the assumption of unidimensionality in modern test models mean? Which
assumption is the equivalent in classical test theory?

Answer 2

It means that the test measures one construct across all items. In classical test theory this is represented by the assumption of homogeneity.

Question 3

What does the assumption of monotonicity in modern test models mean? How can you roughly
verify this assumption using statistical methods from classical psychometrics?

Answer 3

It means that the likelihood of giving a certain answer on an item increases steadily for an increase in the underlying Theta level. This is reflected in classical test theory in the fact, that the True Score is the expected value of the outcome on an item. So, as the true score increases, the probability of giving a certain answer should also increase.

Question 4

What does the assumption of local independence in modern test models mean? Does this assumption
imply that all item-item correlations are equal to zero? How can you verify this assumption?

Answer 4

It means that for any two items and a given level of Theta, the probability of a correct response on both items obeys the product rule for independent events. This means that all item-item correlations are equal to zero only if the theta level stays constant.
Mathematically, you can show this using the equation for item-item-correlation:
Item-Item-R = True Variance/(True Variance + Error Variance)

Question 5

Which parameters does the 1-parameter (Rasch) model from modern test theory include?
And the 2-parameter model?

Answer 5

The 1-parameter model includes the item difficulty b, and the 2-parameter model additionally includes the discrimination factor a.

Question 6

For each parameter in Birnbaum models, indicate which statistic in classical item analysis more or
less corresponds with this.

Answer 6

b -> Item Mean / Item p-value

a -> Item-Rest Correl. / Corrected Item-Total Correl. / Item Reliability

Question 7

The true score model from classical test theory does not include item parameters. What type
of result should an item analysis produce according to that model?

Answer 7

In the true score model, item analysis consists of item-means, item-variances, item-rest correlations…

Question 8

In education, there is often a need to compare (groups of) individuals based on their test
results, although not all of these individuals have answered the same items. Examples are the
Dutch CITO tests in primary education (the tests change year to year) and the progress tests
administered by UM (each student selects the items he or she will answer). How is it possible to
draw fair comparisons between individuals and what assumptions are required to guarantee this?

Answer 8

Test Equating: Because in modern psychometrics, we know the item parameters a and b, we can then control for varying average levels of these as a result of different individuals answering different items.

Question 9

What does the concept of item information mean? Will this information only depend on the
item or does the individual tested also have an impact?

Answer 9

Item information is a measure of how informative an item is regarding the underlying theta level of an individual. This depends both on the item parameters and the individuals’ theta level.

Question 10

How can item information be derived from a graph containing item characteristic curves?
What probability of a correct answer produces maximum item information and for what
probability is this information minimal?

Answer 10

In an ICC, item information is associated with the slope of the curve. When b = Theta, so the probability of answering correct is 50%, the item information is maximal for a given a-level. Item information is minimal if a = 0 and/or b is very different from Theta.

Question 11

What does the concept of test information mean and how does it depend on item
information?

Answer 11

Test information tells us, to what extent the test is appropriate in differentiating between individuals in terms of their Theta level. It is the sum of all item information.

Question 12

Let’s assume a test is used to decide on a pass or a fail.
What test construction should be used to obtain maximum information? Should the test include
items that vary from very easy to very difficult or should the test simply consist of moderately
difficult items?

Answer 12

Since we’re dealing with a binary decision here, where the tails of the distribution aren’t of interest, the test should focus its items on obtaining maximum information around the cut-off value. Thus, A high a parameter should be used and b should be at the average Theta level of all test subjects.

Question 13

Let’s assume a test is used to obtain the best possible measurements across the entire
skills spectrum. What test construction should be used to obtain maximum information? What role does the item discrimination parameter play in this?

Answer 13

Here, test information should be spread out over the entire spectrum of possible Theta values. So various levels of b should be used. The parameter a determines how focused the item information is around the point where Theta = b, so a somewhat lower a would be advisable, since this way, a broader area of Thetas yield similar information.