Week 4 Test Development

Question 1

Define test conceptualisation

Answer 1

Starts with a question.
1. Review the literature - relevant theories/ constructs, definition, parameter setting

2. Obtain a clear, theory-informed conceptualisation and definition of the target construct
3. Develop an initial item pool

Question 2

Define test construction

Answer 2

.

Question 3

Define tryout

Answer 3

Administering the test on a representative sample using standardised instructions.

Data from the test tryout used to narrow down number of items

Question 4

What are 6 writing guidelines in test construction?

Answer 4

-

-

-

-

-

-

Question 5

What are 8 writing guidelines in test construction?

Answer 5

-

-

-

-

-

-

-

-

Question 6

What are the four types of scales used in psychological test construction?

Answer 6

.

Question 7

What are the pros/cons of Likert scales?

Answer 7

Pros
-
-
-
-

-

Question 8

What are the two types of response formats in test construction?

Answer 8

• Likert

- B. C S

Question 9

What are the pros/cons of Likert scales?

Answer 9

Pros
-
-
-
-

-

Question 10

What are the pros/cons of binary choice scales?

Answer 10

Pros
-
-
-
-

-
-

Question 11

What are the 4 types of response formats in test construction?

Answer 11

• Likert
• B. C S
• P. C
• C. S
• E or W

Question 12

What are the pros/cons of binary choice scales?

Answer 12

Pros
-
-
-
-

-
-

Question 13

What are the pros/cons of essay/ written format tests?

Answer 13

-
-

cons
-
-
-
-
-

Question 14

What are the 3 benefits of having an expert review an initial item pool before administering to the target sample?

Answer 14

• confirm/invalidate definition of construct by asking…
• evaluate the items c___ and c___
• identify other

Question 15

Articulate the criteria that assess whether an item is a ‘good item’

Answer 15

good test items = good tests
achieved through item analysis.

criteria:

• reliability
• validity
• discriminates at different level of trait/ability

Question 16

Articulate the criteria that assess whether an item is a ‘good item’

Answer 16

good test items = good tests
achieved through item analysis.

criteria:

• reliability
• validity
• discriminates at different level of trait/ability

Question 17

What are the item properties which you may investigate when trying to assess whether an item is a ‘good item’?

Answer 17

• item difficulty/ distribution
• ## --

Question 18

what is the formula for item difficulty index?

what does a high index mean?

What must also be considered when looking at optimal item difficulty index?

Answer 18

formula:
item difficulty index = examines who answered correctly/ total number of examinees

index= range between 0 & 1

high index =

consideration: probability of ___

Question 19

Binary choice scales (BCS)(true/false) have a high probability of one guessing the item correctly (0.5 out of 1 on the item difficulty-index).

a) What is an optimal item-difficulty index for BCS?

Answer 19

a) 0.75 out of 1 on the item-difficulty index. It is better to set the optimal item-difficulty index higher because more people w4

Question 20

Binary choice scales (BCS)(true/false) have a high probability of one guessing the item correctly (0.5 out of 1 on the item difficulty-index).

a) What is an optimal item-difficulty index for BCS?

A Multiple Choice Question test with 4 items has a item difficulty index of 0.25/ 1.

b) What is an optimal item-difficulty index for multiple choice questions?

Answer 20

a) 0.75 out of 1 on the item-difficulty index. It is better to set the optimal item-difficulty index higher because test takers have a high chance (0.5) of guessing correctly.
b) 0.625. It is lower because MCQ takers have a lower probability of guessing correctly (0.25/1)

Question 21

In item analysis, there is an item property called dimensionality (factor analysis). Why would you use it?

Answer 21

Having a set of items doesn’t mean you have a scale

• items may not have underlying variable OR
• they may have multiple underlying variables

Question 22

What can factor analysis help with?

Answer 22

• determining the # of underlying latent variables or constructs
• help condense information
• define content or meaning of factors
• identify that are performing better/worse e.g. items which don’t fit into any factor/fit into multiple –> elimination

Question 23

What are the factor analysis decisions

Answer 23

number of factors to extract

• Eigenvalues (>1)
• Scree plot

Rotation

• helps interpret the data
• oblique: assumes factors are correlated
• Orthogonal: assumes factors uncorrelated

Question 24

How is item reliability measured?

Answer 24

measures internal consistency of test

Question 25

How is item reliability measured?

Answer 25

measures internal consistency of test - item-discrimination index - item-characteristic curve

Question 26

When a test developer is deciding to retain or delete an item, is poor performance on ONE task/ aspect of item analysis sufficient enough to delete the item?

Answer 26

no

Question 27

What is an indication of item with good a good item-characteristic curve?

Answer 27

e.g. individuals with a low ability score lowly, moderate ability score in the middle range and those with a high ability score highly on a test.

Question 28

In the final stage of test development, test revision, one must undergo cross validation. what is cross validation?

Answer 28

administering revised test in another population to determine its applicability for this population. it is a test of validity. sometimes validity shrinkage occurs