Test Development

Question 1

What are the first six steps to developing a measure?

Answer 1

1) overall goal and pre-planning
2) content definition
3) test specifications
4) item development
5) test design and assembly
6) test production

Question 2

What are the last six steps to developing a measure?

Answer 2

7) test administration
8) scoring responses
9) establishing passing scores
10) reporting results
11) item banking
12) test technical report

Question 3

What is the first step for developing a measure and what does it aim to do?

Answer 3

overall goal and pre-planning: provides a systematic framework for the project

Question 4

what happens during the overall goal and pre-planning stage of developing a measure?

Answer 4

(1) what is the aim or purpose of the test
(2) what construct will be measured
(3) what will the test format be
(4) how will it be administered. decide on a timeline for developing the test, how we will check the quality of the test, who will produce the test or publish or print it.

Question 5

What is an aim?

Answer 5

In the first step we have to be clear to specify the aim of the test/measure - specify what you want to achieve with the test.

Question 6

what are the different assessments?

Answer 6

screening or in-depth assessment

Question 7

describe screening tests

Answer 7

include fewer items and cover less content, quick test, easy to administer.

Question 8

describe in-depth assessments

Answer 8

more items, cover more content of the construct. A detailed test is generally more reliable, but can be much more time-consuming. Often detailed tests may also need special training on how to administer and interpret the test.

Question 9

what are the modes for interpretating outcomes?

Answer 9

Normative, Ipsative and Criterion-referenced

Question 10

what is the normative mode of interpretation?

Answer 10

compares scores to a norm group, score is compared to the average score of the rest of the sample of test-takers.

Question 11

what is the Ipsative mode of interpretation?

Answer 11

test that makes it possible to compare different aspects - so a test that allows for intra-individual comparisons

Question 12

what is the criterion-referenced mode of interpretation?

Answer 12

the performance is compared to a pre-defined status - what constitutes a ‘pass’ or ‘fail’

Question 13

what is the second step of developing a measure and what does it aim to do?

Answer 13

content definition: aims operationally define the construct you are measuring

Question 14

what does the word ‘operationalization’ mean?

Answer 14

refers to the act of defining or making a ‘fuzzy’ concept measurable. Define what you mean by the concept you are measuring. There are many ways in which you can define a specific concept, eg; The construct of intelligence.

Question 15

What is the second part content definition when developing a measure (step two)?

Answer 15

defining the purpose of the measure - what will you use the scores on the test for?

Question 16

What is step three of developing a measure and what does it aim to do?

Answer 16

test specifications: the test blueprint

Question 17

what are some factors in test specification?

Answer 17

1) test/response format
2) item format
3) test length
4) content areas of the constructs tested
5) whether items will contain visual stimuli
6) how test scores will be interpreted
7) time limits

Question 18

what are the types of response formats?

Answer 18

selected response, constructed response, performance response

Question 19

define the selective response format

Answer 19

A selected response, is something like a questionnaire with a Likert scale where a test-taker must choose to what extent they agree or disagree with a statement, selected one of the options, hence selected response.

Question 20

define the constructive response format

Answer 20

A constructed response is one where the test-taker must construct the answer, and generate it from their own knowledge. eg; where you are expected to respond by writing an essay, or filling-in-the-blank

Question 21

define the performance response format

Answer 21

a performance response is one where the test-taker needs to perform a task, such as where you are asked to build a puzzle, or build a design.

Question 22

what are the types of formats for response formats?

Answer 22

objective or subjective

Question 23

what is an objective response format?

Answer 23

objective response format would be very structured, where you pick one response – there is a right or wrong, or a definite answer such as with an MCQ test or a Likert scale for a questionnaire assessing personality (pick to what extent something resonates with you/applies to you)

Question 24

what is a subjective response format?

Answer 24

subjective format would be where the interpretation of the response depends on the examiner’s judgement, so more unstructured.

Question 25

give examples of psychological tests that use subjective response formats

Answer 25

Rorschach or TAT.

Question 26

what are the types of item formats?

Answer 26

open-ended items, forced-choice items, sentence-completion items, performance-based items

Question 27

define an open-ended item format

Answer 27

where there are no limitations on the test-taker.

Question 28

define a forced-choice item format

Answer 28

limited to how you can answer. You have to select within the options you are given

Question 29

define a sentence-completion item format

Answer 29

asked to complete a sentence - some open-endedness

Question 30

define a performance-based item format

Answer 30

items require you to perform a task, like write an essay or present an oral

Question 31

what is important when considering test length?

Answer 31

the amount of administration time available, what the purpose of the measure is, test fatigue, enough items to measure in enough detail

Question 32

what does ‘test content areas’ refer to?

Answer 32

ensure that all domains of the construct are tested

Question 33

how do we ensure all domains of a construct are tested?

Answer 33

a test structure (blueprint) outlining content areas (different categories/subcategories of the construct) and manifestations (How does the construct manifest)
content areas in table column, manifestations in table rows.

Question 34

what is the fourth step for developing a measure and what does it aim to do?

Answer 34

item development: involves setting the items in the test

Question 35

what are the guidelines for item development?

Answer 35

1) use clear wording to avoid ambiguity
2) use appropriate vocabulary
3) avoid double negatives
4) if writing a test for children consider a different format
5) don’t make questions too obvious

Question 36

what is the fifth step for developing a measure and what does it aim to do?

Answer 36

Test design and assembly: assembling the psychological test

Question 37

what things need to be considered when assembling/designing a test?

Answer 37

1) placement of correct items
2) check for errors
3) manual or computer assembly
4) how are people going to answer the test?
5) does the test look aesthetically pleasing?

Question 38

what is pre-testing?

Answer 38

where you administer the test to a representative sample from the target population to see how well it is working or if it is working in the way that you want it to.

Question 39

what does pre-testing aim to do?

Answer 39

Running a pilot assessment in this way, can tell us also whether people answer how we expect them to, if there are any problems or issues with the test, it the items are too easy or difficult, whether the time limit set for the test is enough.

Question 40

what is the sixth step for developing a measure and what does it aim to do?

Answer 40

test production: everything that goes into putting together the final version of the test, which includes finalizing which items are included, the sequence they are presented in, and the necessary visual stimuli such as images you’d like to include - It also involves quality control

Question 41

what is the seventh step for developing a measure and what does it aim to do?

Answer 41

test administration: Test administration is the most public and visible aspect of testing, when you administer your test to a sample.

Question 42

what are important factors of test administration?

Answer 42

security and standardization of testing conditions

Question 43

name methods for standardizing test administration

Answer 43

control extraneous variables, make conditions identical for all examinees, standard instructions, same time limits

Question 44

why is standardizing test administration important?

Answer 44

comparability between samples, fairness when interpreting test scores, increases validity of findings, increases reliability of findings

Question 45

what are steps eight and nine in developing a measure and what do they entail?

Answer 45

scoring responses and establishing a passing score: deciding how we will score a test or questionnaire

Question 46

what aspects go into scoring criteria for a psychological test?

Answer 46

developing a scoring key, decide when we drop participants from the study sample, investigate whether there is a response bias, evaluate how good or bad items are through item analysis

Question 47

what is item analysis?

Answer 47

analyze whether the test items are good or bad to decide whether to keep or omit their data

Question 48

what are the considerations when doing item analysis?

Answer 48

whether the items are congruent with the test objective, and whether they are valid – so questions surrounding validity, questions surrounding reliability, poor performing items that need to be discarded, which items are easy/difficult, how long it takes an examinee to complete each item

Question 49

what is step nine in developing a measure and what does it aim to do?

Answer 49

establishing passing scores: set up norms – what the distribution of scores look like in our target population, set performance standards, or decide on cut-off scores

Question 50

what do you have to achieve before establishing scores can take place?

Answer 50

ensure the test is reliable and valid

Question 51

what is step ten in developing a measure and what does it entail?

Answer 51

reporting results: all examinees have a right to accurate, timely and useful reports of their performance, in understandable language.

Question 52

what is step eleven in developing a measure and what does it entail?

Answer 52

Item banking: refers to storing items for future use.
NB! security

Question 53

what is step twelve in developing a measure, and what does it aim to do?

Answer 53

publishing and refinement: a technical report detailing the how one would go about administering the test and scoring it and much of the information detailed in the previous steps. You would also then submit the measure to be classified as a psychological test or not and sort out publishing and marketing should you want to charge for it

Question 54

what factors need to be considered when developing a multi-cultural test?

Answer 54

education level and language

Question 55

why is it important to develop multi-cultural tests?

Answer 55

South African samples are often multi-cultural. If you are targeting South Africans in general, you then have to be especially mindful during the planning phase of several things that might influence performance on tests.