Week 10: Measurement Error

Question 1

Measurement error

Answer 1

The difference between the observed score and the true score

Question 2

List some sources of measurement error

Answer 2

• the test, if it is poorly designed
• the test taker and how they are feeling on the day
• the testing environment
• cultural factors e.g. language understanding

Question 3

Standard error of measurement

Answer 3

Provides the necessary information for assessing the measurement error associated with an individuals test performance. How far the mean from a sample is from the true mean of the population

Question 4

List in order the test development process

Answer 4

• conceptualisation
• construction
• tryout
• analysis
• revision

Question 5

Conceptualisation

Answer 5

How do you define a test? What are you measuring? What is useful and what is outside the bounds of this?

Question 6

Construction

Answer 6

Essentially developing the items, and considering why items are important for the test

Question 7

Tryout

Answer 7

Collecting test data and qualitative feedback on the test

Question 8

How is best to measure knowledge

Answer 8

Yes/no, multiple choice, short answer

Question 9

How is best to measure degree of symptoms/attitudes

Answer 9

Likert scale

Question 10

How is best to measure health

Answer 10

Visual analogue scale

Question 11

How is best to measure neurological function?

Answer 11

• visual items
• memory
• visuospatial tasks

Question 12

How is best to measure personality?

Answer 12

Q-sort

Question 13

List some important considerations when designing a test

Answer 13

• generate an item pool
• short and concise items
• age appropriate level
• no double barrelled items
• include negatively worded items

Question 14

Dichotomous

Answer 14

Two answers; Yes/no, true/false

Question 15

Polytomous

Answer 15

More than 2 alternatives, like a multiple choice

Question 16

Likert

Answer 16

Typically 5 items ranging from strongly disagree to strongly agree

Question 17

Likert type format

Answer 17

“Never”, “rarely”, etc.

Question 18

Q-sort

Answer 18

Cards with descriptions that need to be sorted via categories. Can be useful for clients but not for large samples.

Question 19

Describe semantic differential

Answer 19

Where a person falls between two opposing poles by marking a line

Question 20

Describe visual analogue

Answer 20

Indicating where you fall on a line

Question 21

Benefits of reverse scored items

Answer 21

Bias is controlled for and requires extra concentration from participants

Question 22

Category format

Answer 22

e.g. ‘On a scale from 1-10..’

Question 23

Some characteristics of category format

Answer 23

• has situational effects
• need clearly defined endpoints
• increasing categories above 10 can decrease reliability

Question 24

Classical test theory

Answer 24

• true score = observed score + error

- interested in total scores, of which one proportion is error and one proportion is true scores

Question 25

Item response theory

Answer 25

More about individual items and focuses on looking for a characteristic curve.

Question 26

The item characteristic curve

Answer 26

Describes the relationship between latent ability and performance on a test item

Question 27

What will the item characteristic curve ideally show

Answer 27

Someone who has a high ability should have a higher chance of doing well than someone who has low ability

Question 28

Describe criterion reference tests

Answer 28

Look at how well someone performs relative to others. Having a norm referenced test as the one we use most commonly.

Question 29

List the various response item formats

Answer 29

• dichotomous
• polytomous
• MC
• likert
• likert type
• Q sort
• true false
• two choice
• forced choice

Question 30

Describe item difficulty in terms of item analysis

Answer 30

The % of people who get an item correct informs the difficulty level, for example if 69% of individuals get it right, the difficulty level for that item is set at .69

Question 31

A good test…

Answer 31

• discriminates at many levels

- has varying degree of difficulty

Question 32

Extreme group method

Answer 32

The top 33% is compared to the bottom 33%, subtracting the proportion of items correct. The top proportion should have more items correct than the bottom.

Question 33

Point biserial method

Answer 33

Correlation between performance on particular items and performance on whole test