Extreme Psychometrics 2

Question 1

Often the accuracy of tests is overestimated due to what?

Answer 1

People not considering the base rate (the likelihood of the disease occurring in the population)

Question 2

What are two other terms for base rate in this context?

Answer 2

Pre-test probability or prior probability

Question 3

Explain the difference between Sensitivity and Specificity in this context

Answer 3

Sensitivity - percentage of people with the disorder who test positive (correct positive rate; i.e. correct positive divided by total with disease);
Specificity - percentage of people WITHOUT the disorder who test negative (correct negative rate; i.e. correct negatives divided by total without disease)

Question 4

What is Pre-test probability?

Answer 4

Percentage of people in the relevant population who have the disease (base rate/prior probability)

Question 5

How could you design a diagnostic test that would correctly diagnose every person who had a disease as having the disease?

Answer 5

By using a 2x2 contingency table, and calculating correct positives and negatives, and false positives and negatives

Question 6

Why might you want a test to have a liberal response bias?

Answer 6

It’s considered better to minimize false negatives, as they’re regarded as worse than false positives; it’s better to err on the side of giving a positive test result, even if there’s only a small chance of someone having the disorder, rather than giving them a negative result when they really do have the disorder

Question 7

Why is it important that health professionals understand the calculation described in this lecture?

Answer 7

So they don’t give people the wrong diagnosis and either make them panic unnecessarily, or think they’re in the clear when they’re not

Question 8

What procedure would you use to choose the optimal pass mark for your diagnostic test, if you wanted to maximize the discriminatory power of the test?

Answer 8

Use a ROC curve

Question 9

What is a ROC curve?

Answer 9

Receiving Operating Characteristic - a plot of correct positive rate (sensitivity) vs. false positive rate (1-specificity) where each point on the curve is a different “pass mark” for the test

Question 10

Why might we want to choose different cut offs for diagnostic tests under different clinical conditions?

Answer 10

So we can more specifically discriminate between people with a disorder and controls

Question 11

On a ROC curve, how can we quantify the accuracy (diagnostic ability) of the test?

Answer 11

By calculating the area under the curve; the more the line curves away from the diagonal, the better the test is at discriminating between people with or without the disorder

Question 12

In terms of sensitivity and specificity, how would you choose the “most discriminating” pass mark for a diagnostic test?

Answer 12

By looking at the point on the curve where the sum of sensitivity and specificity is highest (this point maximizes correct hits and correct misses, and minimizes false positives and false negatives)

Question 13

If your ROC curve tells you that the overall diagnostic accuracy of your test is .50, what does this mean?

Answer 13

This would be a straight diagonal, which is chance; i.e. you’re getting the same rate of correct positives as false positives so it’s pointless to perform the test

Question 14

Describe the Thompson Effect, and give 2 examples as to how this might lead to increased car crash risk

Answer 14

It’s an optical illusion, in which images that are reduced in contrast appear to move slower than they are; e.g. fog and eye cataracts (people with cataracts have 2.5 times the crash risk of controls)

Question 15

What is the Method of Constant Stimuli?

Answer 15

It involves generating a range of stimuli that vary in a particular region of interest (e.g. driving speed), presenting them in random order, and getting people to make a judgment on them (e.g. deciding whether each stimulus is faster or slower than a reference speed)

Question 16

What were the two main conditions and procedure in Mark’s experiment testing the Thompson Effect?

Answer 16

Both 60kph reference scene and 50-70kph test scenes clear; 60kph reference scene clear and 50-70kph test scenes foggy; got Ps to say whether they think the test scene is faster or slower than the reference scene and plotted a psychometric function for each P

Question 17

When plotting the psychometric functions of Mark’s driving experiment (using method of constant stimuli), what does the slope of the graph tell us?

Answer 17

People’s ability to discriminate between different speeds; the steeper the slope, the better people can tell them apart

Question 18

What does the point where the line crosses the 0.5 mark tell us on the psychometric function graph?

Answer 18

It tells us if there’s any systematic bias (e.g. if it crosses the 0.5 mark at the reference speed of 60kph then there’s no bias: 50% are likely to say the test scene speed is faster and 50% are likely to say it’s slower)

Question 19

How were the results of Mark’s driving experiment calculated and what were the findings?

Answer 19

They calculated the slope and 0.5 crossing point of the psychometric function for both the fog an clear conditions for each person by fitting a mathematically-defined curve to the data; they found that reduced contrast makes speeds look slower and makes it harder to tell different speeds apart

Question 20

What if people with cataracts calibrate for the shift in their perception of speed using their speedometers?

Answer 20

They may still have an increased chance of misjudging speed because they find it harder to tell different speeds apart

Question 21

What can we do about reducing the Thompson effect in driving?

Answer 21

Remove cataracts faster; increase contrast in road environment; give drivers perceptual training to improve their ability to tell apart different speeds