Lecture 2

Question 1

Identify ways of measuring subjective judgements

Answer 1

1. EAI scales
2. VA scales
3. Paired comparisons - two stimuli are compared at a time
4. Direct magnitude estimation: listeners are provided with an “anchor”, which is assigned an arbitrary number

Question 2

What is the difference between reliability and agreement?

Answer 2

Reliability assesses the extent to which two judges rate the voices with the same RANKS (i.e. in the same order).

Agreement assess the extent to which the judges assigned identical SCORES to the same voices (within +/- 1 scale point).

Therefore the key difference is that reliability evaluates the relative rankings of scores, and agreement evaluates the raw scores themselves.

Question 3

How can we measure reliability and agreement if there is more than two judges?

Answer 3

Calculate the intraclass correlation, convert the scale scores to ranks, and use Kendall’s coefficient of Concordance.