W11 - Reliability

Question 1

What is the difference between psychological research and psychological assessment

Answer 1

Psychological Research:

Generalisations about representative samples of people.

Psychological Assessment:

Generalisations about specific individuals (n = 1)

Question 2

What are some psychological assessment standards

Answer 2

1. Nature of underlying construct(s)
2. Basic psychometric principles and procedures: Requirements and Limitations
3. Directions for administration and properties: Purpose, relevant standard errors, reliabality and validity

Question 3

What is a valid test

Answer 3

A test is valid if it accurately measures what it purports to measure

Question 4

What is a reliable test

Answer 4

Property of consistency in measurement.

Question 5

Reliability and Validity. Necessary and Sufficiency.

Answer 5

Reliability is a necessary, but insufficient, requirement for validity.

(i.e. valid test cannot be unreliable, but reliable test may not valid)

Question 6

Is reliability a binary decision?

Answer 6

No. Reliability is continous, not categorical (Reliable/Not Reliable)

Question 7

What is the first equation of Classical Test Theory

Answer 7

X_i = T + E_i

X_i = Observed score on test occassion i

T = True Score

E_i = Error on test occassion i, Unsystematic variance.

Question 8

What are the two properties of errors in classical test theory

Answer 8

1. Endogenous: Factors about test-taker (client’s condition)
2. Exogenous: Factors outside test-taker (psychologist measurement)

Question 9

What are the assumptions of Classical Test Theory

Answer 9

1. Expected value of error = 0
2. Errors do not correlate with one another
3. Errors do not correlate with true scores
4. Expected value of test = True Score (On repeated administration of test, on average people will score their true score)

Question 10

Elaborate on first assumption of classical test theory

Answer 10

1. Expected value of error is zero

When all the errors on different test occassion adds up, it will be equal to 0

Question 11

Elaborate on the second assumption of classical test theory

Answer 11

2. Errors do not correlate with one another

Errors on Testi does not affect error on Testj

Question 12

Elaborate on the third assumption of classical test theory

Answer 12

3. Errors do not correlate with true score

r (te) = 0. Positive/Negative error is unrelated to true score

Question 13

Elaborate on the fourth assumption of classical test theory

Answer 13

4. Expected value of test equals to true score

Average of all observed scores = True Score

Question 14

What is the second equation of Classical Test Theory

Answer 14

𝜎²_{x =} 𝜎²_t + 𝜎²_𝜖 + 2cov(t,𝜖)

𝜎²_x : Variance of observed scores

𝜎²_t : Variance of true scores

𝜎²_𝜖: Variance of error scores

2𝐶𝑜𝑣(𝜏,𝜖): Covariance between true scores and error scores, which is 0 under assumption (3)

Question 15

What is the third equation of classical test theory, relating to how reliability is calculated

Answer 15

• 𝑅𝑒𝑙𝑖𝑎𝑏𝑖𝑙𝑖𝑡𝑦
• 𝜌²_𝑥𝜏
• (𝜎_𝜏²/𝜎_𝑥²)
• (𝜎_𝜏²)/ (𝜎_𝜏²+𝜎_𝜖²)
• (𝑆𝑖𝑔𝑛𝑎𝑙)/(𝑆𝑖𝑔𝑛𝑎𝑙+𝑁𝑜𝑖𝑠𝑒)

𝜌²_𝑥𝜏: Theoretical reliability coefficient

𝜎_𝜏² : Variance True Scores

𝜎_𝑥² : Varaince Observed Scores

𝜎_𝜖² : Variance Error Scores

Question 16

Persepctives on reliability: Conceptual and statistical basis

Answer 16

Conceptual: Observed score in relation to:

(a) True Score; (b) Measurement error

Statistical; (a) Proportion of variance (b) Correlations

Question 17

Perspective 1: Using true score and proportion of variance

Answer 17

Ratio of true score variance to observed score variance (Same as second equation)

r_xx = S_t²/ S_x²

Question 18

Perspective 2: Using measurement error and proportion of variance

Answer 18

Reliability is the lack of error variance

r_xx = 1 - (S²_𝜖/ S²_x)

Question 19

Perspective 3: Using true score and correlations

Answer 19

Reliability is the squared correlation between
observed scores and true scores

r_xx = r²_xt

Question 20

Perspective 4: Using measurement error and correlation

Answer 20

Reliability is the lack of correlation between
observed scores and error scores

r_xx = 1 - r²_x𝜖

Question 21

Reliability in practice, since we do not know the true score varaince, what are some assumptions we add on in a parallel test.

Answer 21

We run parallel test: This test must have:

(1) Tau equivalent. True score on both test is the same
(2) Same level of error variance

Question 22

What are the 3 ways of testing reliability

Answer 22

1. ) Test-retest
2. ) Parallel-form reliability
3. ) Split-half reliability

All three assumes that they are parallel forms of the test

Question 23

What is test-retest reliablity

Answer 23

Correlation between original test and retest (Same test, different time)

Question 24

What is the use of test-retest reliability

Answer 24

Useful for stable traits, not useful for transient states

Question 25

What are the cons of test-retest relability

Answer 25

* Carryover Effects (Smaller gap between test and re-test) * Googling Answers * Bored * Remember test items * True score may vary * Participants may fail to return (Bigger gap between test and re-test) Trade off between paticipants failing to return and carry over effects

Question 26

What is parallel-form reliability

Answer 26

Correlation between two parallel forms of the test

Question 27

Parallel form reliability: What must be ensured?

Answer 27

1. Parallel form must measure same set of true scores 2. Parallel form must have equal vairance as original form

Question 28

What are the pros of parallel form reliablity

Answer 28

It can be used on the same day

Question 29

What are the cons of parallel form reliablity

Answer 29

* Might not truly be parallel * Affects true score * Carryover effects * Even though there is no direct memory effects from orginal test, they might still learn stuff

Question 30

What is split-half reliability

Answer 30

Correlations between 2 sub-tests split from 1 test

Question 31

What is the pros of split-half reliability

Answer 31

Only one test. Easy

Question 32

What are the cons of split-half reliability

Answer 32

* Might not truly be parallel * *Deflation* of reliability estimate as subtests have only half the items compared to main test

Question 33

What is cronbach's alpha

Answer 33

Means of all possible split-half reliabilites, scaled up to a full test instead of a half-test

Question 34

Is cronbach's alpha legit. Why?

Answer 34

Not really. Provides a conservative, lower-bound estimate for reliability and recent study suggest it's of limited use.

Question 35

What does the reliability coefficient (r_xx) fail to do?

Answer 35

It does not tell us in test score units how much measurement error is 'typical' as it is not expressed in test units.

Question 36

What is standard error of measurenet (SEm)

Answer 36

Average error score (i.e. SD of erorrs)

Question 37

What is the formula for SE_m

Answer 37

SE_m = s_x [√(1 - r_xx)] s_x = standard deviation of observed scores r_xx = reliability coefficient

Question 38

If the test is completely unreliable, what is the standard error of measurement

Answer 38

SE_m = s_x [√(1 - r_xx)] If r_xx = 0 Hence, SE_e = S_x Standard error of measurement = Standard deviation of observed scores

Question 39

If the test is completely reliable, what is the standard error of measurement

Answer 39

SE_m = s_x [√(1 - r_xx)] Since r_xx = 1, Hence, SE_e = 0 No standard errors of measurement

Question 40

What is the direction of association between reliability and SEm as a proportion of SD

Answer 40

Negative. As reliability increases, SEm decreases

Question 41

According to Nunnally, what is the reliablility needed?

Answer 41

Bare minumum = 0.9 Desirable = 0.95

Question 42

What is the equation to predict a client's true scores

Answer 42

T\_hat = (r_xx)(x) + (1 - r_xx)(𝜇_T) * T\_hat = predicted true score * r_xx = reliability * x = observed score * 𝜇_T = population mean for the test (e.g. IQ = 100)

Question 43

What if the predicted true score if the test was completely unreliable

Answer 43

T\_hat = (r_xx)(x) + (1 - r_xx)(𝜇_T) If r_xx = 0, T\_hat = 𝜇_T (population mean)

Question 44

What if the predicted true score if the test was completely reliable

Answer 44

T\_hat = (r_xx)(x) + (1 - r_xx)(𝜇T) If r_xx = 1, T\_hat = x (that is, the observed scores)

Question 45

What is the direction of association between reliability and predicted true scores (T\_hat)

Answer 45

As reliability increases, T\_hat moves closer to observed scores. As reliability decreases, T\_hat regress towards the population mean.

Question 46

What is true score confidence intervals built upon

Answer 46

Standard error of estimation

Question 47

What is the equation for standard error of estimation

Answer 47

SE_e = S_x [√r_xx(1 - r_xx )] * Similar to standard error of measurement (SE_m). but with an extra r_xx. * Note: S_x = Standard Deviation of Test

Question 48

What defines the 95% CI for predicted true scores

Answer 48

Lower Bound: [T\_hat - (1.96 x SE_e)] Upper Bound: [T\_hat + (1.96 x SE_e)]

Question 49

What are the correlations between measures compared to correlations between constucts. Why?

Answer 49

* **Observed correlations** between two _measures_ x and y is always lower than **true correlation** between _underlying constructs_ * Because observed correlations is attenuated/reduced by measurement error

Question 50

What does the disattenuation formula aim to

Answer 50

* Estimates the correlation if 2 constructs were not affected by measurement error * Corrects for the fact that measurement error attenuates the correlation between 2 constructs measured

Question 51

What is the maximum correlation between 2 measures x and y

Answer 51

Max r_xy = √r_xxr_yy * r_xx * _​Reliability of test x * r_xy * _​Reliability of test y * r_xy * _​Observed correlation of underlying constructs x and y

Question 52

What is the disattenuation formula

Answer 52

r^'_xy = r_xy / √r_xxr_yy * r^'_xy * _​Correlation between 2 constructs without measurement error * r_xy * _​Obseved correlation between 2 constructs * r_xx * Reliability of construct x * r_yy * _​Reliability of construct y

Question 53

How to increase the correlation between test scores and constructs

Answer 53

* Increase the relationship between construct and test * Quality of Items * Remove inconsistency in test administration and interpretation * Reduce measurement error (Exogenous) * Increase no. of test items * Quantity of items

Question 54

What is the Spearman-Brown Prophecy Formula

Answer 54

r'_xx = (nr_xx) / [1 + (n-1)(r_xx)] * r'_xx * _​Reliability of expanded test * r_xx * _​Reliability of original test * n * Expansion factor (e.g. n = 2 means double items)

Question 55

What is the relationship between expansion factor and reliability of expanded test. What is the caveat?

Answer 55

* Negative acceleration * Which has practical benefit * New items must be as good. If not, reliability of expanded test (r^'_xx ) might be even wprse

Question 56

If we know what is our desired reliability, what is the forumla to work out the expansion factor

Answer 56

n = (r'_xx)(1 - r_xx) / (r_xx)(1-r^'_xx) r'_xx = Reliability of expanded test r_xx = Reliability of original test