M&D3 Week 1

Question 1

Reliability

Answer 1

the consistency of a measurement repeated within a person or in a sample

Question 2

test score

Answer 2

X = T + E
T (true score) is not known, E (error) is only estimated (X = the obtained score)

Question 3

Accidental measurement errors (unpredictable)

Answer 3

Something you can’t predict beforehand, time of day, headache

Question 4

Systematic measurement errors (constant)

Answer 4

Something you can predict beforehand, happens systematically

Question 5

Confidence interval formula

Answer 5

CI = X +/- z * SE
X = score; z = standardized coefficient, SE = standard error

Question 6

z = 0.99

Answer 6

the chances (odds) is 68% that the true score is located within the interval (lower precision, lower CI)

Question 7

z = 1.96

Answer 7

the odds are 95% that the true score located within the interval (medium precision, medium CI)

Question 8

z = 2.58

Answer 8

the odds are 99% that the true score located within the interval (higher precision, higher CI)

Question 9

Confidence intervals indicate…

Answer 9

the limits within which a certain possible score may be
assumed to be true

Question 10

Standard error (SE)

Answer 10

Standard deviation of raw scores around true scores (SD of measurement errors)

Question 11

SE formula

Answer 11

SE = σ * √(1 - rxx)

Question 12

The higher rxx

Answer 12

the lower SE

Question 13

The lower σ

Answer 13

the lower SE

Question 14

Test-retest

Answer 14

A specific test is used multiple times or at least two times (coefficients between these tests)

Question 15

Parallel (or alternative) versions

Answer 15

Coefficients between the different versions

Question 16

Internal consistency measures (split-half, Cronbach’s alpha, KR-20)

Answer 16

Test used one time! KR-20 is for dichotomous items (only two options)

Question 17

Raters coefficient: interclass correlation (ICC)

Answer 17

Consistency between responses of a specific group/reactions of a specific team or group

Question 18

Alpha coefficient is based on

Answer 18

1. A single measurement of a test
2. (Co)variances of the items
3. The number of items – how many items do you have

Question 19

insufficient alpha coefficient

Answer 19

rxx < .80

Question 20

Validity

Answer 20

The extent to which a test measures what it should measure

Question 21

Face validity

Answer 21

How the test seems externally (to laymen/test takers)

Question 22

Construct validity

Answer 22

To what extent is the test a good measurement of the underlying theoretical concept?

Question 23

Convergent validity

Answer 23

To what extent is the test correlated with other measures measuring the same
concept (should have high pos or neg correlation)

Question 24

Divergent validity

Answer 24

To what extent is the test correlated with other measures measuring a different concept (should have no correlation)

Question 25

Criterion validity, diagnostic validity

Answer 25

How much the test predicts a concrete criterion, how much it has value in diagnosing

Question 26

Content validity

Answer 26

Does the test cover the domain of knowledge, skills, behavior that we’re supposed to measure?

Question 27

Internal structure

Answer 27

the structure of the test alone o Number of dimensions (factors) o Score differences between groups (e.g., high versus low extraversion)

Question 28

External validity

Answer 28

how the questionnaire correlates with other measures

Question 29

Multitrait-multimethod matrix used to test

Answer 29

for convergent and divergent validity

Question 30

Convergent validity

Answer 30

Hetero-method (more than one method) Mono-trait (the same trait measured by different methods)

Question 31

Divergent validity

Answer 31

Across two or more methods Hetero-method (more than one method) Hetero-trait (different traits)

Question 32

Method variance

Answer 32

Within a single method mono-method hetero-trait

Question 33

FOUR RULES FOR TESTING CONSTRUCT VALIDITY BASED ON MTMM MATRIX

Answer 33

1. Convergent validity (average) > 0 2. Convergent validity (average) > divergent validity (average) 3. Convergent validity (average) > method variance (average) 4. Method variance and divergent validity approximately the same pattern in correlation matrix

Question 34

Immediate criteria

Answer 34

work sample in an Assessment Centre

Question 35

Low delay

Answer 35

first client satisfaction

Question 36

High delay

Answer 36

annual evaluation

Question 37

Operationalization

Answer 37

changing the conceptual ‘thing’ into something measurable

Question 38

Incremental = increased validity

Answer 38

how much personality adds to measuring cognitive capacity

Question 39

COTAN standards in personnel selection, a validity coefficient of ... is seen as sufficient

Answer 39

.40

Question 40

Cohen (1977) standards (effect sizes) :

Answer 40

r = .10 low effect size r= .30 medium r= .50 high

Question 41

Possible problems with values that are too high (applies to other values too such as correlation coefficients)

Answer 41

1. Incorrect conceptualizations of your measure 2. Selection of instrument to test for criterion validity is not correct

Question 42

The correction for attenuation formula

Answer 42

assumes there is an ideal value of validity & at some point, if instruments are perfectly reliable, the maximum available can be calculated