Chapter 5

Question 1

Reliability (def)

Answer 1

consistency in measurement

Question 2

Reliability coefficient

Answer 2

0 to 1 statistic.

Question 3

4 types of reliability coefficients

Answer 3

1) test-retest reliability

2) alternate-forms reliability

3) split-half reliability

4) inter-scorer reliability

Question 4

Measurement error (textbook def)

Answer 4

Inherent uncertainty with any measurement, even after minimizing preventable mistakes

Question 5

2 influences that interfere with repeated measurement (in psych)

Answer 5

1) changes in object (eg. a constant flux of mood, alertness, motivation)

2) the act of measurement (i.e., carryover effects like fatigue, practice)

Question 6

“True Score”

Answer 6

not actually true to concept. True score is tied to the specific measurement instrument.

Question 7

What ‘score’ measures the truth independent of measurement?

Answer 7

Construct score.

the underlying score of some construct (eg. depression)

Question 8

variance is made of what two subtypes of variance?

Answer 8

True variance (actual differences between people?) + Error variance (random variances that are irrelevant)

Question 9

Define reliability in terms of variance

Answer 9

Proportion of total variance attributed to true variance

Question 10

Random vs. Systematic Error

Answer 10

Random: unpredictable, inconsistent, without pattern

Systematic: predictable, constant, can be adjusted for

Question 11

Bias (in error)

Answer 11

The degree of systematic error that influences measurement

Question 12

How does item/content sampling contribute to error variance?

Answer 12

The specific content in some test may affect the results (eg. i hope they ask this question and not this)

Question 13

What test administration effects contribute to error variance?

Answer 13

Environment: war, heat, gum, pencil, etc.

Testtaker variables: lack of sleep, emotions, drugs, etc.

Examiner-related variables: physical appearance, presence/absence

Question 14

How does test scoring and interpretation contribute to error variance?

Answer 14

Some subjectivity in certain tests (eg. essays, creativity, etc.) can influence measurement.

Question 15

test-retest reliability coefficient is also called what?

Answer 15

Coefficient of stability

Question 16

What might affect test-retest reliability estimates?

Answer 16

Experience, practice, memory, fatigue, etc. may intervene.

Question 17

alternate-forms/parallel-forms reliability estimates coefficient name

Answer 17

Coefficient of equivalence

Question 18

Parallel vs. Alternate forms reliability

Answer 18

Parallel forms: Means and variances of test scores are equal

Alternate forms: different versions of same test, but aren’t parallel

Question 19

2 similarities between parallel/alternate and test-retest reliability

Answer 19

1) two test administrations with same group

2) test scores can be affected by factors like fatigue, practice, learning, etc.

Question 20

What additional source of error variance is present in alternate/parallel-forms reliability?

Answer 20

Item/Content sampling

Question 21

Split-half reliability

Answer 21

Correlating two pairs of scores from a single test.

one half of a test Pearson r with another half, then adjust with Spearman-Brown formula

Question 22

Odd-even reliability

Answer 22

split-half reliability by using odd vs. even numbers

Question 23

How do number of items affect reliability coefficient? What method can see how many items needed?

Answer 23

Spearman-Brown.

More items is more reliability

Question 24

What coefficient for inter-item consistency?

Answer 24

Coefficient alpha

Question 25

Inter-scorer reliability What coefficient?

Answer 25

Degree of consistency between 2 or more scorers. Coefficient of inter-scorer reliability

Question 26

DSM-5 Inter-rater reliability

Answer 26

Kappa = 0.44 (fair level moderately greater than chance)

Question 27

Transient error

Answer 27

Error due to testtaker's feelings, moods, or mental state over time

Question 28

Homogeneity vs. Heterogeneity of test items

Answer 28

Homogenous: Functionally uniform items. Measures one factor (eg. one ability/trait). High internal consistency should happen Heterogenous: Not just one factor measured in the test.

Question 29

Does high internal consistency mean homogeneity of items?

Answer 29

Not necessarily. More items will lead to high internal consistency coefficients as long as they're positively correlated

Question 30

Dynamic vs. static characteristics

Answer 30

Dynamic: Presumed to be relatively situational and changing Static: presumed to be relatively unchanging

Question 31

Restriction/Inflation of range

Answer 31

When some subgroup inflates or restricts the correlational analysis??

Question 32

Power Test

Answer 32

Enough time to attempt all items, but so difficult that nobody gets perfect score

Question 33

Speed test

Answer 33

Same level of difficulty in items and testtakers should complete everything correctly if unlimited time. But only some will be able to complete the whole test

Question 34

What's differences in assumptions between CTT and IRT? (not specific, but ya..)

Answer 34

CTT assumptions are weak and easily met. IRT are rigorous.

Question 35

Domain Sampling Theory

Answer 35

Reliability is based on how well a score assesses the domain of where a sample is drawn.

Question 36

What is universe score in generalizability theory?

Answer 36

The true score (given same conditions, the same score will be obtained)

Question 37

Generalizability Study Coefficient of generalizability

Answer 37

how generalizable scores from a particular test are if administered in different situations.

Question 38

Decision study

Answer 38

Usefulness of test scores in helping user make decisions. Follows generalizability study

Question 39

Another way to say Item response theory

Answer 39

Latent-trait theory

Question 40

Within CTT, what is the weight assigned to each item on a test?

Answer 40

Equal weight. IRT is differentital weight.

Question 41

Dichotomous test items

Answer 41

can only answer with one of two responses

Question 42

Polytomous test items

Answer 42

3 or more alternative responses

Question 43

Rasch Model

Answer 43

a type of IRT model with underlying distribution assumption

Question 44

Which measure is used to compare differences between scores?

Answer 44

Standard error of the difference