Chapter 5: Reliability

Question 1

this term refers to the consistency of a
measurement

It indicates whether a test yields
stable and consistent results over time and across different contexts

Answer 1

reliability

Question 2

what is reliability determined by?

Answer 2

Reliability is determined by the proportion of total variance in test scores that can be attributed to true variance (i.e., actual differences in the ability being measured). The higher this proportion, the more reliable the test.

Question 3

this type of variance represents actual
differences in ability.

For example, differences in math skills among friends taking a test reflect ______ ______

Answer 3

true variance

Question 4

this term refers to an index that quantifies reliability, expressed as the ratio of true score variance to total variance.

Answer 4

reliability coefficient

Question 5

this type of variance represents variability
due to irrelevant or random factors, such as distractions or fatigue, affecting scores even if the true ability is constant

Answer 5

Error Variance

Question 6

this term encompasses all factors
affecting the measurement process that are not related to the variable being assessed.

Answer 6

Measurement Error

Question 7

this type of measurement error is caused by unpredictable fluctuations, resulting in inconsistencies without a discernible pattern

This type of error can lead to score variability without biasing the results
systematically.

Answer 7

Random Error

Question 8

this type of measurement error refers to a consistent error that may affect scores in a predictable manner, making it possible to identify and correct

This type of error does not compromise the consistency of scores

Answer 8

Systematic Error

Question 9

this source of error variance is seen during test construction and arises from differences among test items, both within a single test and across different test

Answer 9

Item Sampling/Content Sampling

Question 10

what are the three main sources of error variance?

Answer 10

1. Test Construction
2. Test Administration
3. Test Scoring and Interpretation

Question 11

this source of error variance refers to the conditions under which the test is administered can impact performance.

Answer 11

Test Environment

Question 12

this source of error variance refers to the subjectivity in scoring, technical issues,
or glitches can lead to inconsistent
results.

Answer 12

Scorers and Scoring Systems

Question 13

this source of error variance refers to factors such as emotional distress, physical discomfort, lack of sleep, or the influence of drugs and medication can affect the test-taker’s performance and focus.

Answer 13

Test-Taker Variables

Question 14

this source of error variance refers to the physical appearance and demeanor of
the examiner, their presence or absence, and their level of professionalism can influence the test-taker’s experience.

Answer 14

Examiner-Related Variables

Question 15

this source of error variance refers to the extent to which the sample population accurately represents the broader population.

Discrepancies in demographics, political affiliation, or other relevant factors can lead to biased results.

Answer 15

Sampling Error

Question 16

this type of reliability estimate measures the consistency of results between different versions of a test designed to assess the same construct

it assesses how well two different forms of a test yield similar scores when administered to the same individuals.

Answer 16

Parallel-Forms and Alternate-Forms Reliability Estimates

Question 17

this source of error refers to issues such as ambiguous wording in questionnaires or biased items can skew results, favoring one response or candidate over another

Answer 17

Methodological Error

Question 18

this type of reliability estimate is defined as a method for estimating the
reliability of a measuring instrument by
administering the same test to the same group at two different points in time.

Answer 18

Test-Retest Reliability Estimate

Question 19

when do we apply test-retest reliability estimate?

Answer 19

it is best used for measuring stable
traits (e.g., personality) rather than fluctuating characteristics.

Question 20

this term refers to the different versions designed to be equivalent but may not meet the strict criteria of parallel forms

Answer 20

Alternate Forms

Question 21

this term refers to the two forms of a test are statistically equivalent in terms of means and variances

Answer 21

Parallel Forms

Question 22

what is the index for parallel and alternate-forms reliability?

Answer 22

coefficient of equivalence

Question 23

what is the method for test-retest reliability?

Answer 23

correlate scores from the same individuals across two test administrations.

Question 24

what is the index for test-retest reliability?

Answer 24

coefficient of stability

Question 25

what are the reliability estimates categorized as external consistency estimates?

Answer 25

test-retest reliability parallel-forms and alternate-forms reliability estimates

Question 26

what are the reliability estimates categorized as internal consistency estimates?

Answer 26

internal consistency reliability split-half reliability

Question 27

this type of reliability can be assessed without creating alternate forms or re-administering tests This involves evaluating the consistency of test items.

Answer 27

Internal Consistency Reliability

Question 28

this type of reliability estimate estimates reliability by correlating scores from two equivalent halves of a single test.

Answer 28

Split-Half Reliability

Question 29

what are the acceptable splitting methods for split-half reliabilty?

Answer 29

1. Randomly assigning items to halves 2. Using odd and even item assignments (odd-even reliability). 3. Dividing by content to ensure both halves measure equivalent constructs

Question 30

what are the steps in conducting split-half reliability?

Answer 30

1. Divide the test into two equivalent halves. 2. Calculate the Pearson correlation between the two halves 3. Adjust the reliability using the Spearman-Brown formula.

Question 31

this type of measurement error refers to unpredictable fluctuations that introduce variability without a systematic pattern.

Answer 31

Random Error

Question 32

this term refers to all factors affecting the measurement of a variable that are not related to the variable itself

Answer 32

Measurement Error

Question 33

this type of measurement error refers to consistent errors that affect scores predictably, allowing for correction once identified.

Answer 33

Systematic Error

Question 34

what are the methods for estimating internal consistency reliability?

Answer 34

1. KR-20 2. KR-21 3. Cronbach's Alpha

Question 35

when is KR20 appropriate?

Answer 35

when it is a true dichotomous test (true or false)

Question 36

when is KR21 appropriate?

Answer 36

when it is an artificial dichotomy (multiple choice, binary scoring)

Question 37

this reliability estimate measures the degree of difference between item scores rather than similarity is calculated based on absolute differences between item scores and is less affected by the number of items on a test

Answer 37

Average Proportional Distance

Question 38

when is Cronbach's alpha appropriate?

Answer 38

when it is a true multiple choice test (Likert Scale)

Question 39

this reliability estimate measures the correlation among all items on a scale from a single test administration, assessing the homogeneity of the test

Answer 39

Inter-Item consistency

Question 40

when is inter-scorer reliability used?

Answer 40

Frequently used in coding nonverbal behavior. For example, a researcher may create a checklist of behaviors (like looking downward or moving slowly) to quantify aspects of nonverbal cues indicating depressed mood.

Question 41

this term refers to the degree of agreement or consistency between two or more scorers regarding a particular measure

Answer 41

Inter-Scorer Reliability

Question 42

how is inter-scorer reliability calculated?

Answer 42

it is calculated using a correlation coefficient, referred to as the coefficient of inter-scorer reliability

Question 43

what number should the coefficient be for it to be considered as highly reliable? Considered excellent (grade A); crucial for high-stakes decisions.

Answer 43

0.90s

Question 44

what number should the coefficient be for it to be considered as moderately reliable?

Answer 44

0.80s

Question 45

what number should the coefficient be for it to be considered as having a low reliability? Weak, indicating potential issues with the test's effectiveness

Answer 45

0.65-0.70s

Question 46

what number should the coefficient be for it to be considered as having an unacceptable reliability?

Answer 46

below 0.50

Question 47

when is a test considered homogenous?

Answer 47

A test is considered homogeneous if it is functionally uniform, measuring a single factor (e.g., one ability or trait).

Question 48

these tests are designed to indicate how a test-taker performs relative to a specific criterion or standard (e.g., educational or vocational objectives). these tests focus on measuring whether test-takers meet predetermined criteria rather than comparing their scores to those of others

Answer 48

criterion-referenced tests

Question 49

when is a test considered heterogeneous?

Answer 49

A test is heterogeneous if it measures multiple factors or traits. In such cases, internal consistency estimates may be lower, whereas test-retest reliability might provide a more appropriate measure of reliability

Question 50

what reliability estimates are appropriate for static characteristics?

Answer 50

test-retest or alternate-forms methods

Question 51

these tests consist of items of uniform difficulty (typically low), allowing all test-takers to complete all items correctly within generous time limits

Answer 51

Speed Tests

Question 52

these tests are administered time limit that allows test-takers to attempt all items Contains difficult items, with the expectation that no test-taker can achieve a perfect score

Answer 52

Power Tests

Question 53

this theory is also known as the true score model is the most widely used model of measurement in psychology due to its simplicity relative to more complex models

Answer 53

Classical Test Theory

Question 54

this term represents the value that genuinely reflects an individual's ability or trait level as measured by a particular test. This value is highly dependent on the specific test used

Answer 54

True Score

Question 55

this theory posits that a test's reliability is determined by how accurately the test score reflects the domain from which it samples

Answer 55

Domain Sampling Theory

Question 56

this theory suggests that test scores can vary across different testing situations due to various situational factors

Answer 56

Generalizability Theory

Question 57

this term refers to the complete range of items that could measure a specific behavior, viewed as a hypothetical construct.

Answer 57

Domain of Behavior

Question 58

this term assesses how well scores from a specific test can be generalized across different contexts. Coefficients of generalizability represent the influence of particular facets on test scores

Answer 58

Generalizability Study

Question 59

this type of study evaluates the utility of test scores in assisting users in making informed decisions.

Answer 59

Decision Study

Question 60

this theory is an alternative to CTT that models the probability of an individual with a certain level of ability performing at a specific level Often referred to as latent-trait theory because it measures constructs that are not directly observable (latent).

Answer 60

Item Response Theory (IRT)

Question 61

what does discrimination mean for IRT?

Answer 61

discrimination measures the extent to which an item can differentiate between individuals with higher or lower levels of the trait or ability being assessed.

Question 62

what are the key concepts in IRT?

Answer 62

Difficulty and Discrimination

Question 63

what does difficulty mean for IRT?

Answer 63

The attribute of an item indicating how challenging it is to accomplish, solve, or comprehend.

Question 64

what are the two types of test items?

Answer 64

1. Dichotomous Test Items 2. Polytomous Test Items

Question 65

this type of test items have two possible responses, such as true/false or yes/no.

Answer 65

dichotomous test items

Question 66

this type of test items have three or more possible responses, where only one response is correct or aligned with the targeted trait or construct

Answer 66

polytomous test items

Question 67

this term refers to the range or band of test scores that is likely to contain the true score

Answer 67

confidence interval

Question 68

this statistical tool is used to estimate or infer the extent to which an observed score deviates from a true score

Answer 68

standard error of measurement

Question 69

this statistical measure can aid a test user in determining how large a difference should be before it is considered statistically significant

Answer 69

standard error of the difference

Question 70

how does restricted variance influence correlation coefficients?

Answer 70

it leads to lower correlation coefficients since the diversity of scores is limited

Question 71

how does inflated variance influence correlation coefficients?

Answer 71

it results in higher correlation coefficients due to a broader spread of scores.