Measurement Reliability Flashcards

1
Q

Measurement Reliability

A
  • extent to which repeated measurements agree with one another and are believable and useful
  • also referred to as stability, consistency, and reproducibility
  • sources of error include errors made by examiners, subject variability, and instrumentation flows or failures
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Types of Measurement Reliability

A
  • instrument: test-retest, internal consistency, parallel forms, split-half
  • rater: intra-tester (within), inter-tester (between or among)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Test-Retest Reliability

A

-obtained by administering the same test twice over a period of time

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Instrument Reliability

A
  • internal consistency: measure of reliability used to evaluate the degree to which different test items that cover the same construct produce similar results; self-report instruments; grouping of questions that each measure different constructs or concepts; all items in one domain should not relate to another domain
  • parallel forms: two forms of an instrument are administered; score equally on both instruments;
  • split half: combine two forms of an instrument that cover the same concept into one longer version; compare scores on one half with the other
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Quantification of Reliability

A
  • relative: if measurement is reliable, individual measurements within a group will maintain their position within the group upon repeated measurement-very reliable but may not be near measurement score
  • absolute: the extent to which a score varies upon repeated measurement-no change from day one to day two
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Measurement Validity

A
  • degree to which a measurement captures what it is intended to measure
  • reliability is a necessary but not sufficient condition for validity…
  • a test may be reliable because it consistently reports the same measurement
  • however it may not be valid because the measurement is incorrect
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Types of Measurement Valididty

A
  • face validity
  • content validity
  • construct validity: convergent, discriminant
  • criterion related validity: concurrent validity, predictive validity
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Face Validity

A
  • does the test or instrument appear, on the face of it, to assess what is intended
  • addressed from the standpoint of the tester and from the standpoint of the patient or family member
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Content Validity

A
  • extent to which an instrument reflects all the meaningful elements of a variable
  • judged by content experts or people with experience with the variable
  • usually only pertinent to multidimensional measurements
  • disability measures, functional measures, self-reported tools, knowledge assessment
  • look at example on page 19 of 9/18 notes
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Construct Validity

A
  • degree to which a measure reflects the operational definition of the concept it is said to represent
  • achieved via operational definitions, logical arguments, theoretical arguments, and research evidence
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Forms of Construct Validity-Convergent Validity

A
  • comparison of scores between two similar instruments expected to produce similar results
  • positively correlate with each other
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Forms of Construct Validity-Discriminant Validity

A
  • differentiation among different levels of characteristics of interest
  • degree of disability
  • does an instrument or test differentiate between individuals with shoulder impingement and those without
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Criterion Validity

A
  • extent to which one measure is systematically related to other measures or outcomes
  • required direct comparison of the index measure with a standard (criterion) measure
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Forms of Criterion Validity-Concurrent Validity

A
  • ability of an index measure to capture an outcome similar to that of another measure
  • compare the index measure to the criterion measure, that was obtained at the same time
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Forms of Criterion Validity-Predictive Validity

A
  • the ability of an index measure to predict a figure outcome
  • compare the index measure to the criterion measure that was obtained at a later point in time
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Responsiveness to Change

A
  • ability to a measure (instrument) to detect change in the phenomenon of interest
  • depends upon…
  • the fit between the instrument and the operational definition of variable=construct validity
  • the number of values on the measurement scale: more values on the scale, the greater the opportunity to detect change
  • standard error of measurement (SEM): extent to which observed scores are disbursed around the true score
17
Q

Floor and Ceiling Effects

A
  • floor effect: failure of a measure to detect lower scores for patients whose status has declined
  • ceiling effect: failure of a measure to detect higher scores for patients whose status has improved
18
Q

Threats to Research Validity

A
  • subjects: selection, assignment, attrition, maturation, compensatory rivalry/resentful demoralization, diffusion or imitation of treatment, statistical regression to the mean
  • investigators: compensatory equalization of treatments
  • study logistics: history, instrumentation, testing
19
Q

Threat-Selection

A
  • problem: selection process leads to sample that is not representative of the population from which it is drawn
  • potential solutions: multiple study sites, probabalistic selection
20
Q

Threat-Assignment

A
  • problem: group assignment process leads to unequal distribution of subject characteristics
  • solution: probabilistic assignment
21
Q

Threat-Maturation

A
  • problem: changes over time that are internal to the subjects and may influence the outcome
  • solutions: control or comparison group, repeated baseline measures, scheduling
22
Q

Threat-Compensatory Rivalry/Resentful Demoralization

A
  • problem: subjects change behavior in response to learning they are in control group
  • solutions: masking, instructions re: adherence to protocol, separation of subjects
23
Q

Threat-Diffusion or Imitation of Treatment

A
  • problem: contact among subjects from different groups

- solutions: masking, instructions re: adherence to protocol, separation of subjects

24
Q

Threat-Statistical Regression to the Mean

A
  • problem: appearance of change due to an extreme score for the baseline measure on the outcome of interest
  • solutions: trim outliers, average repeated baseline measures
25
Q

Threat-Compensatory Equalization

A
  • problem: purposeful or inadvertent supplementation of the control or comparison group
  • solutions: masking, protocols for intervention administration, different locations
26
Q

Threat-History

A
  • problem: events occurring unrelated to the study that are out of the investigators’ control and may influence outcome
  • solutions: control comparison group, scheduling
27
Q

Threat-Instrumentation

A
  • problem: wrong measurement approach or device, limitation of measure, malfunction, inaccurate application
  • solutions: operational definitions, selection of most rigorous instrument, calibration, protocols, training and verification
28
Q

Threat-Testing

A
  • problem: appearance of improvement due to familiarity with test procedures or in response to different instructions/cues
  • solutions: practice sessions, repeated measures, protocols for test administration, training and verification
29
Q

Investigator Bias

A
  • purposeful or inadvertent interference with the study’s procedures…
  • selection
  • assignment
  • instrumentation
30
Q

Statistical Adjustment

A
  • method for controlling extraneous variables during statistical analysis
  • used when design features cannot or do not successfully control confounding influences
31
Q

Threats to Construct Validity

A
  • construct under-representation
  • experimenter expectancies
  • interaction between different treatments
32
Q

Study Relevance

A
  • external validity (quantitative)/transferability (qualitative)
  • extent to which results of the research study can be generalized: across groups, settings, times vs. to particular persons, settings, times
33
Q

Threats to Study Relevance

A
  • inadequate sample selection
  • differences in settings
  • differences in circumstances due to passage of time