Ch 5 Improving & Assessing the Quality of Behavior Measurement Flashcards
extent to which data actually measure what they are intended to measure- how accurately the measurement reflects the true behavior of interest in the real-world context
-ensures accurate, meaningful data
-“are we measuring the right thing?”
~key components~
1.direct measurement=measuring actual bx of interest
2.relevance to goals=measurement must reflect what matters to the client or stakeholders
3.accurate representation=bx is measured under relevant conditions, in the right context, with representative samples
~threats~
indirect measurement (measuring a proxy instead of bx itself), poor operational definition (ambiguous or vague), unrepresentative sampling (measuring in a way that doesn’t reflect typical performance)
validity
extent to which the observed value (what was recorded) matches true value of behavior being measured. tells us how correct the data are in representing what actually happened.
-degree to which measurement corresponds to the true value of the event it is intended to represent
-“are we measuring the true value?”
e.g. student flaps hands 5x, therapist records 5 instances of hand flaps
accuracy
behavior is measured in a consistent and repeatable way, across time, observers, and conditions. essential for trustworthy data.
-high __= low variablity in measurement across observers
-“are we measuring the same way every time?”
e.g. two rbts record data on a client’s tantrums. they both record 4 tantrums during the same session, so the data are reliable (even if true number was off, they agree)
reliability
gradual change in how an observer records behavior, leading to inconsistent data. threat to reliable and valid measurement.
observer drift
occurs when observer’s awareness of being monitored/evaluated affects how they record data. aka observer changes their measurement behavior because they know someone is watching
-can threaten accuracy and reliability of data
observer reactivity
measured result recorded by an observer during data collection
e.g. child hits 5x, observer records 5 instances.
child hits 6x, observer records 3 instances, but it’s inaccurate in this case
observed value
someone who is unaware of the purpose of the study, the specific hypotheses being tested, or the expected outcomes of the intervention. this lack of knowledge helps ensure the risk of observer bias or observer reactivity.
-helps reduce expectancy effects (e.g. seeing what they expect to see)
-increases objectivity of measurement
-strengthens internal validity of the study
naive observer
form of error in which observer consistently overestimates or underestimates occurrence of a behavior. occurs when an observer’s expectations, preferences, or other influences lead to inaccurate but predictable error in recording behavior.
measurement bias
actual amount or extent of the behavior that occurred
-different from observed value
true value
collecting data about bx without directly observing it; information is gathered through interviews, questionnaires, rating scales, or self-reports. used when direct observation isn’t possible; provides less valid and less reliable data
indirect measurement
observing behavior during selected time periods (not entire session) and recording whether the behavior occurred during those intervals or moments
-some instances of bx may be missed because observation occurs only during specific intervals or moments in time
-provides estimates of bx rather than a complete count.
discontinuous measurement
measuring the actual behavior of interest, NOT something else that’s related to it
e.g. measuring hand raising when that is the target behavior (vs measuring general participation)
direct measurement
observing and recording every instance of a behavior during a session
-provides most accurate and complete picture of bx because no occurrences are intentionally missed
-types- frequency, rate, duration, latency, IRT
continuous measurement
process of ensuring all observers or measurement systems are consistently using the same definitions and procedure when recording bx. goal is to align everyone’s understanding of what’s being measured so data are valid, accurate, and reliable
calibration
degree to which data are considered trustworthy because they are consistently recorded across observers or time
-specifically, whether data are consistent with how the bx actually occurred
-assessed through IOA, reliable measurement procedures, clear operational definitions, calibration of observers
-confirms bx change is real
-ensures data is accurate and replicable
believability
degree of consistency between observers’ data collection; essential to ensure reliable, accurate, believable data
IOA (interobserver agreement)
used to assess consistency of data recording between observers during DTT or other procedures that incolve clearly defined, separate response opportunities
of trials with agreement/total # of trials X 100
trial-by-trial IOA
compares duration recorded by observers for each instance of a behavior; used when each instance of bx has clear start and end
IOA per occurrence= shorter duration/longer duration X 100
_____= (sum of all IOA percentages)/(# of bx occurrences)
mean duration-per-occurrence IOA
evaluates agreement in non-occurence of behavior. calculates IOA for interval recording; focuses only on intervals in which at least one observer recorded bx as not occurring.
-best when target bx occurs in low frequency
(#of intervals both observers scored as non-occurrence)/(#of intervals where at least one observer scored nonoccurrence) X 100
unscored IOA
average percentage of agreement between observers on the count of bx occurrences in each interval
for each interval:
smaller count/larger count X 100
then:
sum of all interval IOAs/# of intervals
mean count-per-interval IOA
calculates IOA when using interval recording; compares observer agreement on occurrence or non-occurence of a behavior during each interval
of agreements/total # of intervals X 100s
interval-by-interval IOA
stringent (strict) method of assessing IOA when measuring event-based behaviors (like frequency) across equal time intervals. % of intervals where observers recorded exact same count of bx
total # of intervals with exact agreement/ total # intervals X 100
exact count-per-interval IOA
simplest method of IOA when bx is measured using event recording. compares total # of responses between observers over an entire observation period, regardless of when they occurred
smaller count/larger count X 100
e.g.
observer A records 18 instances, observer B records 20 instances.
(18/20) X 100= 90%
total count IOA
calculates IOA for interval recording, focuses onlt on intervals where at least one observer recorded bx as occurring. best for infrequent bxs.
intervals both observers scored bx/ # intervals where at least one observer scored bx X 100
scored-interval IOA
