CPXP Exam Module 2 - Measurement & Analysis Flashcards
Measurement & Analysis
1
Q
What are the components of an effective measurement system?
A
Observation, question, hypothesis, test/experiment, analysis, conclusions
2
Q
What is a hypothesis?
A
A stated prediction about a specified outcome
3
Q
What are some sources of bias in measurement and analysis?
A
Cultural biases, erroneous assumptions, ego-based, inaccurate observations
4
Q
What are the two basic data types?
A
Quantitative (numbers) and qualitative (words)
5
Q
What are the advantages of quantitative data?
A
Structured, test hypotheses or assumptions, answers “what” and “how”, high cerebral impact
6
Q
What are the advantages of qualitative data?
A
Description, dynamic, observations and interviews, themes, answers “why” and “how” questions, high emotional impact
7
Q
What is mixed-methods?
A
The use of a combination of qualitative and quantitative measurement and analysis methods
8
Q
What are some of the ethical challenges in measurement?
A
Bias, under-reporting, over-reporting, using unreliable or invalid measures or methods, and not using the data for assessment or improvement
9
Q
What is your biggest ethical responsibility in using measures?
A
Protection of patients and families
10
Q
Define “validity” in practical terms- what makes a measure “valid” or “validated”?
A
The measure accurately measures the phenomenon it is supposed to measure
11
Q
What is “content validity”?
A
How well a measurement instrument (or test) covers all relevant parts of the construct it aims to measure
12
Q
What is “criterion validity?”
A
How well a test or instrument measures a phenomenon compared to an established standard of comparison vs. an established “gold standard” measure (a measure believed to be the best available).
13
Q
What is “construct validity?”
A
The degree to which a test is able to measure a construct
14
Q
What is a construct?
A
An abstract concept that is not directly observable, e.g. depression, intelligence, etc.
15
Q
What are the three main types of validity?
A
Content, Criterion, and Construct
16
Q
What is “reliability”?
A
The ability of a measure (or test) to consistently produce similar results under consistent conditions
17
Q
What are the four types of reliability?
A
1) Test-retest
2) parallel forms
3) internal consistency
4) inter-rater
18
Q
What is test-retest reliability?
A
The ability of a test or measure to yield the same results over time when measuring a consistent or fixed phenomenon
19
Q
What is parallel forms reliability?
A
Different forms of the same test get similar results when measuring the same phenomenon
20
Q
What is internal consistency reliability?
A
The individual items or components of a measure (e.g. on a survey or questionnaire) are significantly associated or highly related to one other (e.g. are correlated)
21
Q
What is inter-rater reliability?
A
Different people administering the same test will get similar results when measuring the same phenomenon
22
Q
What are the four types of quantitative data?
A
Nominal, ordinal, interval, and ratio
23
Q
What is a nominal measure?
A
Categorical or dichotomous and non-sequential, e.g. yes/no, alive/dead, gender, etc.
24
Q
What is an ordinal measure?
A
Categorical, sequential, and not scaled, e.g. date ranges, age groups
25
What is an interval measure?
Numerical, sequential, and scaled but not at equal intervals or ratios, e.g. a Likert Scale (1= disagree, 2= somewhat disagree, 3= neutral, 4= agree, 5= strongly agree)
26
What is a ratio measure?
A ratio measure is one that is numerical, sequential, scaled, and at equal intervals that allows ratio based comparisons of any size of increment, e.g. fully continuous data, e.g. temperature
27
What is a population?
A group of people within a certain demographic or set of demographics
28
What is a sample?
A selected subset of a given population
29
What is sampling?
The act of selecting and/or recruiting a sample from a given population
30
What are some important things to watch out for when sampling?
Sampling bias, time, feasibility, representativeness, diversity, inclusion, culture, language, literacy
31
What does capital "N" stand for when describing sample size?
N refers to the TOTAL overall number of participants and data points in a population- you need N to be large enough to sample adequately
32
What does a lower case "n" stand for in describing a sample size?
“n” refers to the size of a selected sample of a population or a subsample of a sample - a sample should be big enough to draw actionable conclusions from the data and analyses and avoid Type II error
33
What n size is considered an absolute minimum for most samples?
n = 30
34
What is Type II error?
The inability of a measure or test to detect a significant change when a significant change actually exists. This can happen when n size is too small, e.g. false negative.
35
What is Type I error?
An error of a measure in which the measure states that a significant change has occurred when in fact no significant change exists, e.g. "false positive"
36
What is a significant result?
An observed change that has a very low probability of occurring due to chance or random variation alone, usually >2 SD from the mean (less than 5% chance probability)
37
Descriptive Statistics
Statistics used to describe a population or sample including measures of central tendency and do not assess inferences or variation
38
What are the three measures of central tendency?
Mean (average), median, and mode
39
Define "mean" (average) and describe how it is calculated
The mean is the arithmetic average of all values in a sample. A mean is calculated by summing all of the values in the sample and dividing this by the number of observations (n) in the sample
40
Define median and describe how to determine it
The median is a point in a distribution of values in a sample at which 50% of the remaining values fall on either side. To determine the median, order all the values in sequence and find the middle most point in the sequence.
41
Define mode and describe how to determine it
The mode is the most frequently occurring value in a distribution or sample. To determine the mode, you can count the frequencies of each data value and select the highest frequency value.
42
When is the mean the preferred measure of central tendency?
The mean is the preferred measure of central tendency when the distribution of values in a sample is normally or near-normally distributed
43
When is the median the preferred measure of central tendency?
The median is preferred when the distribution of values in a sample is skewed or non-normal, or when the n size is low resulting in the ability of single outlier values to skew the mean
44
What is a normal distribution?
A group of values in a sample which has a "bell curve" appearance when arranged around the mean
45
What is a skewed distribution?
A group of values that does not have a bell curve appearance when arranged around a given mean, but rather shifts to the left or the right of the mean
46
What are the most common measures of variation used in inferential statistics?
Range
Standard Deviation (SD)
Variance
47
Define "range" and describe how to determine it
The distance between the highest and lowest values in a data distribution or sample. It is calculated as the difference between the highest and lowest values.
48
Define "standard deviation" and describe how it is calculated
A standard measure of the degree of spread, i.e. how far values are from the mean (average) and is often depicted visually on a distribution data curve. This can be calculated on Excel using the STDEV.S(data cell range) command.
49
Define "variance" and describe how it is calculated
In statistics, variance measures variability from the average or mean. It is calculated by taking the differences between each number in the data set and the mean, then squaring the differences to make them positive, and finally dividing the sum of the squares by the number of values in the data set
50
What is a histogram?
A picture of a data distribution based on data distribution frequencies, usually shown as a bar graph or a curve - a normal distribution has a "bell curve" appearance
51
What are the three types of statistics?
Descriptive, inferential, and improvement
52
What can you use inferential statistics for?
To assess associations (correlations), test hypotheses (e.g. t-tests, ANOVA), and predict outcomes (e.g. regression), i.e. make inferences
53
What can you use improvement statistics for?
Improvement statistics (Run Charts, Statistical Process Control) can be used to assess performance variation and inform strategic planning, improvement, and implementation efforts, i.e. to assess if changes result in improvements and that observed changes are sustained
54
Define "statistic"
A measure which describes a characteristic of a given data sample
55
Define "parameter"
A measure which describes a characteristic of a population
56
What is a "point estimate"?
A single value estimate of a parameter, e.g. a sample mean is a point estimate of a population mean.
57
What is an "interval estimate"?
Gives a range of values within which a parameter is expected to reside, e.g. a confidence interval (used in regression), and control limits (used in Statistical Process Control)
58
What is a "sampling error"?
A difference between a parameter and its corresponding statistic, i.e. the degree to which the measured statistic is "off target" compared to the actual parameter. This can be produced by a failure to include data that is fully representative of the population being measured. This is a type of systematic error.
59
What is a confidence interval?
A type of interval estimate which states the amount of uncertainty (measurement precision) associated with the statistics of a data sample, and generates a range within which data values may reliably measure a true value of a population parameter, includes consideration of error.
60
How are confidence intervals usually stated?
Usually stated as a point estimate followed by a 95% confidence range (95% CI) representing +/- 2 SDs variation around the point estimate (capturing 95% of the total data distribution), e.g. 5.6 (95% CI: 2.2, 8.5).
61
What is "hypothesis testing?"
The use of inferential statistical tests to assess a stated hypothesis. Hypotheses testing can compare populations (e.g. A vs. B) or exposures (e.g. intervention vs. control). This is most effective in parametric samples.
62
What are the assumptions of a parametric test?
The population or sample is normally (or nearly normally) distributed, the sample n is large enough to detect significance differences (has acceptable Type I and Type II error properties), and the variances of the groups being compared are similar.
63
Define "comparison test" in inferential statistics
Tests which compare things, e.g. differences in mean, rankings, etc. For example, a t-test assesses the differences in the means of two groups or samples. Requires interval or ratio level data.
64
What is the function of "correlation" analyses in inferential statistics?
Assesses the degree of association between two or more variables, i.e. the degree to which they are related.
65
What is the function of regression analysis in inferential statistics?
Used to assess the degree to which a set of variables (independent variables and covariates) can predict (cause) an outcome variable (dependent variable)
66
What are the necessary factors required to establish causation?
Association, directionality, and lack of confounding
67
What is confounding?
The effect of one or more unknown or unmeasured variables on an outcome, i.e. smoking could be a confounding variable in the relationship between age and mortality, etc.
68
What are some common inferential tests used for parametric comparisons?
t Test (compares 2 means), and Analysis of Variance (ANOVA) which compares 3 or more means. Requires interval or ratio level data
69
What are some common inferential tests used for nonparametric comparisons?
Mood's median test (compares 2 or more medians), Wilcoxon signed-rank test (compares 2 samples/distributions), Wilcoxon rank-sum test (compares sums of rankings for 2 samples), and Kruskal-Wallis test (compares mean rankings of 3 or more samples)
70
What correlation test is used for parametric samples?
Pearson r is used for parametric correlational analyses. Requires interval/ratio level data
71
What correlational tests are used for nonparametric samples?
The Spearman r correlation can be used for ordinal level data or continuous variables.
The Chi Square test can be used for nominal or ordinal level data
72
How are correlations stated and interpreted?
Correlations range from -1 to +1. A positive correlation indicates that variables move in the same direction (positive association). A negative correlation indicates that the variables move in opposite directions (negative association). "0" means no association.
73
What types of regression analyses can be used for interval and ratio data?
Simple linear regression can be used for 1 predictor (independent) variable predicting an outcome (dependent variable).
Multiple linear regression can be used for 2 or more predictors.
Generalized linear modeling (GLM) can also be used for 2 or more predictors.
74
What is an independent variable?
A variable in a regression analysis that is considered to be a "predictor" of an outcome
75
What is a dependent variable?
A variable in a regression analysis that is considered to be the outcome predicted by the independent variable(s)
76
What types of regression analysis can be used for nominal or ordinal data?
Logistic regression can be used for binary data (e.g. "alive/dead") for one or more predictors.
Nominal regression can be used for non-binary nominal level data (1 or more predictors).
Ordinal regression can be used for ordinal data (1 or more predictors)
77
What is a scatter-plot?
A visual data display which shows a distribution of point associations between two variables. A regression line can be added to this showing the direction and degree of association between the variables.
78
What are some commonly used data sources used in PX and CX?
Surveys (HCAHPS, engagement, etc.), rounding, focus groups, complaints/grievances, compliments, PFACs
79
What is HCAHPS?
Hospital Consumer Assessment of Healthcare Providers and Systems. Assesses global experience of inpatient care.
80
What is CAHPS?
Consumer Assessment of Healthcare Providers and Systems. Includes HCAHPS for inpatient care and also includes assessments of ambulatory and hospice care.
81
What is Value-Based Purchasing (VBP)?
The use of total performance measurements to determine payment and reimbursement amounts that CMS will pay health systems. 25% of total performance is determined by Person and Community Engagement
82
What do HCAHPS and CAHPS Patient Experience Surveys measure?
Patient experience, patient satisfaction, communication (doctors, nurses), understanding medication and treatment instructions, care coordination and transitions, environment (clean & quiet), and discharge planning.
83
What are the goals of using HCAHPS and CAHPS surveys?
Inform improvement and strategic planning, create incentives to optimize patient experience performance, increase accountability and transparency for PX performance
84
What are CMS Star Ratings?
1-5 Stars reflecting overall inpatient care quality performance for a health facility or system. Star ratings can be reported overall and by category (including for PX), by site or health system.
85
Which individual HCAHPS items are used for Star Ratings?
Cleanliness and Quiet (environment)
86
What is a HCAHPS "Global" Item?
Reflects an overall experience measure, e.g. "Overall Rating of Hospital," or "Likelihood to Recommend Hospital," or "Likelihood to Recommend Provider," or "Likelihood to Recommend Practice"
87
What is a HCAHPS Composite Measure?
Reflects the aggregated results of a number of individual items grouped in category, these include communication with doctors, communication with nurses, responsiveness, communication about meds, discharge information, and care transitions.
88
What are the "three don'ts" related to HCAHPS and CAHPS surveys?
1) Don't encourage patients to answer surveys in a certain way
2) Don't incentivize patients to answer surveys or answer in a certain way.
3) Don't replicate the survey questions in other data collection approaches (e.g. rounding).
89
What are three types of error affecting HCAHPS and CAHPS survey data?
Sampling error, systematic error, and random error
90
What is systematic error in survey data collection?
An error inherently designed into the study or activity. A sampling error is an example of a systematic error.
91
What is a random error in survey data collection.
An error experienced due to an unknown, unexpected, or uncontrollable occurrence, e.g. unexpected postal service error leads to a patient not receiving a survey.
92
What is a trend?
A direction that a cluster of points takes (up or down) longitudinally over time.
93
Define "longitudinal analysis"
A data display or analysis conducted "over time," requires 2 or more time points
94
What is a "cross-sectional" analysis?
A data analysis conducted at a single time point or in a designated time frame, e.g. in 2023. Benchmarking analyses often use cross-sectional analyses.
95
What is "benchmarking"?
A data analysis comparing performance of two or more groups to each other and the overall average performance. This can be used for accountability and improvement purposes.
96
What are the three different ways that "trends" can be defined?
A simple trend, an inferential trend, and a SPC trend
97
What is a "simple trend"?
In a descriptive statistical analysis, 3 or more points moving in the same direction. Note that this type of trend is not statistically determined (it is descriptive)
98
How is a trend defined in inferential statistics?
This is used in inferential statistics to describe the slope of a regression line plotted through a distribution of 3 or more data points. The slope can be positive (up) or negative (down) and the magnitude of the slope can range from -1 to +1. A slope of "0" indicates no trend.
99
How is a trend defined in Run Charts?
Per Institute for Healthcare Improvement (IHI) detection rules, a trend is defined for Run Charts as five or more consecutively increasing or decreasing points, regardless of slope.
100
How is a trend defined in Statistical Process Control (SPC)
Per Institute for Healthcare Improvement (IHI) detection rules, an SPC trend is defined as six or more consecutively increasing or decreasing points, regardless of scope.
101
What is a Run Chart?
A time plot + a measure of central tendency (usually the median)
102
What is common cause variation in improvement measurement and analysis?
As used in Run Charts and Statistical Process Control (SPC), common cause variation refers to chance or random variation ("due to chance")
103
What is special cause variation in improvement measurement and analysis?
As used in Run Charts and Statistical Process Control (SPC), special cause variation refers to non-random variation ("not due to chance"), and is often statistically significant
104
What is a Statistical Process Control (SPC) chart?
A visual display consisting of a time plot + a measure of central tendency (the mean) + upper and lower control limits which are approximately +/-1 three sigma deviations (approximately +/- 2.67 standard deviations) from the mean.
105
What is a time plot?
A simple longitudinal data display (graph) showing observations (y axis) of a variable over time (x axis)
106
What are the Special Cause detection signals for Run Charts?
Per Institute for Healthcare Improvement (IHI) detection rules:
(1) Trend (5 or more consecutively increasing or decreasing points)
(2) Shift (6 or more consecutive points above or below the measure of central tendency)
(3) Astronomical points (subjectively determined to be "way off")
107
What is an outlier?
A data point that is substantially (or statistically significantly) separated from the remainder of the data distribution. In small samples, outliers can skew the mean (use median instead).
108
What are the Special Cause signals for Statistical Process Control (SPC)?
There are many. However the basic three rules (IHI criteria) are:
(1) Trend (6 or more consecutively increasing or decreasing points)
(2) Shift (8 or more points consecutively above or below the center line)
(3) Points outside of the the 3 Sigma control limits.
109
How many data points are required for appropriate statistical power for Run Charts and SPC charts?
A minimum of 12. Fifteen to 20 points are preferred.
110
What is percentile rank and how is it determined?
Percentile rank is a statistical measure used to categorize or stratify where an individual value falls within a rank ordered specified group or population. Common stratifications used in percentile ranking include deciles (10% groupings) or quartiles (25% groupings) ranked from lowest to highest.
111
What are percentile ranks used for?
Often used for relative performance benchmarking to inform improvement and strategic planning.
112
What is "relative benchmarking"?
Benchmarking based on the overall performance of a specified group (which can vary over time) rather than against a fixed constant value.
113
What is a "key driver"?
A factor that is known (or believed) based on evidence to be causally related to a specified outcome
114
What is a "key driver diagram"?
A visual display linking an outcome (goal) to the key drivers which influence that outcome. Drivers can be primary (major drivers) or secondary drivers (smaller component drivers that influence primary drivers)
115
Which key driver approaches are commonly used with HCAHPS and CAHPS data in PX?
Driver index scores are used to weight the importance of known drivers and key drivers are identified using percentile ranks of top-box scores
116
What is a top box score?
In a distribution of responses to an interval or ratio level scale, the top 10% (top decile of those scores), e.g. on a 10 point scale, it would be all of the "9" and "10" scores. Top box scores are typically displayed as a proportion = (count of top box responses / count of total responses)
117
What is priority indexing?
A method of ordering survey responses which uses the inverse rank of correlation + performance rank. It is an alternative approach to "ordering" strategies (such as "top box" scores).
118
What are some common visualization tools used in improvement activities?
Dashboards, process maps, affinity diagrams, graphs, fishbone diagrams (Ishikawa diagrams), key driver diagrams
119
What are some data informed improvement actions that can be used?
Service recovery and recognition (immediate), process improvement (intermediate), strategic planning and systems redesign (long-term)
