Barriers to appraising organisational evidence Flashcards
(6 cards)
Barrier 1: Absence of logical model
a logical model spells out the process by which we expect underlying causes to lead to a problem and produce certain organisational consequences. iit is a grahpic representation of logical connections between inputs activites and processes, outputs, and outcomes. think of a logic model as a short narrative explaining hwy or when a problem occurs and how this leads ot a particular outcome. logical models provide a way of conceptualising problems and processes that allow more information and data points to be factored into our thinking than is possible in unaided human judgement.
Barrier 2: irrelevant data
Organizations often rely on metrics to guide decisions, but many metrics are based on easily accessible data rather than meaningful insights. Flashy, irrelevant data can create a false sense of understanding. Useful metrics focus on outcomes (e.g. sales per week) rather than just activities (e.g. time spent). Collecting too many irrelevant metrics can be distracting—quality matters more than quantity. Always ask whether a metric is truly relevant to your goals and decision-making, and whether it fits into your logic model.
The key takeaway is: more data is not always better. It’s more effective to rely on a smaller number of relevant and accurate metrics than to be overwhelmed by flashy, irrelevant ones. Always ask whether a data point is central to your logic model—if it doesn’t connect to your strategic goals or decisions, it may not be worth collecting.
Barrier 3: inaccurate data
Data vs. Facts
Data may look factual and objective, but in reality, they are often just numbers recorded by people and may involve biases.
Even subjective opinions (like satisfaction ratings) can be turned into numbers, giving a false sense of objectivity.
Collection
This is the first step, where data are gathered manually or automatically.
Objective data (like machine-recorded errors or absentee records) are generally more accurate.
Subjective data (like self-reported incidents or opinions) are prone to biases and memory errors, especially when collected after the fact.
Extraction
Refers to pulling data from its original source.
Direct extraction (e.g. automated systems pulling from a live database) is best.
Manual processes (e.g. copying from spreadsheets) introduce more human error and opportunities for data loss or distortion.
Aggregation
Involves combining data from different departments or systems.
Errors occur when incompatible data types are combined, such as aggregating patient treatments across medical units with very different functions.
This can result in misleading averages or totals.
Conversion and Reformatting
Necessary when integrating data from different sources.
May include splitting, merging, or transforming data into a consistent format.
Each conversion adds a risk of introducing new errors, especially if done manually or without clear rules.
Interpretation
Raw data must often be translated into KPIs or metrics.
This step involves human judgment, such as deciding thresholds (e.g. what counts as “good” vs “poor” performance).
Without standardized rules, this step is highly prone to bias and inconsistency.
Summarization
Final step often involves creating summaries or dashboards for managers.
While helpful for readability, summarization can lead to oversimplification and loss of nuance, especially if key outliers or contextual factors are ignored.
Human Involvement = Risk
The more stages that involve people, the **greater the chances of:
Errors (mistakes in data entry, formatting)
Bias (subjective interpretation)
Manipulation (presenting favorable numbers)
Misunderstanding (not knowing what the data really represent)
Barrier 5: Measurement error
Measurement error occurs when the measured value deviates from the true value. It’s common in organizational data, especially when using difference scores like profit, where small errors in components can compound. Metrics with correlated variables tend to have larger errors, making them less reliable. Thus, managers must be cautious when interpreting derived metrics.
Barrier 6: The small number problem
The “small number problem” occurs when using small samples of organizational data, which are more likely to deviate from the true values compared to larger samples. This problem arises in three situations: comparing smaller and larger units within an organization, collecting data from small samples rather than the entire population, and having access to a limited market sample. Smaller samples tend to yield less accurate results. To address this issue, organizations can take steps such as sampling from the entire population and aggregating internal data to create larger sample sizes.
