15 Data Quality and Management Flashcards
What is quality control?
The process of testing data to ensure data integrity
Quality control is essential because bad data can lead to misleading results.
When should you check for data quality?
Any time there is a major change, such as:
* Data acquisition
* Data transformation
* Data manipulation
* Final product review
Regular checks are also important beyond routine maintenance.
What is data acquisition?
The process of obtaining new data
It requires checking for bias and the current state of the data.
What does data transformation involve?
Changing data from one form to another, including:
* Intrahops
* Pass-throughs
* Conversions
Transformations should ideally be done in new variables.
What is data manipulation?
Changing the shape of the data without altering its content
Examples include breaking down or combining variables.
What are the data quality dimensions?
Key dimensions include:
* Data consistency
* Data accuracy
* Data completeness
* Data integrity
* Data attribute limitations
These dimensions help assess the quality of data.
What is data consistency?
Ensuring data is uniform and reported the same way across different levels
This applies to both individual variables and broader databases.
What is data accuracy?
Whether the data is correct
Checking data accuracy often involves verifying it against an outside source.
What is data completeness?
Checking for gaps in data, such as missing values or entire variables
This is essential for valid analyses.
What does data integrity encompass?
It includes consistency, accuracy, completeness, and security
Data integrity is crucial in regulated fields like pharmaceuticals.
What are data quality rules and metrics?
Guidelines that define acceptable data standards and formats
These include cutoff scores and conformity rules.
What is cross-validation?
A statistical analysis that checks if results can be generalized
It helps assess model effectiveness and reduce test error.
What are sample/spot checks?
Quick checks focusing on one or two data quality dimensions
They are often prompted by unusual data observations.
What are reasonable expectations in data quality?
Assessing whether data values make sense based on historical norms
This can involve formalized processes for flagging outliers.
What is data profiling?
A formal process that checks data quality across entire databases
It usually includes structure, content, and relationship discovery.
What is a data audit?
A systematic check to see if a dataset meets specific goals
Audits are often scheduled and performed at all stages of the data lifecycle.
What is master data management (MDM)?
The process of creating and managing a centralized data system
MDM aims to create a ‘golden record’ for improved data quality.
When should MDM be used?
During:
* Mergers and acquisitions
* Compliance checks
* Streamlining data access
MDM helps integrate disparate data sources and manage protected data.
What is the benefit of having a golden record?
It provides a single source of truth with clean, standardized data
This facilitates faster access and higher data quality.
What challenges are associated with implementing MDM?
It can be labor-intensive and expensive to set up
Many companies may only implement MDM for specific data types.
What is policy in the context of data management?
Policy is in reference to compliance, ensuring all records are organized for easier regulation checks.
What does streamlining data access mean?
Streamlining data access allows faster retrieval of data from a single table without complex queries.
What is the first step in the MDM process?
Consolidation
What does consolidation involve in MDM?
Consolidation involves creating the golden record by combining data from multiple sources into one place.
