4 Cleaning and Processing Data

Question 1

What is the primary issue with duplicate data in a dataset?

Answer 1

Duplicate data can cause issues with skew, bias, or completely invalidate your analysis.

Question 2

Define duplicate data.

Answer 2

Duplicate data is when a specific data point recurs multiple times within a dataset.

Question 3

What is the impact of duplicate data on descriptive statistics?

Answer 3

It can distort averages and percentages, leading to incorrect conclusions about the dataset.

Question 4

What is redundant data?

Answer 4

Redundant data refers to columns that can be used to perfectly predict other columns.

Question 5

How does redundant data differ from duplicate data?

Answer 5

Duplicate data is a copy of a row, whereas redundant data is a copy of a column.

Question 6

What is multicollinearity?

Answer 6

Multicollinearity occurs when multiple independent variables in a model are highly correlated.

Question 7

What is a common approach to handle duplicate data?

Answer 7

The most common approach is to delete all duplicate rows.

Question 8

What are some potential issues with having redundant data in a statistical model?

Answer 8

It can make results harder to interpret and can lead to inaccurate models when applied to the population.

Question 9

What is missing data?

Answer 9

Missing data refers to gaps in a dataset where no information is available for certain entries.

Question 10

Why is missing data problematic for data analysts?

Answer 10

Most analyses won’t run with null values, leading to errors and reduced statistical power.

Question 11

What are the three main categories of missing data?

Answer 11

• Missing Completely at Random (MCAR) * Missing at Random (MAR) * Missing Not at Random (MNAR)

Question 12

What does Missing Completely at Random (MCAR) mean?

Answer 12

Data is MCAR when there is no connection between the missing values and the present values.

Question 13

What does Missing at Random (MAR) imply?

Answer 13

MAR means the missing data is related to another recorded variable.

Question 14

Describe Missing Not at Random (MNAR).

Answer 14

MNAR occurs when the missing data is related to some unrecorded variable or factor.

Question 15

What is a recommended practice when working with datasets?

Answer 15

It is generally good practice to work on a copy of your data instead of the original.

Question 16

What can happen if too much redundant data is included in a dataset?

Answer 16

It can lead to multicollinearity, complicating the interpretation of statistical models.

Question 17

Fill in the blank: Redundant data can lead to _______ in statistical models.

Answer 17

multicollinearity

Question 18

True or False: All methods for dealing with missing data are universally accepted.

Answer 18

False

Question 19

How can one create a subset of data excluding redundant columns?

Answer 19

By using functions like drop() to exclude the redundant variables.

Question 20

What is the consequence of having missing data that is not random?

Answer 20

It can introduce bias into the results.

Question 21

What is the main reason for identifying the type of missing data?

Answer 21

It helps determine how much the missing data will influence the outcome and potential bias.

Question 22

What does MNAR stand for?

Answer 22

Missing Not At Random.

Question 23

What is a key characteristic of MNAR data?

Answer 23

It has a connection to some variable or type of information that was not recorded.

Question 24

Why is MNAR data considered problematic?

Answer 24

It is the most likely to cause bias in results.

Question 25

What is the easiest approach to handle missing data?

Answer 25

Deletion.

Question 26

What is a critical note to remember when working with deletion methods?

Answer 26

Always work on a copy of your data.

Question 27

What is listwise deletion?

Answer 27

Deleting an entire observation if a single value is missing.

Question 28

What is pairwise deletion?

Answer 28

Deleting specific missing values while retaining the rest of the data in the row.

Question 29

When is variable deletion appropriate?

Answer 29

When over half of the values for a specific variable are missing.

Question 30

What is filtering in the context of missing data?

Answer 30

Removing values to create a subset of data that has no missing data.

Question 31

What is imputation?

Answer 31

Filling in missing data instead of removing it.

Question 32

What does mean, median, or mode imputation involve?

Answer 32

Estimating the middle of a dataset to fill in gaps.

Question 33

What is hot deck imputation?

Answer 33

Using random values from elsewhere in the dataset to fill in missing values.

Question 34

What is interpolation?

Answer 34

Estimating specific values for missing data points using other values as reference.

Question 35

Why is MNAR data difficult to handle with common methods?

Answer 35

There is an unknown reason for the missing data.

Question 36

What is invalid data?

Answer 36

Data that does not match expected values or ranges.

Question 37

What causes specification mismatch?

Answer 37

A value having a different data type than the other values in a variable.

Question 38

What is data type validation?

Answer 38

Checking the data type of a variable to avoid specification mismatches.

Question 39

What is non-parametric data?

Answer 39

Data that does not follow a normal or well-known distribution.

Question 40

What is an example of a common distribution in statistics?

Answer 40

Normal distribution.

Question 41

What can happen if a single value in a variable has the wrong data type?

Answer 41

It can cause errors in data analysis.

Question 42

What should you do if you identify invalid data?

Answer 42

Generate a list of unique values and check for discrepancies.

Question 43

Fill in the blank: Listwise deletion is also known as _______.

Answer 43

casewise deletion.

Question 44

True or False: Pairwise deletion is less likely to introduce bias compared to listwise deletion.

Answer 44

True.

Question 45

What do you risk by deleting data that is not MCAR?

Answer 45

Introducing bias.

Question 46

What is a common method to reduce invalid data during data entry?

Answer 46

Using a drop-down menu.

Question 47

What is a normal distribution?

Answer 47

A distribution used to predict the probability that a new value will be any specific number.

Question 48

What characterizes non-parametric data?

Answer 48

Non-parametric data does not follow any of the common distributions.

Question 49

Why can non-parametric data be problematic?

Answer 49

Majority of common statistical analyses are inherently parametric and assume that data is in a specific distribution.

Question 50

What are distribution-free tests?

Answer 50

Statistical analyses that do not assume any specific distribution.

Question 51

What are outliers?

Answer 51

Data points that are significantly larger or smaller than the rest and can skew the entire dataset.

Question 52

What issue can a single outlier cause?

Answer 52

It can artificially pull all of the results up and to the right.

Question 53

How are outliers typically identified?

Answer 53

By calculating the standard deviation or the interquartile range (IQR).

Question 54

What is the common cutoff for identifying outliers using standard deviation?

Answer 54

More than three standard deviations away from the mean.

Question 55

What is the common cutoff for identifying outliers using IQR?

Answer 55

1.5 times the interquartile range (IQR).

Question 56

What is the pragmatic approach to handling outliers in data analytics?

Answer 56

Creating a range; anything outside of that range is considered an outlier and is deleted.

Question 57

What types of data issues are addressed in data cleaning?

Answer 57

* Duplicate data * Redundant data * Missing data * Invalid data * Specification mismatch * Data type validation

Question 58

What is the importance of cleaning data?

Answer 58

It removes elements that will cause errors in analysis.

Question 59

What is the next chapter about after cleaning data?

Answer 59

Data wrangling and manipulation.

Question 60

Fill in the blank: Non-parametric data is a problem because you can’t use _______ analyses on it.

Answer 60

parametric

Question 61

True or False: All statistical analyses are suitable for non-parametric data.

Answer 61

False

Question 62

When identifying outliers, what should you avoid doing?

Answer 62

Eyeballing it and guessing.

Question 63

What is the typical method for deleting missing data?

Answer 63

* Listwise * Pairwise * Variable

Question 64

What error type is indicated by a mismatch in department names, like 'Sales' and 'Sale'?

Answer 64

Invalid data

Question 65

If you find a value of 8,000 lb in a dataset of human baby weights, what should you consider?

Answer 65

It is probably an outlier, and you should check your ranges to be sure.