Outliers

Question 1

An _ is a data point that differs significantly from other values in a dataset.

Answer 1

Outlier

Question 2

A single data point that significantly deviates from the rest of the dataset.

Answer 2

Point Outlier

Question 3

This type of outlier is an isolated data point that is far away from the main body of the data.

Answer 3

Global outlier

It is often easy to identify and remove.

Question 4

This type of outlier is a data point that is unusual in a specific context but may not be outlier in a different context.

Answer 4

Contextual outlier

It is often more difficult to identify and may require additional information or domain knowledge to determine its significance.

Question 5

What causes outliers?

Answer 5

• variability in data
• measurement errors
• novel phenomena

Question 6

Why is it important to identify and handle outliers?

Answer 6

Outliers can skew results and affect the performance of machine learning models. By identifying and removing or handling outliers effectively, we can prevent them from biasing the model, reducing its performance, and hindering its interpretability.

Question 7

The analysis of outlier data is referred to as _.

Answer 7

Outlier Analysis or Outlier Mining

Question 8

_ plays a crucial role in ensuring the quality and accuracy of machine learning models.

Answer 8

Outlier detection

Question 9

What are some techniques to detect outliers?

Answer 9

• statistical tests

1. Z-Score
2. Interquartile Range (IQR)

• visualization techniques
  1. box plots
  2. histograms
• machine learning algorithms

Question 10

What are some ways to handle outliers?

Answer 10

• removing outliers
• transforming data
• using models that are robust to outliers

Question 11

A _ is relatively unaffected by extreme values, such as the median.

Answer 11

resistant statistic

A statistic is resistant if it is relatively unaffected by extreme values.

Question 12

Which statistic is resistant, the mean or the median?

Answer 12

The median (middle value) is resistant while the mean (average) is not.

df.cgpa.mean()

Example: World Gross (in millions)

With Harry Potter
Mean = $150,742,300
Median = $76,658,500

Without Harry Potter
Mean = $141,889,900
Median = $75,009,000

Question 13

What should you do if an outlier is not a mistake?

Answer 13

Run the analysis twice: once with the outlier and once without, to assess its impact.

Question 14

This measures the spread of the data from the mean.

Answer 14

Standard deviation

df.cgpa.std()

Sample standard deviation: s
Population standard deviation:  (“sigma”)

Question 15

What does a larger standard deviation indicate?

Answer 15

A larger standard deviation indicates more variability and that the data are more spread out.

Question 16

For a bell-shaped distribution, about _ of the data falls within two standard deviations of the mean.

Answer 16

95%

For a population, 95% of the data will be between µ – 2 and µ + 2
 = sigma symbol

Question 17

A _ indicates how many standard deviations a value is from the mean.

Answer 17

Z-score

Z = (X - mean) / Standard Deviation

For a population, !𝑥 is replaced with µ and s is replaced with 
 = sigma symbol

Remove outliers using z-score
from scipy import stats
df[‘cgpa_zscore’] = stats.zscore(df.cgpa)
df[(df.cgpa-zscore > -3) & (df.cgpa-zscore < 3)]

Question 18

Advantages of Z-score

Answer 18

• Straightforward
• Easy to use
• Useful for normally distributed data
• Quantifies deviation

Question 19

Disadvantages of Z-score

Answer 19

• Sensitive to non-uniform or skewed data distributions
• Not effective in datasets with many outliers
• Assumes data follows a normal distribution, making it less reliable for non-normal distributions

Question 20

What is considered an extreme z-score?

Answer 20

A z-score beyond -2 or 2

Question 21

The _ divide data into four equal parts.

Answer 21

Quartiles

Q1 is the median of the lower half, Q3 is the median of the upper half.

Question 22

What is a five-number summary?

Answer 22

minimum (Min) = smallest data value
Q1= median of the values below m
median (m) = middle data value
Q3 = median of the values above m
maximum (Ma) = largest data value

Question 23

The _ is the value which is greater than P% of the data

Answer 23

Pth percentile

We already used z-scores to determine whether an SAT score of 2100 or
an ACT score of 28 is better

 We could also have used percentiles:
 ACT score of 28: 91st percentile
 SAT score of 2100: 97th percentile

Question 24

The _ is Q3 - Q1, representing the middle 50% of the data.

Answer 24

Interquartile Range (IQR)

Remove outliers using IQR

q1 = df.placement-exam-marks.quantile(0.25)
q3 = df.placement-exam-marks.quantile(0.75)
iqr = q3 - q1
iqr

upper = q3 + (1.5 * iqr)
lower = q1 - (1.5 * iqr)

df[(df[‘placement-exam-marks’] < upper) & (df[‘placement-exam-marks’] > lower)]

Question 25

Advantages of IQR

Answer 25

* Robust in non-normal distributions * Less affected by extreme values or outliers * Effective for skewed datasets

Question 26

Disadvantages of IQR

Answer 26

* Will not detect outliers effectively in small datasets * May miss outliers in the tails if the distribution is highly skewed

Question 27

Is the IQR resistant to outliers?

Answer 27

Yes, the IQR is resistant to outliers.

Question 28

How can outliers be identified using the IQR?

Answer 28

Outliers are data points smaller than Q1 − 1.5(IQR) or larger than Q3 +1.5(IQR). ## Footnote upper = q3 + (1.5 * iqr) lower = q1 - (1.5 * iqr) df[(df['placement-exam-marks'] < upper) & (df['placement-exam-marks'] > lower)]

Question 29

A _ visualizes the five-number summary and shows outliers as points beyond the whiskers.

Answer 29

Boxplot | sb.boxplot(df.placement-exam-marks) ## Footnote Lines (“whiskers”) extend from each quartile to the most extreme value that is not an outlier

Question 30

What is the relationship between range and outliers?

Answer 30

The range is **not resistant** to outliers because it **depends entirely on extreme values**. | Range = Max – Min IQR = Q3 – Q1

Question 31

When should you use the mean and standard deviation?

Answer 31

Use the mean and standard deviation when you want to incorporate all data points into the analysis, but be cautious of outliers.

Question 32

What percentile is a data value at if it is greater than 75% of the data?

Answer 32

The 75th percentile (or Q3)