2. Exploring Data and Building Data Pipelines

Question 1

What is data visualization?

Answer 1

It is a data exploratory technique to find trends and outliers in data.
It helps data cleaning and feature engineering.

Question 2

What are the two ways to visualize data?

Answer 2

Univariate analysis (range & outliers)
Bivariate analysis (correlation between features)

Question 3

What does box plot consist of?

Answer 3

minimum, 25th, 50, 75 quartiles, maximum

Question 4

What is line plot for?

Answer 4

It is for showing the relationship between two variables and analyze trends

Question 5

What is bar plot for?

Answer 5

It is for analyzing trends and compare categorical data.

Question 6

What is scatterplot for?

Answer 6

Visualize clusters and show the relationship between two variables.

Question 7

What are the three measures of central tendency?

Answer 7

Mean
Median
Mode

Question 8

Which one of the three measures is affected by outliers?

Answer 8

Mean

Question 9

What is standard deviation?

Answer 9

It is the square root of the variance.
It is a good way to identify outliers.

Question 10

What is covariance?

Answer 10

It measures how much two variables vary from each other.

Question 11

What is correlation?

Answer 11

It is a normalized form of covariance ranging from -1 to +1.

Question 12

Can correlation be used to detect label leakage?

Answer 12

Yes, for example, hospital name shouldn’t be used as a feature as the name (cancer hospital) may give out the label.

Question 13

What are the two elements determining your model quality?

Answer 13

Data Quality
Reliability (missing values, duplicate values and bad features)

Question 14

How do you make sure a dataset is reliable?

Answer 14

Check for label errors
Check for noise in features
Check for outliers and data skew

Question 15

What is normalization?

Answer 15

It is to transform features to be on a similar scale.

Question 15

What does data skew mean?

Answer 15

It means the normal distribution curve is not symmetric. There are outliers.
If skewness is in the target variable, you can use oversampling or undersampling.

Question 15

What is scaling?

Answer 15

Convert floating-point feature values from their natural range into a standard range.

Question 16

What are the benefits of scaling?

Answer 16

Help gradient descent converge better in DNN
Remove NaN traps
Won’t give too much importance to features with wider ranges

Question 16

What is log scaling?

Answer 16

When some data samples are in the power of law or very large. Log will bring them to the same range.

Question 17

What is clipping?

Answer 17

Cap all features above or below a certain value.
It can be used before or after normalization.

Question 17

What is Z-score?

Answer 17

scaled value = (value - mean) / stddev
The value is calculated as standard deviations away from the mean.

Question 18

What visualization or statistical techniques can be used to detect outliers?

Answer 18

Box plots
Z-score
Clipping
Interquartile range
You can remove outliers or impute them

Question 19

What are the purposes for data analysis and exploration?

Answer 19

Lead to key insights
Define a schema

Question 19

What is TensorFlow Data Validation for?

Answer 19

Understand, validate, monitor ML data at scale to detect data and schema anomalies.

Question 20

What are the benefits of having a schema?

Answer 20

Enable metadata-driven preprocessing
Validate new data and catch anomalies

Question 21

What are the key TFX libraries?

Answer 21

TF Data Validation
TF Transform: data processing and feature engineering
TF Model Analysis: model evaluation and analysis
TF Serving: Serving models

Question 22

What are the uses of TFDV?

Answer 22

Produce a data schema
Define the baseline to detect skew or drift during training and serving.

Question 23

What are the characteristics of TFDV?

Answer 23

Built on Apache Beam for building batch and streaming pipelines. It can be run on Google Cloud Dataflow.
Dataflow is a managed service for data processing
Dataflow integrates with data warehousing serverless service, e.g., BigQuery, Cloud Storage and Vertex AI Pipelines.

Question 24

What is imbalanced data?

Answer 24

Two classes in a dataset are not equal.
You can perform oversampling or undersampling.
Or, you can do downsample and upweight the majority class. It is faster to converge.

Question 25

What is dataset splitting?

Answer 25

Training: train the model
Validation: hyperparameter tuning
Test: evaluate the performance

Question 26

How do you split dataset for online systems?

Answer 26

Split the data by time as the training data is older than the serving data.

Question 27

What are the ways to handle missing data?

Answer 27

Delete a row if it has more than one missing feature values
Delete a column if it has more than 50% missing data
Replace missing data with mean, median or mode
Replace missing data with most frequent category
Replace with last observation (last observation carried forward)
Use interpolation in time-series
Some ML algorithms can ignore missing values
Use machine learning to predict

Question 28

What is data leakage?

Answer 28

Expose test data during training
Lead to overfitting

Question 29

What are the reason for data leakage?

Answer 29

Add the target variable as your feature
Include test data in the training data
Expose information about the target variable after deployment
Apply preprocessing techniques to the entire dataset.

Question 30

What are the situation indicating data leakage?

Answer 30

The predicted output is as good as the actual output.
Features are highly correlated with the target.

Question 31

How do you prevent data leakage?

Answer 31

Select features not correlated with the target
Split data into test, train and validation sets
Preprocess training and test data separately
Use a cutoff value on time for time series.
Cross-validation when you have limited data