[MLS] Exploratory Data Analysis

Question 1

What is the pandas python library used for?

Answer 1

Used to manipulate your data by placing it in dataframes or series

Question 2

What is the matplotlib python library used for?

Answer 2

For visualising data

Question 3

What is the seaborn python library used for?

Answer 3

More expansive version of matplotlib (for visualising data) including heat maps, joint plot

Question 4

What is the scikit_learn python library used for?

Answer 4

For the ML modelling itself

Question 5

What are the 3 major types of data?

Answer 5

Categorical, numerical and ordinal

Question 6

What is the difference between categorical and ordinal data?

Answer 6

Ordinal has an inherent numerical meaning, e.g. 1 star is worse than 4 stars, whereas categorical has no inherent numerical meaning, e.g. gender, race

Question 7

What is a bernoulli distribution?

Answer 7

A discrete probability distribution with only 1 trial

Question 8

What is a probability mass function for?

Answer 8

Discrete data, looks like a histogram, NOT a curve

Question 9

Are QuickSight dashboards read-only?

Answer 9

Yes

Question 10

What would you use a scatter plot or heat map for?

Answer 10

To visualise a correlation

Question 11

What would you use a bar chart for?

Answer 11

Comparison and distribution

Question 12

What is a geospatial chart?

Answer 12

A map with an overlay of information on it

Question 13

What does a master node do in an EMR cluster?

Answer 13

Manages the cluster, assigns tasks

Question 14

What are the differences between a core node and a task node in an EMR cluster?

Answer 14

Core node is constant for the task, connects to the Hadoop File System to share data
Task node is ephemeral, does not connect to HDFS

Question 15

When would you use a transient vs a long-running EMR cluster?

Answer 15

Transient to just shut down the cluster once the list of tasks you have specified are complete, a long-running EMR cluster would be if you’re not sure exactly what you want to do with it so you have easy access when you want it

Question 16

What is important to note about HDFS when closing down a cluster?

Answer 16

It is transient - the data on it will be lost if not backed up when the cluster is shut down

Question 17

What happens if a node in EMR fails in terms of the data on that node?

Answer 17

Nothing - multiple copies of the data are propagated across the nodes to make sure there is no data loss when one fails

Question 18

What is EMRFS?

Answer 18

Elastic Map Reduce File System - uses S3 as if it was HDFS. Still fast, but not transient any more

Question 19

How does EMR charging work?

Answer 19

By the hour

Question 20

Can you resize the core nodes running in an EMR cluster on the fly?

Answer 20

Yes

Question 21

What is Spark Streaming?

Answer 21

Streaming analytics that works by breaking the stream into mini-batches, kind of like continuously appending rows to a table

Question 22

What are EMR notebooks? Where are they backed up to and what do they allow for?

Answer 22

Notebooks for Spark that are a more visual way to interact with the service. These notebooks are backed up to S3. Allows for collaboration.

Question 23

Which nodes in a Spark cluster are most likely to need the most processing power?

Answer 23

Core and Task nodes, Master node is just routing traffic around so doesn’t need to be as beefy.

Question 24

What is the curse of dimensionality?

Answer 24

Too many features can be a problem as they lead to a huge solution space within which the data is sparsely distributed (e.g. if you have 10,000 features and 10,000 rows but only 1 has the entry ‘2’ for one of those features, it will be a very very sparse data set for that one feature with the one entry of ‘2’.)

Question 25

What are 2 algorithms that can be used to reduce the dimensionality of the training set?

Answer 25

Principal component analysis and K-means

Question 26

What type of data works well with deep learning for training set imputation?

Answer 26

Categorical

Question 27

What is MICE?

Answer 27

Multiple imputation by chained equations, a regression technique used for imputations in the training set

Question 28

What is SMOTE?

Answer 28

Synthetic Minority Over-Sampling Technique, a technique to artificially balance a dataset by creating new samples of the minority class using nearest neighbours

Question 29

What is binning?

Answer 29

Bucketing together observations based on a range of values. Transforms numerical data into ordinal data.

Question 30

What is transforming a feature?

Answer 30

Applying a function to a feature to make it better suited to training. Not necessarily replacing data, but making new features.

Question 31

What is one-hot encoding?

Answer 31

Creating buckets for every category, and then giving each entry a 1 or 0 to signify if they are in that category or not.

Question 32

Why can shuffling be useful of the training data?

Answer 32

As it can get rid of any residual patterns resulting from the order in which the data was collected

Question 33

What is SageMaker Ground Truth?

Answer 33

A service that uses humans to do data labelling at the start, and then trains a model off their labelling to do the labelling progressively more and more. The humans can be internal teams or from Mechanical Turk.