Data Integration, Learning from data, Supervised Learning

Question 1

Define Data Integration… Why is it needed?

Answer 1

Process of combining data from heterogenous sources into a single, coherent data store.

Data sources are usually disparate and siloed. Data integration enables the access and interpretation of data from different sources and types.

Question 2

What are the 5 main ways of integration data? Describe each…

Answer 2

Common User Interface : Manual data integration by a data manager from retrieval to presentation.

Middleware Data Integration : A piece of Middleware that facilitates integration between systems. Usually legacy and new systems.

Application-Based Integration : A Software Application that locates, retrieves and integrates data into storage. Essentially, conducting the entire process, as opposed to Middleware.

Uniform Data Access : Provides a consistent view of data from a variety of sources, but doesn’t retrieve or manipulate the data.

Common Data Storage : E.g a Data Warehouse.

Question 3

For each type of Data Integration process, give a pro and a con…

Answer 3

Common User Interface :
Pro = Total control and handling.
con = Poor scaling.
Middleware Data Integration :
Pro = Automated integration.
con = Must be maintained.
Application-Based Integration :
Pro = Automated end to end process.
con = Complex setup.
Uniform Access Integration :
Pro = Low storage requirements.
con = Hosts struggle w/ data request count.
Common Data Storage :
Pro = Reduces burden on host system.
con = Increased storage costs.

Question 4

What are the 3 categories of learning from data? Define each…

Answer 4

Supervised : Learning that has an Input set and Output set. The goal is to establish the mapping function that gives the most precise continuous target or outcome.

Unsupervised : Learning in which we only have input, and we are tasked with making sense of it.

Semi-supervised :

Question 5

What is the most common type of learning?

Answer 5

Supervised

Question 6

What are the 2 types of Supervised Learning?

Answer 6

Regression
Classification

Question 7

Define Regression…

Answer 7

The process of finding a continuous target or outcome.

Question 8

Define Classification…

Answer 8

The process of classifying inputs.

Question 9

In Supervised Learning, what are we trying to find?

Answer 9

The mapping function.

Question 10

What are the inputs of a supervised learning model?

Answer 10

Features, covariates, predictors etc.

Question 11

What are the outputs of a supervised learning model?

Answer 11

Target, label, response etc.

Question 12

Give an example of a usage of Supervised Learning. Define the Inputs and Outputs.

Answer 12

An input set of dog photos, a boolean output set, and a model that predicts whether each photo is a dog.

Question 13

Define Unsupervised Learning…

Answer 13

The process of making sense of a data set by recognising patterns.

Question 14

Define a Regression Problem and give an example…

Answer 14

A problem in which we need to find a continuous target or outcome.

Question 15

Define what is meant by inputs, outputs and parameter variables of a Mapping Function…

Answer 15

Inputs : Input value
Parameters : The values that will change as the model learns from the data.
Output : The predicted value

Question 16

Regarding the a data frame in linear regression, what does a column represent? And what does a row represent?

Answer 16

Each column is a Feature of the input data e.g join data, monthly price etc.

Each row is an Observation of the customer.

Question 17

Regarding the input and output data frames, what is the difference in their number of rows and columns?

Answer 17

The number of rows will always be the same, N, if N is the count of input data.

The number of columns in the output Y will be 1. The number of columns in input X will vary depending on features.

Question 18

Define what parameters are in a mapping function.

Answer 18

Values that will change over time to give us the most accurate regression line.

Question 19

What makes good parameters?

Answer 19

When the parameter values give us the most accurate model.

Question 20

Define Hyper-Parameters…

Answer 20

Parameters that we select as the model progresses. These are not learned from the data.

Question 21

What are the 2 phases of learning the parameters? Define each…

Answer 21

Training phase : Use past data to find quality parameters. The more past data used, the better the parameters.

Prediction phase : Run new data into our model, and assess accuracy via a Loss Function.

Question 22

What is a Loss Function and when is it used?

Answer 22

A Loss Function is used in the Prediction Phase of Machine Learning.

Its purpose is to output a value that represents how close our models output value is to the actual value. Parameters are then updated depending on the Loss Functions output.

Question 23

What is the reason for updating the parameters after the prediction phase? What determines the values to update to?

Answer 23

To improve the prediction accuracy of the model.

The Loss Function determines the best parameters to update to.

Question 24

Give a simple Linear Regression function…

Answer 24

yb(x) = b0 + b1x + e

Question 25

In order to find the linear regression value, what 2 properties of the linear regression line do we need to know?

Answer 25

Slope of the line.
Y-intercept.

Question 26

What is the equation of finding the Y-intercept of the simple regression line?

Answer 26

b0 = y median - (b1 * x median)

Question 27

What is the equation to find the slop of the simple linear regression line?

Answer 27

b1 = r * ( sy / sx )

Question 28

Define the Pearson Correlation…

Answer 28

A measure of strength of the linear relationship between 2 samples.