Topic 3: Machine Learning: Regression, Support Vector Machine & Time Series Models

Question 1

Define information

Answer 1

A quantity which reduces uncertainty about something

Question 2

Define prediction in the context of data science

Answer 2

A formula for estimating an unknown value of interest: the target

Question 3

Compare and contrast predictive modeling with descriptive modeling.

Answer 3

Predictive Modelling tries to estimate a value while Descriptive Modelling tries to gain insight into the underlying phenomenon or process.

Question 4

Define attributes or features

Answer 4

Attributes or features are selected variables used as input to estimate the value of the target variable. In database terminology these are the columns (instances or feature values are the rows).

Question 5

Describe model induction

Answer 5

the procedure that creates the model from the data is called the induction algorithm or learner.

Induction = generalizing from specific cases to general rules

Question 6

Contrast induction with deduction

Answer 6

Deduction starts with general rules and specific facts and creates other specific facts from them.

Question 7

Define the training data and labeled data.

Answer 7

training data is the input data for the induction algorithm. They are called labeled data because the value for the target variable is known.

Question 8

Describe supervised segmentation

Answer 8

To determine which are the most informative attributes (columns) when predicting the value of the target you can use supervised segmentation.

Question 9

List the complications arising from selecting informative attributes.

Answer 9

• Attributes rarely split a group perfectly
• Not all attributes are binary
• Some attributes take on numeric values

Question 10

When is a segmented group considered pure?

Answer 10

If every member of the group has the same value for the garget, then the group is pure.

Question 11

What do you call outcome of a formula that evaluates how well each attribute splits a set of examples into segments?

Answer 11

purity measure or splitting criterion (most common one is information gain which is based on entropy)

Question 12

Define entropy

Answer 12

Entropy measures the general disorder of a single set and corresponds to how mixed (impure) the segment is with respect to properties of interest.

high mix = high impurity = high entropy

Question 13

Calculate the value of entropy

Answer 13

Parent set = 10, 7 non-write off, 3 write off

P(non-write off) = 7/10 = 70%
P(write off) = 3/10 = 30%

entropy = -[0.7 x log2(0.7) + 0.3 x log2(0.3) = 0.88

Question 14

Define information gain

Answer 14

a measure how much an attribute improves (decreases) entropy over the whole segmentation it creates.

IG -> change in entropy due to any amount of new information added in

Question 15

Formula information gain

Answer 15

Parent entropy - (weighted average of children’s entropy)

Question 16

Calculate information gain for a set of children from a parent set

Answer 16

IG(parent, children) = entropy(parent) - [p(c1) x entropy(c2) + p(c2) x entropy(c2) + …]

Question 17

How does entropy relate to information gain?

Answer 17

entropy is a measure of disorder in the dataset, information gain is a measure of the decrease in disorder achieved by segmenting the original data set

Question 18

Discuss the issues with the numerical variables for supervised segmentation

Answer 18

Does it make sense to create a segment for each number? Numeric values are often discretized by choosing a split point (e.g. larger than or equal to 50%)

Question 19

Define variance and discuss its application to numeric variables for supervised segmentation.

Answer 19

Variance is a measure for numerical values. You can look at the information gain by reductions in variance between parents and children.

Question 20

Define an entropy graph/chart

Answer 20

X-axis proportion of the dataset, Y-axis is the entropy
the shaded area is the entropy when divided by some chosen attribute.

Goal is to decrease the shade.

Question 21

Describe how an entropy chart can be used to select an informative variable.

Answer 21

Select the attribute which decreases the shaded area the most and does so for most of the values

Question 22

Define a classification tree and decision nodes.

Answer 22

A classification tree (supervised segmentation) starts with a root node with branches to nodes (decision nodes) and ultimately to a terminal node or leaf.

Question 23

Define a probability estimation tree, and tree induction.

Answer 23

probability estimation tree -> leafs contain probabilities
tree induction -> at each step select an attribute to partition the current group into subgroups that are as pure as possible with regards to the target variable (e.e.g Oval Body/Square Body)

Question 24

Define a decision surface or decision boundaries.

Answer 24

Lines separating the regions in an instance space (scatterplot)

Question 25

Describe the relationship between the decision surface and the number of variables.

Answer 25

n variables gives n-1 dimensional hyperplane

Question 26

Define frequency-based estimation of class membership probability

Answer 26

At a leaf if you have n positives and m negatives the frequency based probability of n is n/(n+m)

Question 27

Describe how Laplace correction is used to modify the probability of a leaf node with few members.

Answer 27

If you have one observation at a leaf the probability is 100%, LaPlace corrects for that. n+1 / (n+m+2) The higher the number of instances the less effect you have of the LaPlace correction

Question 28

Define a linear classifier.

Answer 28

Weighted sum of the values for the various attributes

Question 29

Define a linear discriminant.

Answer 29

Decision boundary where you classify instances of x (e.g. + or -)

Question 30

Describe decision boundaries in 2-dimensions, 3-dimensions, and higher dimensions.

Answer 30

Decision boundaries: 2-dimension = above or below the line 3-dimension = a plane Higher-dimensions = hyperplane

Question 31

Interpret the magnitude of a feature’s weight in a general linear model.

Answer 31

heavier weight = more importance

Question 32

Describe how linear discriminant functions can be used for scoring and ranking instances.

Answer 32

the output of the function gives a ranking itself (the further away from the decision boundary the more certain the instance belongs to the class)

Question 33

Describe the objective function of the Support Vector Machine (SVM).

Answer 33

SVM (linear discriminants) fits the fattest bar between the classes (Maximizing margin) and the linear discriminant will be the center line.

Question 34

Describe the important ideas behind the SVM.

Answer 34

- Margin-maximizing boundary gives leeway for classifying such points - Points on the wrong side will incur a penalty OR causes the margin to be changed

Question 35

Define the hinge-loss function, zero-one loss function, and squared error.

Answer 35

Hinge-loss function: for points beyond the margin loss increases linearly with the distance from the margin Zero-one loss function: 0 for correct decision, 1 for incorrect decision Squared error function: squares the distance (i.e. large mistakes are grossly penalized)

Question 36

Describe the reason for not using a squared loss function in classification problems.

Answer 36

It also penalizes points far on the correct side of the decision boundary

Question 37

Describe the major drawback of the least-squares regression.

Answer 37

Sensitive to the data (outliers cause skews)

Question 38

Calculate odds and log odds.

Answer 38

odds: P(event happening)/P(event not happening) Log odds: log(odds) -> so now event not happening odds range between infinity to 0 (before log odds it is between 0 and 1)

Question 39

List the important features of the logistic regression.

Answer 39

- For probability estimation it uses the same linear model as for linear discriminants - Output is interpreted as the log-odds of class membership - Log odds can be translated directly into the probability of class membership

Question 40

Calculate class probability using the logistic function.

Answer 40

1/1+e^-f(x)

Question 41

Describe the shape of the logistic function.

Answer 41

s shape (sigmoid function)

Question 42

Compare and contrast classification trees with linear classifiers.

Answer 42

- Classification tree uses boundaries that are perpendicular, linear can take any direction or orientation - Classification tree is a "piecewise" classifier (cuts up instance space into smaller attributes. Linear classifier places a single decision surface through the entire space.

Question 43

Define the residual and the residual sum of squares (RSS).

Answer 43

- the residual is the difference between y and y hat | - RSS is the sum off all residuals squared

Question 44

Calculate the value of RSS.

Answer 44

error1^2+error2^2+error3^2....errorN^N

Question 45

Calculate the least-squares coefficient estimates.

Answer 45

Least-squares uses the intercept (b0) and slope (b1) to minimize the errors

Question 46

interpret the least-squares coefficients.

Answer 46

b0 - intercept - expected value of Y when X = 0 b1 - slope - avg. increase in Y for a unit rise in X error = catchall for all errors missed with this simple model

Question 47

Define the population regression line and least-squares line.

Answer 47

population regression line: best linear approximation to the true relationship of X and Y. (unobserved) least-squares-line: best estimate based on observed data

Question 48

Define the concept of bias and unbiased estimators.

Answer 48

an unbiased predictor -> statistics calculated from a large set of observations will approach the population statistic statistics (bo, b1, avg) calculated on a small set of observations will under- or overestimate the statistic

Question 49

Define standard error

Answer 49

Standard error (variance/n) is the average amount that the estimate differs from the actual value (the higher the n the smaller the standard error).

Question 50

Define residual standard error

Answer 50

the approximation of the standard deviation from the observed data (RSE = SQRT(RSS/N-2)

Question 51

Calculate the 95% confidence interval.

Answer 51

b1 +- 2 x SE(b1) 95% chance that the interval will contain the true value of b1

Question 52

Describe null and alternative hypotheses

Answer 52

null Hypothesis: No relationship between X and Y | alternative Hypothesis: There is some relationship between X and Y

Question 53

Calculate the t-statistic.

Answer 53

(b1-0)/SE(b1) this measures the number of standard deviations b1 is away from 0.

Question 54

Explain the rules for rejecting the null hypothesis using p-values.

Answer 54

a low p-value -reject null hypothesis (there is a relationship between X and Y).

Question 55

Assess the accuracy of linear regression

Answer 55

RSE (lack of fit) smaller if there is a better fit | R^2 (absolute measure of lack of fit), higher = better

Question 56

Calculate the R2 statistic given TSS and RSS.

Answer 56

TSS-RSS/TSS or SSE/SST or 1-RSS/TSS RSS measures the amount of variability that is left unexplained after performing the regression.

Question 57

Interpret the given values of R^2

Answer 57

Close to 1 -> large part of the variability is explained by regression.

Question 58

Describe the advantages of the R2 statistic over the RSE

Answer 58

Interpretational advantage (between 0 and 1)

Question 59

Describe the relationship between R2 and correlation

Answer 59

Both a measure of linear relationship

Question 60

Define the total sum of squares

Answer 60

Total variance in response to Y

Question 61

Describe how the relationship between responses and predictors is tested in multiple linear regression.

Answer 61

``` H0 = B1=B2=Bn=0 H1 = at least one bj is non-zero ``` This is done by an F-statistic

Question 62

Calculate the F-statistic given TSS, RSS, n, and p.

Answer 62

F = [(TSS-RSS)/k]/[RSS/n-k-1]

Question 63

Explain how the F-statistic can be used for hypothesis testing

Answer 63

High F-statistic suggest at least one variable is related to the Y variable. It is dependent on n and k.

Question 64

Explain why the value of the t-statistic can be a misleading indicator of variable importance in a multiple regression

Answer 64

High chance we will incorrectly conclude there is a relationship

Question 65

Describe how to determine the importance of variables in a given multiple regression.

Answer 65

1. Mallow's Cp 2. Akaike information criterion (AIC) 3. Bayesian information criterion (BIC) 4. Adjusted R2 5. Plotting residuals

Question 66

Define forward selection, backward selection, and mixed selection in the context of variable selection in MLR.

Answer 66

Forward selection: keep adding variables that add the lowest amount of RSS. Backward selection: start with all variables and reduce the variable with the largest P-value. Mixed selection: Combination of Forward and Backward selection.

Question 67

Calculate RSE given the values of RSS, n, and p.

Answer 67

Square[1/(n-p-1)*RSS]

Question 68

Define dummy variables.

Answer 68

Turning a qualitative value into a numerical one with two possible values 0 and 1

Question 69

Describe how to use qualitative variables with more than two levels in multiple regression.

Answer 69

You can create additional dummy variables (splitting the attributes into different predictors)

Question 70

Describe additive and linear assumptions for the linear regression model

Answer 70

Additive: Association between a predictor and Y does not depend on depend on the values of other predictors. linear: the change in Y is constant for an increase in Xj

Question 71

Define the interaction effect

Answer 71

interaction effect = synergy between predictors

Question 72

Explain when an interaction term should be added to a multiple regression model.

Answer 72

1. when they have large main effects. | 2. the interaction has been proven in earlier studies

Question 73

Describe the hierarchical principle for multiple regression.

Answer 73

If the interaction between X1 and X2 seem important we should always include the main effects even if p-values are low

Question 74

Define polynomial regression.

Answer 74

Accommodates non linear relationships

Question 75

Describe the potential problems for a linear regression model, such as non-linearity, correlation of error terms, a non-constant variance of error terms, outliers, high-leverage points, and collinearity.

Answer 75

non-linearity: real relationship might not be linear correlation of error terms: underestimates true standard errors (can lead to narrow prediction intervals) non-constant-variance: or heteroskedasticity can be reduced by taking the log or square root (to shrink outcomes to better fit the line). outliers: can have a large effect on R2 and RSE High-leverage points: unusual value for x1 collinearity: two or more variables are closely correlated. Increases standard error and thus declines the t-statistic

Question 76

Recognize and apply the relationship between arithmetic and geometric returns

Answer 76

When the arithmetic returns are small there will be little difference between geometric and arithmetic returns. When volatility increases and time decreases the difference grows larger.

Question 77

Describe the shape of the plotted line when geometric returns are plotted against arithmetic returns.

Answer 77

Arithmetic follows a straight line while geometric returns follow a curve

Question 78

Define time resolution and time horizon.

Answer 78

resolution: how densely data is covered (the finer the resolution the fatter the tails) time horizon: short periods = fatter tails

Question 79

Describe a random walk model and an autoregressive model.

Answer 79

random walk: Yt = drift term + Yt-1 + error AR(1) model: Yt = drift + aYt-1 + error a = mean reversion

Question 80

Define stationarity

Answer 80

stationarity: no trend, and covariance doesn't change

Question 81

Recognize and apply the formula for the autocorrelation function.

Answer 81

Yt - Yt¡1 = ¹ + (a-1) Yt¡1 + et

Question 82

Describe a GARCH(1,1) model

Answer 82

A mixture of normal distribution with different variances

Question 83

List the conditions that must be satisfied by the parameters of a GARCH(1,1) model

Answer 83

0 =< a =< 1, 0 =< b =< 1

Question 84

Describe the goodness-of-fit for a GARCH model

Answer 84

checking significance of the parameter estimates and how well it models the volatility of the process.