Statsmodels

Question 1

What is the linear regression process in short?

Answer 1

Get sample data

Design a model that works for that sample

Make predictions for the whole population

Question 2

Explain dependent and independent variables

Answer 2

There’s a dependent variable labeled Y (predicted)

And independent variables labeled x1, x2, …, xk (predictors)

Y = F(x1, x2, …, xk)

The dependent variable Y is a function of the independent variables x1 to xk

Question 3

Explain the coefficients in y = ß0 + ß1x1 + ε

Answer 3

These are the ß values in the model formula

ß1 –> Quantifies the effect of x on y –> Differs per country for example (income example)

ß0 –> Constant –> Like minimum wage in the income vs education example

ε –> Represents the error of estimation –> Is 0 on average

Question 4

Easiest regression model in formula?

Answer 4

Simple linear regression model

y = ß0 + ß1x1 + ε

y –> Variable we are trying to predict –> Dependent variable

x –> independent variable

Goal is to predict the value of y provided we have the value of x

Question 5

What is the sample data equivalent of the simple linear regression equation?

Answer 5

ŷ = b0 + b1x1

ŷ –> Estimated / Predicted value

b1 –> Quantifier

x1 –> Sample data for independent variable

Question 6

Correlation vs. Regression

Answer 6

Correlation
Measures relationship between two variables
Movement together
Formula interchangeable –> p(x,y) = p(y,x)
Graph –> Single point

Regression
One variable affects the other or what changes it causes to the other
Cause and effect
Formula –> One way
Graph –> Line

Question 7

What can you do with the following modules:

numpy
pandas
scipy
statsmodels.api
matplotlib
seaborn
matplotlib
sklearn

Answer 7

numpy
Working with multi-dimensional arrays

pandas
Enhances numpy
Organize data in tabular form
Along with descriptive data

scipy
numpy, pandas and matplotlib are part of scipy

statsmodels.api
Build on top of numpy and scipy –> Regressions and statsmodels

matplotlib
2D plotting specially designed for visualization of numpy computations

seaborn
Python visualization library based on matplotlib

sklearn
scikit learn –> Machine learning libraries

Question 8

Check which packages are installed

Answer 8

Anaconda Navigator

CMD.exe Prompt

Write “conda list”

Question 9

Upload and access data

Answer 9

Put the csv file in the same folder as the notebook file

data = pd.read_csv(‘filename’)

write ‘data’ –> The data will show up in the output

Question 10

Pull up statistical data of your dataset

Answer 10

data.describe()

Question 11

What are the steps for plotting regression data?

Answer 11

Import relevant libraries
Load the data
Declare the dependent and the independent variables
Explore the data
Regression itself
Plot the regression line on the initial scatter

Question 12

How to find ß0?

Answer 12

Read from the tables to find the numbers for plotting the scatter line

coef

const –> ß0

Question 13

How to determine whether variable is significant?

Answer 13

Hypothesis testing based on H0: ß=0
In the results table this is the same as t + P>|t|
p-value < 0.05 means that the variable is significant

Question 14

What to do if ß0 is not significanty different from 0

Answer 14

They are not calculated in the formula and thus left out of the prediction of the expected value

Question 15

Explore determinants of a good regression

Answer 15

Sum of squares total (SST or TSS)

Sum of squares regression (SSR)

Sum of squares error (SSE)

Question 16

Sum of squares total (SST) formula plus meaning?

Answer 16

∑(yi - ȳ)²

Can think this as the dispersion of the variables around the mean

Measures the total variability of the dataset

Question 17

Sum of squares regression (SSR) formula plus meaning?

Answer 17

∑(ŷi - ȳ)²

Sum of predicted value minus mean of dependent variable, squared

Measures how well the line fits the data

If SSR = SST –> Then the regression line is perfect meaning all the spots are ON the line

Question 18

Sum of squares error (SSE) formula plus meaning?

Answer 18

∑e(i)²

Difference observed value and the predicted value

The smaller the error, the better the estimation power of the regression

Question 19

What is the connection between SST, SSR & SSE

Answer 19

SST = SSR + SSE

In words: The total variability of the dataset is equal to the variability explained by the regression line plus the unexplained variability (error)

Question 20

What is OLS?

Answer 20

OLS –> Ordinary least squares

Most common method to estimate the linear regression equation

Least squares stands for minimizing the SSE (error) –> Lower error –> better explanatory power

OLS is the line with the smallest error –> Closest to all points simultaneously

There are other methods to calculate regression. OLS is simple and powerful enough for most problems

Question 21

What is R-squared and how to interpret it?

Answer 21

R-squared –> How well your model fits your data

Intuitive tool when in the right hands
R² : SSR / SST
R² = 0 –> Regression explains NONE of the variability
R² = 1 –> Regression explains ALL of the variability

What you will typically observe is values from 0.2 til 0.9

Question 22

What is a good R squared?

Answer 22

No rule of thumb!

Depends on the context and complexity of the topic whether the number is a strong indicator

With a mediocre score of R, one might need additional indicators to explain the correlation

The more factors you include in your regression –> The higher the R squared

Question 23

Why multiple regressions?

Answer 23

Good models require multiple regressions in order to address the higher complexity of problems

Population (mulitple regression) model

More independent variables (more than one)
ȳ –> Inferred value
b0 –> Intercept
x1…xk –> Independent variable
b1…bk –> coefficient

ȳ = b0 + b1x1 + b2x2 + b3x3 + … + bkxk

Question 24

What is the goal of building a model?

Answer 24

Goal is to limit the SSE as much as possible

With each additional variable we increase the explanatory power!

Question 25

What is adjusted R-Squared?

Answer 25

Adjusted R-squared is a modified version of R-squared that has been adjusted for the number of predictors in the model. The adjusted R-squared increases when the new term improves the model more than would be expected by chance. It decreases when a predictor improves the model by less than expected.

The R-squared measures how much of the total variability is explained by our model

Multiple regressions are always better than simple ones –> As explained more variables lead to a better explanatory power

Denoted as: r̄²

Question 26

How is the adjusted R squared formula build up, with which variables?

Answer 26

1 - (1 - R-squared) * ((n - 1)/(n - p - 1))

So:
R-squared
n = Total sample size
p = number of predictors

Question 27

How big is Adjusted R squared compared to R squared?

Answer 27

r̄² is always smaller than R²

It penalizes excessive use of variables

Question 28

How to use p-value to determine if a variable should stay?

Answer 28

It’s only significant when p-value < 0.05

Question 29

How to interpret Adjusted R squared?

Answer 29

Check before and after if Adjusted R-squared was lowered or highered

Question 30

What are the consequences of adding useless data?

Answer 30

The formula changes –> Different number for the intercept number of ß0

Thus the bias of this useless variable is reflected into the coefficients of the others

Question 31

What is the simplicity/explanatory power tradeoff

Answer 31

Simplicity is better rewarded than having a high explanatory power!

Question 32

What is the F-statistic and how is it used?

Answer 32

F-statistic > Follows an F-distribution
It is used for testing the overall significance of the model

F-test
Null Hypothesis is that all betas are equal to 0 –> H0: ß1 = ß2 = ß3 = 0
H1: at least one ßi≠0

If all Beta’s are 0 than the model is useless

Compare F-statistic with or without variable –> Lower F-statistic means closer to a non-significant model

Prob(F-statistic) can still be significant but notice the change –> If it’s higher then drop the variable

Question 33

Interpretation of F-statistic?

Answer 33

Prob(F-statistic) very low –> We say overall model is significant

The lower –> The closer to a non-significant model

Don’t forget to look for the 3 zeroes after the dot

Question 34

How to verify linearity?

Answer 34

Plot the data –> If data points form something that can be explained as a straight line –> Then linear regression is suitable

Question 35

Regression assumption:
Explain Endogeneity

Answer 35

σxε = 0 : ∀x, ε

The error (difference observed and predicted values) is correlated with the independent variable –> This is problem referred to with ‘omitted variable bias

Question 36

Explain Omitted variable bias

Answer 36

In general
Omitted variable bias occurs when you forget to include a variable. This is reflected in the error term as the factor you forgot about is included in the error. In this way, the error is not random but includes a systematic part (the omitted variable).

You either include or omit the X variable leading to a difference in error –> Therefore the x and ε are somewhat correlated

Question 37

Regression assumption:
Explain Normality and homoscedasticity

Answer 37

ε ~ N(0, σ²)

Comprises:
Normality –> We assume the error term is normally distributed

Zero mean

Homoscedasticity

Question 38

When in doubt about including variable, what should you do?

Answer 38

Just include the variable –> Worst thing that can happen is that it leads to inefficient estimates

You can then immediately drop that variable

Leaving out a great variable does a lot more harm!

Question 39

What if error term is not normally distributed in Normality and homoscedasticity?

Answer 39

CLT Applies

Remember:

In probability theory, the central limit theorem (CLT) establishes that, in many situations, for independent and identically distributed random variables, the sampling distribution of the standardized sample mean tends towards the standard normal distribution even if the original variables themselves are not normally distributed.

Question 40

What does diffusing mean?

Answer 40

Diffusing means the correlation is there for lower values but not for higher values –> We don’t like this pattern –> Heteroscedasticity

Question 41

Example of heteroscedasticity?

Answer 41

Poor person will have the same dinner every day –> Low variability

Rich person will eat out and then dine in the next day –> High variability thus we expect heteroscedasticity

Question 42

How to prevent heteroscedasticity?

Answer 42

Check for omitted variable bias:OVS

Look for outliers

Log Transform –> A statistician’s best friend

Question 43

Apply logarithmic axes

Answer 43

By changing the scale of X, reduces the width of the graph
New model is called: Semi-log model

Denoted as:
ŷ = b0 + b1(log x1)

or:
log ŷ = b0 + b1x1

Meaning: As X increases by 1 unit, Y increases by b1 percent

Question 44

Log-log model?

Answer 44

When using log on both axes:
log ŷ = b0 + b1(logx1)

Interpretation: As X increases by 1 percent, Y increases by b1 percent

Relation is known as elasticity

Question 45

Regression assumption: Explain ‘No autocorrelation’

Answer 45

Autocorrelation is a mathematical representation of the degree of similarity between a given time series and a lagged version of itself over successive time intervals. It’s conceptually similar to the correlation between two different time series, but autocorrelation uses the same time series twice: once in its original form and once lagged one or more time periods.

No autocorrelation
σ(εiεj) = 0 : ∀i ≠ j

Errors are assumed to be uncorrelated
Highly unlikely to find it in cross sectional data
Very common in time-series data such as stock prices

Question 46

Spot autocorrelation

Answer 46

Look at the graph –> If you can’t find any, you are safe

Durbin-Watson test

Generally its values fall between 0 and 4

2 –> No autocorrelation

<1 and >3 are a cause for alarm

Conclusion: When in the presence of autocorrelation avoid the linear regression model

Question 47

Regression assumption: No Multicollinearity

Answer 47

ρ(xixj) ≈ 1 : ∀i,j; i ≠ j

Is observed when 2 or more variables have a high correlation among each other

Example: a = 2 + 5 * b

In this case there is no point in using both a and b because they are correlated

ρ(cd) = 0.9 –> Imperfect multicollinearity

Question 48

How to deal with catagorical data?

Answer 48

Use a dummy instead and explain it later on –> Transform yes and no into 1 and 0

You can do calculations with 0 and 1

Question 49

What will the categorical data graph look like?

Answer 49

You will get two different models:
ȳ = b0 + b1x1 + b20 = b0 + b1x1
ȳ = b0 + b1x1 + b20 = b0 + b1x1 + b2 = (b0+b2) + b1.x1

Results in two lines with equal slope but different intercept