Simple Linear Regression

Question 1

What is the goal of regression?

Answer 1

To predict Y (outcome variable) from X (predictor).

Question 2

Which variable is fixed in a regression equation?

Answer 2

X is a fixed variable and Y is always the random variable.

Question 3

T or F: there is no sampling error involved Y.

Why or why not?

Answer 3

F. There is no sampling error involved in X because X is a fixed variable, while Y is a random variable.

Question 4

Concerning population parameters to go along with sample statistics in a simple linear regression, what do we predict Y from?

Answer 4

We predict outcome Y from beta naught (intercept) and beta 1 is (slope), multiplied by the predictor. Also the epsilon, or the residual (e) is the population error term.

Question 5

What is our purpose of the modeling error?

Answer 5

Our purpose is the find the line that best summarizes the line between X and Y.

Question 6

What is error called for population and for sample ?

What is model error?

Answer 6

Epsilon for population, e or residual for sample.

It is the difference between the people that deviate from the model line.

Question 7

Define sampling error.

Answer 7

The difference between a population parameter and sample statistics.

Question 8

What is goal in simple linear regression, in regards to the line?

Answer 8

Our goal is to be able to find the best fit of line.

We are trying to find from all possible lines, which one will result in the least amount of difference between the observed data and line.

Question 9

How do we use the regression line to predict values?

Answer 9

We fit a statistical model to the data in the form of a straight line. This line is the line that BEST FITS the pattern of data.

Question 10

What does y-hat indicate?

Answer 10

The line itself, to mark that it’s different than the situation where we have error is notated as y-hat.

Question 11

Which contains error: The line or the model?

Answer 11

The model contains error.

Question 12

Why is Y-hat considered a predictive Y?

What does this have to do with residuals?

Answer 12

Y-hat is a predictive because it signifies the Y-values that are predicted from the line

The difference between what’s predicted from the line and the observed value (Y) is the residual.

Question 13

What is y-hat’s equation?

Answer 13

Y-hat = b0 + b1x

B0 = intercept
B1 = Slope
X = predictor value

Question 14

How to we compute a simple linear regression on r? What does it produce?

Answer 14

rcorr(as.matrix(dataset))

Produces n and p value

Question 15

What information do we need to create a regression equation?

Answer 15

We need to fill in the intercept (b0) and slope(b1) - so we need to determine the line of best fit.

Question 16

How is regression conceptually similar to ANOVA?

Answer 16

With an ANOVA, we compared MSbetween and MS within- we want to minimize MSwithin (error), and we want to do the same with regression by making error as small as possible.

Before, we wanted to see how points deviated from the mean, but now we want to see how each point deviates from the regression line.

Question 17

Why do we create a sum of squares for a simple linear regression equation?

Answer 17

We want to minimize the sum of the squared residuals (OLS solution).

Each point from the line gives residual, and we add them up. The problem arises because the distance of the points above the lines are the same as if they were if we added up below the line, and it adds to zero. So we must square.

Question 18

What are the similarities and differences between a correlation and the simple linear regression?

Answer 18

If there’s just 1 predictor, we see a lot of similarities. The only thing that changes is how we treat the variables (prediction vs. description).

Question 19

After we run the rcorr function on r and we see a significant result of a predictor value, what do we do next?

Answer 19

Since it’s highly correlated, we can predict the direction of that relationship.

Question 20

Why do we compute the Ordinary least squares (OLS) solution? What is the criterion to be minimized in OLS?

Answer 20

We compute the OLS solution because it’s an estimation procedure done for regression where we minimize the sum of the squared residuals.

Question 21

As long as we can put numbers to b0 and b1, what are all the information we can find?

Provide and example if b0 = 1 and b1 = 4.

Answer 21

If I have b0 equal to 1 and b1 equal to 4. I could fill that in into my regression and I would have a slope and residuals… I could fill in 1 + 4x… fill in all of my x values and get my y hat values from that and also the difference between y and y-hat to obtain the residuals… if I have my residuals, then I can do my sum of squared residuals…

Question 22

Explain what derivative is and why we use the function of a derivative.

Answer 22

The derivative is when we find a particular location at which we can draw a line right next to the curve and get a slope, because we can never get a slope of a curve but we can find the lines that are straight that touches our curve at exactly 1 point away and this slope that is tangent to the curve will tell us how good our model is.

When we get to the minimum of sum of squared residuals function after a variety of guesses the computer makes, we will ultimately end up at the minimum of the function - the way we know we are at the minimum of the function is because where the line is tangent to the curve has a slope of zero.

This happens when we take the function and set the derivative to zero - and by the magic of calculus, we will have an equation for b0 and b1.

Question 23

Why is SSy (sum of squares y) missing in the bivariate information in SL regression?

Compare this to correlation equation.

Answer 23

In correlation equation, we divided SSCPxy over the sqrt of SSx and SSy because we were interested in how our variables related to each other after removing the independence about each other those variables.

When I’m interested in predicting Y, I don’t care about how Y varies with itself - only about how Y varies along with X. We aren’t interested in the univariate information of Y, just how the 2 variables relate to each other after removing what is unique to X, so I’m left with Y and everything that’s shared with Y.

Question 24

Conceptually, what is b1 and b2?

What is the equation for both?

Answer 24

b1 (slope) tells me for every 1 unit increase in X, how much Y changes. It tells me the change in Y based on changes in X.

b1 = SSCPxy / SSx

b0 (intercept) is mean of Y minus b1 times the predictor.

To obtain b0, we compute the slope first; then we multiply it by the mean of X.

b0 = Y-bar - b1(X-bar)

Question 25

If b1 equals 0.758, what does this mean? Put this into context when predicting the number of doctor visits and health problems.

Answer 25

For every 1 unit increase in X, Y changes .758. For every 1 unit of additional health problems, the number of doctor visits go up .758 times.

Question 26

In the context of health problems on doctor visits, what does the b0, or the intercept formula, tell us exactly?

Answer 26

It tells us if we had NO physical health problems, we would expect to go to the doctor .036 times (almost zero times).

Question 27

What does the output for regression in R show? What is the function?

Answer 27

The function is fit. The output for regression in R shows the residuals and coefficients.

Question 28

What are the interpretations of b0 and b1 regression equations? Utilize doctor visits and health problems in both interpretations: ``` b0 = .036 b1 = .758 ```

Answer 28

b0 interpretation: The expected number of dv is (b0 value) when no iv has been reported. ex) The expected number of doctor visits is .036 when no physical health problems have been reported. b1 interpretation: The expected number of dv is expected to increase by (b1 value) for every additional iv. ex) The expected number of doctor visits is expected to increase by .758 for every additional physical health problem.

Question 29

How do we find the best fit of a line?

Answer 29

By setting the derivative to 0, we found the best fit of the line in solving for the 2 unknown equations, b0 and b1.

Question 30

Based on OLS, what is the equation of the smallest residual given the information we have?

Answer 30

The SS residual (E squared), and on the information tables, the formula is (Y-ŷ )squared.

Question 31

Which of the following indicates the best fitting line: ``` y ŷ SSCPxy b0 b1 ```

Answer 31

ŷ = we sum up the b0+b1X of all the observations to get the best fitting line.

Question 32

What are given properties of regression equations BASED on OLS solutions being true?

Answer 32

1. Sum of residuals = 0 (y-y-hat = 0) 2. Sum of the squared residuals is at a minimum (y-y-hat squared) 3. Sum of observed values equals sum of the fitted values (y-bar = ŷ ) - The sum of the observed values are equal to the fitted or predicted value. 4. The regression line always goes through the point (X,Y) 5. Residuals are uncorrelated with predictor - the relationship between x and y is uncorrelated. 6. The fitted Y value is less extreme on Y than the associated X value is on X (this property is called Regression towards the mean).

Question 33

What does the 6th property of regression equations mean(Regression towards the mean): "The y-hat values are less extreme on Y than the associated X value is on X"

Answer 33

If we plug in the SD value (above or below the mean), into the regression equation, I will get closer to or regress towards the mean and the outcome will be less extreme. So this makes the ŷ values less extreme (location is closer to the mean) on Y than their corresponding x-values on X. Ex) ŷ = .036 + .758 (7.58) = 5.97

Question 34

Define standard deviation and variance.

Answer 34

SD = the average deviation from the mean Variance = the average SQUARED deviation from the mean

Question 35

If I want to know the variability of the points around the regression line (instead of the variability of points around the mean), what must I compute? Provide the formula for the Standard Error of the Estimate (Residual standard error).

Answer 35

We take the estimate of the scores around the regression line by computing the SD (sum of squares over df) and variance (squared). Formula for regression line: sqrt of Σ (Y-ŷ)² / n-k-1 *n is # of rows or observations *k is # of predictors (1 for simple linear regression).

Question 36

What is the definition of variance of the estimate?

Answer 36

The average squared distance of each point around the regression line

Question 37

In order to use standard error of the estimate of the meaningful way, what has to happen?

Answer 37

Our residuals have to distributed normally.

Question 38

What is the formula for the standard error of the slope? *it will be on the exam (is on the HW assignment).

Answer 38

The square root of MSresidual over SSx. Numerator: It is the square root of the residual, or the variability of our model after removing everything associated to X. Denominator: Square root of the SSx.

Question 39

Why do we compute a one-sample t-test for regression? Conceptually, what's the numerator and denominator indicative of? What is the formula? What does a significant t-value indicate?

Answer 39

T-tests are a measure of whether the predictor is making a significant contribution to the model. We are testing the hypothesis that a b-coefficient significantly differs from zero (meaning it's significantly different from the mean). >If the slope (b1) is not different from zero then we can't use our x-variable to predict y because the slope would be flat (because if our x was 17, it would still be in the same place on y). We are comparing the differences between the b and the amount of error - if the standard error is small, we can conclude that the b-values are all similar to the b-value in our sample. t = b-observed ÷ standard error. A larger t-value, or smaller p-value indicates that the predictor had a significant effect on predicting the outcome.

Question 40

What does it mean when a t-test produces a significant effect for the slope? What if the intercept is not significant?

Answer 40

Our slope is significantly different from zero, which is what we want. If the intercept is not significant, it is not different from the origin.

Question 41

What are the various synonyms of ŷ?

Answer 41

Predicted line | Fitted value

Question 42

When we examine the model fit, what are we examining?

Answer 42

When we take off the hat and add an e for error, we want to know if we're doing a good job predicting the y-value from our line (which is dependent on x-information).

Question 43

What are the identities of model fit? What are we doing when we examine the model fit?

Answer 43

Total deviation = Regression + Residual We are trying to see how good the predictor is.

Question 44

What is the formula for Regression deviation?

Answer 44

(ŷ-ybar)

Question 45

Compare regression deviation and residual deviation to ANOVA. What are we trying to do when we solve for regression deviation? What's the conceptual equation for the Regression deviation?

Answer 45

The regression deviation (or model) is like SSbetween. It's the amount of variability in Y that's explained by our line (predictor). The residual deviation is what can't be explained by the line. I want to know how my model has improved my ability to predict Y over just having the mean of Y. Because ŷ involves x information, so Im looking at: Am I able to improve my ability to predict Y by including y predictor, or is there no real difference between what I have no based on the model and the mean of y. We are trying to see if ADDING the predictor improves my ability to predict y from x, or is it no different from what I have based on our model. So it's the improvement of model over what's unexplained by my model.

Question 46

In a graph of partitioned quantities, how would we identify whether the regression or residual explains the variability? (p.10) How do we compute the variability of the points?

Answer 46

Anything above the y bar is residual, while everything below the y bar is regression. If there is a larger distance between the line and observed value below the slope, it indicates that the regression value is larger than the residual - it's a good predictor bc we are explaining more variability by including the predictor (or doctor visits). We compute the variability of the points by doing squared sums of squares / df... eventually doing MSregression/MSresidual (similar to ANOVA)

Question 47

Since we can partition the variability in regression (SS model and SSerror), what can we build?

Answer 47

A model fit ANOVA table

Question 48

Build the model fit ANOVA table.

Answer 48

Looks great!

Question 49

What is the interpretation of a significant F-value following a model fit ANOVA table? What does the significant F-value indicate?

Answer 49

The regression model significantly fits the data, such that the number of reported physical health problems significantly predicts the number of doctor visits, F (1,8) = 5.85, p

Question 50

How do you run an ANOVA table in R?

Answer 50

(model

Question 51

If our predictor is found to be good, what do we compute next?

Answer 51

R² = The effect size, or the proportion of variance. We divide SSregression with SStotal and report the number as a percentage.

Question 52

What's the difference in r² and R²?

Answer 52

r² is correlation effect size, and we square the correlation. R² is the effect size of regression, where we divide SSregression by SStotal. R² is also the squared correlation between the predictor and outcome.

Question 53

What is the interpretation for R² if it equals .4278?

Answer 53

47% of the variability in Y (number of doctor visits) can be explained by the number of X (reported physical healthy problems).

Question 54

Is 47% of variability enough to explain the majority of the raw proportions?

Answer 54

No, we're not explaining the majority of variability of X on Y. We're only explaining per degrees of freedom.

Question 55

List and describe the Gauss-Markov Assumptions. There are 7 properties.

Answer 55

Running regressions are contingent on Gauss-Markov Assumptions. We also need to know how to test them (which happens after we run the regression model and then evaluate it - after data screening). If any of these are violated, we can look to other tests. The first 3 assumptions are about our variables: 1. All predictors are quantitative (numeric) or dichotomous (gender dummy coded as 0 or 1), and the criterion is quantitative, continuous, and unbounded (-∞ to ∞). All variables are measured without error. 2. All predictors must have non-zero variance. A zero variance indicates that the predictor is constant. We can't make a line, and mathematically will be Undefined. 3. There is an absence of perfect multicollinearity. Multicollinearity states two or more predictor variables in a multiple regression model are highly correlated.. but predictors should not be the same. On a graph, it will look platykurtic. The last 3 assumptions are about our error: 4. The expected (average) value of the error term is 0 at each value of the predictors. If x = 1, then the average residual for all points where x = 1, the average error would equal to zero. 5. Each predictors is uncorrelated with the error term. 6. The variance of the error term is constant (Homoscedasticity) at each value of the predictors. 7. Error terms for different observations are uncorrelated (Independence of observations).

Question 56

Why do we calculate F-tests? What is the formula of doing the F-test of R²? What is this indicating conceptually? Why is this formula important to know?

Answer 56

The significance of R2 can be tested using an F-ratio. F = R²/k over (1-R²)/ (n-k-1). We take our model R² over the variability left-over This formula allows us to get the F-value without computing the SS or variance.

Question 57

Rather than looking at Sums of squares and computing variances, what can we do to obtain the F-value?

Answer 57

We can take the proportions of variability explained by the model and the proportions of variability unexplained by the model, both per degrees of freedom, to produce the F value.

Question 58

Why is meeting the Gauss-Markov assumptions important?

Answer 58

It is important to meet the 7 assumptions to run regressions.

Question 59

What is the assumption of homoscedasticity? What would homo and heteroscedasticity look like on a scatterplot?

Answer 59

It's that our error term is constant at each value of our predictor. Homoscedasticity refers to the assumption that that the dependent variable (y) exhibits similar amounts of variance across the range of values for an independent (x) variable. On a scatterplot, observations that are shaped as megaphones are heteroscedastic- we want an even distribution at the point of X.

Question 60

If all our of Gauss-Marcov Assumptions are met, what does this mean?

Answer 60

``` It means that OLS is Blue! B - best L - linear U - unbiased E - estimator ``` This means that the distribution of my parameters is unbiased and the mean is in the center of the distribution.

Question 61

What is the final assumption of normality? Why do we need normality?

Answer 61

8. At each value of the IVs, the errors are normally distributed (skews, platykurtic, bi-modal). This CAN be violated but still NOT violate Gauss-Marcov assumptions. However, we need normality because to test my coefficients, we used a one-sample t-test... and one of the assumptions of a one sample t-test is normality. The standard error will be incorrect due to bias, therefore, the t-value will be incorrect.

Question 62

What does 'model' in an equation get replaced by?

Answer 62

It gets replaced by a best fit that define the line better than the model.

Question 63

According to the book, a simple way to look at the residual term is:

Answer 63

The difference between the score predicted by the line and the score that the participant actually obtained.

Question 64

Residuals are synonymous with _______.

Answer 64

Residuals are synonymous with 'deviations'. Residuals are technically the deviations from the line.

Question 65

What is goodness of fit, conceptually?

Answer 65

How well a model that is generated fits the data - based on how well the data predicted by the model actually corresponds to the data that's collected. We still need to asses this model to make sure this is the best one for the data.

Question 66

Conceptually, what does the value of SSresidual represent? Conceptually, what does the value of SStotal represent? Together, what do they calculate conceptually?

Answer 66

It represents the degree of inaccuracy when the best model is fitted to the data. SStotal represents the difference between the mean of the observed Ys and the predicted values. Together they are used to calculate how much better the regression line (line of best fit) is better than the mean, or R-squared.

Question 67

Conceptually, what is F-ratio?

Answer 67

It is the measure of how much the model has improved the prediction of the outcome compared to the level of inaccuracy of the model. So MSreg (improvement) ÷ MSres (errors and inaccuracies).

Question 68

Conceptually, why is predicting an outcome to a mean a bad idea?

Answer 68

The line representing the mean is flat - so as predictor values change, the value of the outcome don't change. *thinking back to author predicting $1 and $100,000 difference in advertising - use of mean would say that they would sell $200,000 either way...

Question 69

If R produces b0, or the intercept estimate as 134.1, what is it telling us when we are trying to predict album sales? (*in terms of dollars - round everything to thousands)

Answer 69

That when no money is spent on advertising (when X = 0), the model predicts that 134,100 albums will be sold.

Question 70

If R produces b1, or the slope estimate of 0.096 in predicting album sales, what does this mean? (*in terms of dollars - round everything to thousands)

Answer 70

That for an increase of $1,000, the model predicts 96 extra album sales.

Question 71

What does it mean when assumptions are met?

Answer 71

When all assumptions are met, we can apply the model we get from sample to population because the coefficients and parameters of that regression is unbiased.

Question 72

What are the 3 assumptions related to Variables in the Glauss-Marcov assumptions for regression?

Answer 72

1. All predictors must be quantitative or dichotomous. All criterion must be quantitative, continuous, and unbounded. All without Error. 2. All predictors must have non-zero variance. A zero variance indicates that the predictor is constant, and there won't be a line and it would be undefined. 3. No Multicollinearity - No correlation between predictors.

Question 73

What are the 4 assumptions related to Error in the Glauss-Marcov assumptions for regression?

Answer 73

4. The expected value of the Error term is zero at each value of the predictor. So if X = 1, then the average residual for all points where X = 1 means the average error would equal to zero. 5. Predictors are uncorrelated with the error term. 6. Variance of the Error Term is constant (HOMOSCEDASTICITY) at each value of predictor. 7. Error terms for different observations are Uncorrelated (Independence of observation).

Question 74

What are the criterions to be minimized in the OLS?

Answer 74

We are MINIMIZING the sum of the squared residuals. The properties of regression equations based on OLS solutions being true: 1. Sum of residuals = 0 (y- ŷ = 0) 2. Sum of the squared residuals is at a minimum (y- ŷ squared) 3. Sum of observed values equals sum of the fitted values (y-bar = ŷ ) The sum of the observed values are equal to the fitted or predicted value. 4. The regression line always goes through the point (X,Y) 5. Residuals are uncorrelated with predictor - the relationship between x and y is uncorrelated. 6. The fitted Y value is less extreme on Y than the associated X value is on X (this property is called Regression towards the mean).

Question 75

What does it mean when the Gauss-Marcov assumptions are met?

Answer 75

It signifies that the distribution of the parameters are UNBIASED and the mean is in the center of the distribution.

Question 76

What do we get when we add the assumption of normally distributed errors?

Answer 76

It allows you to have accurate standard errors which allow us to do t-tests properly.

Question 77

What is the Variance of the Estimate and what other names do we know it by?

Answer 77

Variance of the Estimate AKAK MS Residual. Variance of the Estimate measures the Residual stuff that can't be explained by our Regression. MSReg ÷ MSRes

Question 78

How is the Variance of the estimate used to calculate other measures in regression analysis?

Answer 78

It is the denominator in our F-test.

Question 79

What two things can we do once we have standard errors for b's? Know how to do them.

Answer 79

We can run a t-test to see the significance of the Intercept and coefficient.

Question 80

What is the Coefficient of Determination? How do we calculate it from elements in the ANOVA table? How do we test and interpret it?

Answer 80

It's R-squared, and we calculate R-squared by SSRes ÷ SSTotal It explains the variability of X that's attributed to the Criterion.

Question 81

Given output from an SLR, interpret the following: ``` R-squared b0 b1 Plotting the regression line Calculate the predicted and residual values for given X scores ```

Answer 81

R-squared -> The variability of Y that's explained by X b0 ->Value of Y when X is 0 b1->Change is Y when X increases by 1 unit To plot the regression line, we need to plot the predicted values To calculate residual, we do y-hat - y.