week 5 & 6 Regression Flashcards
1
Q
What is the primary difference between ANOVA and regression in terms of the types of studies they are used for?
A
ANOVA is for experimental studies while regression is for observational studies.
2
Q
What does regression fundamentally begin with?
A
Regression begins with correlation.
3
Q
What is the relationship between correlation and causation?
A
Correlation does not imply causation.
4
Q
What is the common measure of effect used in both correlation and regression?
A
The correlation coefficient (r²) which represents the proportion of variance explained.
5
Q
What can restriction of range on the independent variable lead to regarding the relationship?
A
It can underestimate the relationship.
6
Q
How can extreme cases or outliers affect linear models in regression?
A
Outliers can skew the correlation, inflating or deflating results.
7
Q
What is a potential issue with using poor or proxy measures in correlation?
A
It may underestimate the correlation.
8
Q
What does the strength of effect in regression correspond to in terms of r² values?
A
r² values indicate the proportion of variance explained, with higher values indicating stronger relationships
9
Q
A statistical method for predicting the value of one variable from another, using one or more predictors.
A
Regression
10
Q
A measure that quantifies the direction and strength of a linear relationship between two variables.
A
Correlation Coefficient (r)
11
Q
The portion of variability in a dependent variable that can be attributed to the independent variable(s) in a regression model.
A
Variance Explained
12
Q
What is the difference between simple regression and multiple regression?
A
Simple regression uses one predictor (independent variable), while multiple regression uses two or more predictors.
13
Q
A data point that differs significantly from other observations and can substantially affect the results of statistical analysis.
A
Outlier
14
Q
The variable that is manipulated or varied in an experiment or regression analysis to assess its impact on the dependent variable.
A
Independent Variable (IV)
15
Q
Why is it beneficial to use regression over simple correlation?
A
Regression allows for prediction of the outcome variable while accounting for multiple predictors, enhancing the understanding of variable relationships.
16
Q
What does a linear relationship express in the context of regression?
A
A linear relationship is expressed as a straight line.
17
Q
What is the relationship between a line and a model in regression?
A
Your line is your model.
18
Q
What are the two fundamental features that all lines possess in regression analysis?
A
All lines have a slope and an intercept.
19
Q
What does the term ‘error’ refer to in the context of regression?
A
Error refers to the difference between your modeled line and the actual data points.
20
Q
What does b1 represent in regression analysis?
A
b1 is the regression coefficient for the predictor and represents the gradient (slope) of the regression line, indicating the direction and strength of the relationship.
21
Q
What is represented by b0 in a regression equation?
A
b0 is the intercept, which is the value of Y when X = 0, marking the point where the regression line crosses the Y-axis.
22
Q
How can one estimate the outcome using multiple predictors in regression?
A
By entering the value of the predictor, multiplied by the coefficient, and adding the intercept, one can estimate the outcome.
23
Q
A statistical process for estimating the relationships among variables, allowing for the prediction of one variable based on the values of others.
A
Regression
24
Q
The slope of a regression line, represented by b1, indicates the direction and strength of the relationship between the independent and dependent variables.
A
Slope
25
The intercept of a regression line, represented by b0, is the value of the dependent variable when all independent variables are set to zero.
Intercept
26
The discrepancy between predicted values from the regression model and the actual observed values.
Error
27
A variable that is used in a regression model to predict the outcome of another variable.
Predictor
28
What is the method of least squares used for?
It is used to find the line of best fit for a set of data by minimizing the sum of the squares of the residuals.
29
What is a residual in the context of least squares?
A residual is the difference between the observed data and the predicted values generated by the model.
30
A straight line that best represents the data points in a scatter plot, minimizing the sum of the squares of the vertical distances of the points from the line.
Line of best fit
31
A statistical method used to measure the total variability in a dataset, often decomposed into different components such as total variability, model variability, and residual variability.
Sums of Squares
32
How do you assess the quality of a regression model?
By analyzing how well the model fits the observed data through metrics like Sums of Squares, ANOVA output, and Mean Squared Error
33
Total Sum of Squares, representing total variability in the data, calculated as the variability between individual scores and the mean.
SST
34
Sum of Squares for Residuals, indicating the variability between the actual data and the values predicted by the regression model.
SSR
35
Sum of Squares for the Model, measuring the improvement in variability explained by fitting the regression model compared to the mean.
SSM
36
What does a higher SSM compared to SSR indicate
It suggests that the model provides better predictions than simply using the mean of the observed data
37
What does ANOVA stand for in the context of regression analysis?
ANOVA stands for Analysis of Variance, which tests the differences between the means of several groups and is used to evaluate the performance of the regression model.
38
A metric that quantifies the average squared difference between the observed values and the values predicted by the model, reflecting the error of the model.
Mean Squared Error (MSE)
39
What does the F-ratio represent in regression analysis?
The F-ratio is a statistic calculated to compare the mean of the sums of squares from the model to the mean of the sums of squares from the residuals, indicating whether the model effectively explains the variance.
40
A measure that represents the proportion of variance accounted for by the regression model, indicating the strength of the relationship between the predictors and outcome variable.
r² (R-squared)
41
How is r² similar to Pearson Correlation
r² is similar to squaring the r value obtained from Pearson Correlation, as it provides an understanding of the proportion of variance explained, but it can include multiple predictors in regression.
42
The measure of how much the values in a dataset differ from the mean of that dataset, reflecting the spread of the data points.
Variance
43
What role does the residual play in evaluating the fit of the regression model?
The residual indicates how well the model predicts the observed data; smaller residuals signify a better fit.
44
A line that best describes the relationship between independent and dependent variables, derived from the least squares method.
Regression Line
45
What factor determines whether a regression model is preferred over the mean?
If the model results in significantly lower error and better predictions compared to the mean, it is considered preferred.
46
The process of using a regression model to estimate future values of the dependent variable based on new values of independent variables
Prediction
47
What does a regression analysis output typically include?
It includes statistics such as R-squared, F-ratio, coefficients of the predictors, p-values, and sums of squares (SST, SSR, SSM) to evaluate model performance
48
Write up the regression equation:
Yi = b0 + b1Xi
outcome = intercept + slope * predictor
49
What is multiple regression?
Multiple regression is a model to predict the value of one variable from multiple predictors, extending simple regression to include several variables.
50
How does multiple regression relate to simple regression?
Multiple regression is a natural extension of the simple regression model, which involves predicting a variable from just one predictor.
51
What type of relationship does multiple regression model?
Multiple regression models the hypothetical relationship between several variables.
52
What is the structure of a multiple regression equation?
The equation of multiple regression is similar to that of simple regression but includes additional predictors, forming a straight line
53
In a regression equation, what does b0 represent?
In a regression equation, b0 represents the intercept, which is the value of the Y variable when all X variables are equal to zero.
54
What role do regression coefficients play in a multiple regression equation?
Regression coefficients (bi) quantify the effect of each predictor (Xi) on the outcome variable (Y).
55
Can a multiple regression model have more than one predictor?
Yes, a multiple regression model can include multiple predictors.
56
What is the significance of the intercept in regression analysis?
The intercept indicates the expected value of the dependent variable when all independent variables are zero.
57
General Linear Model
A statistical framework that encapsulates multiple regression, involving the relationship between a dependent variable and multiple independent variables.
58
Multiple Regression
A statistical technique used to model the relationship between one dependent variable and two or more independent variables.
59
A value that represents the effect of a unit change in a predictor variable on the dependent variable in a regression analysis.
Regression Coefficient
60
The point at which the regression line crosses the Y-axis, indicating the expected value of the dependent variable when all predictors are zero.
Intercept
61
What dimensionality does multiple regression introduce?
Multiple regression introduces additional dimensions as it includes multiple predictors, visualized as a regression plane.
62
The outcome variable that a researcher aims to predict or explain in a regression analysis.
Dependent Variable
63
A variable that is hypothesized to influence or predict the dependent variable in a regression analysis.
Independent Variable
64
How is the regression plane defined in a multiple regression model?
The regression plane in a multiple regression model is defined by the intercept and regression coefficients of the predictors
65
Predictors
Variables that are used in a regression analysis to predict the outcome of the dependent variable.
66
What remains constant in the structure of multiple regression?
Despite the number of predictors, the model's structure remains linear.
67
A statistical method that models the relationship between two variables by fitting a linear equation to observed data
Linear Regression
68
Why is the General Linear Model considered a foundational aspect of statistical analysis?
The General Linear Model provides a comprehensive framework for analyzing and understanding relationships between variables in various types of data.
69
What are the three main methods of regression?
Hierarchical, Forced Entry, Stepwise.
70
A regression method where all predictors are entered into the model simultaneously, relying on strong theoretical reasons for variable inclusion.
Forced Entry Regression
71
What is the purpose of Hierarchical Regression?
Hierarchical Regression allows the experimenter to decide the order in which known predictors are entered, thereby assessing their unique contributions to the outcome.
72
A data-driven approach where predictors are selected based on their semi-partial correlation with the outcome, using mathematical criteria rather than theory.
Stepwise Regression
73
Why is Hierarchical Regression considered the best method?
It is based on theory testing and allows the unique predictive influence of new variables to be assessed while holding known predictors constant.
74
Variables that can take on nominal values, such as different species in a study, which can be recoded for mathematical analysis in regression.
Categorical Predictors
75
What is a major drawback of Stepwise Regression?
It may depend on slight numerical differences in semi-partial correlations, leading to significant theoretical implications.
76
A statistical measure that indicates the unique contribution of a predictor to the outcome variable after controlling for other predictors.
Semi-partial Correlation
77
How does a researcher determine the order of variables in Hierarchical Regression?
The researcher uses their theoretical understanding based on past research to decide which predictors to enter first.
78
The portion of variance in the outcome variable that is accounted for by a specific predictor after considering other predictors in the model.
Unique Variance Explained
79
What is the first step in Stepwise Regression when using SPSS?
SPSS looks for the predictor that can explain the most variance in the outcome variable.
80
A variable that has two possible levels, such as having a characteristic versus not having it, which can be used in regression analysis.
Dichotomous Variable
81
What is a critical consideration when using Forced Entry Regression?
The importance of having strong theoretical justifications for the inclusion of specific variables in the model.
82
The process of using statistical methods, like Stepwise Regression, primarily to discover relationships in the data without strong theoretical backing.
Data Exploration
83
What does it mean to recode categorical data for regression
It involves transforming nominal variables into numerical form that can be analyzed mathematically in regression analyses.
84
What is the main goal of generalizing a sample model in multiple regression?
The main goal is to generalize the findings from the sample model to the entire target population.
85
The process of applying findings from a sample model to the entire population, provided that certain assumptions are met.
Generalisation
86
What type of outcome variable must be present in multiple regression analysis?
The outcome must be continuous.
87
A condition where predictors in a regression model must display variance; otherwise, no estimation can occur.
Non-Zero Variance
88
What is the assumption regarding the linearity in multiple regression?
The modeled relationship must be linear; it should not be curvilinear.
89
The assumption that all values of the outcome variable should come from different individuals.
Independence
90
What does multicollinearity refer to in multiple regression?
Multicollinearity exists when predictors are highly correlated with each other.
91
The assumption that the variance of the error term should remain constant for each value of the predictors.
Homoscedasticity
92
What does Cook's Distance measure in the context of multiple regression?
Cook's Distance measures the influence of a single case on the overall model.
93
The residuals that have been transformed into Z-scores; typically, 95% should lie between ±2 in a normal distribution.
Standardised Residuals
94
How can we identify potential outliers in standardized residuals?
Any case with an absolute standardized residual value of 3 or more is considered likely to be an outlier.
95
Cases that have a disproportionate impact on the overall fit of the regression model, as measured by metrics like Cook's Distance.
Influential Cases
96
What should you check regarding the multivariate outliers in regression?
Look for extreme combinations of scores across multiple variables, not just single variable outliers.
97
A measure used to identify multivariate outliers based on the distance of a case from the mean of a distribution, considering the covariance of the variables
Mahalanobis Distance
98
What is the typical threshold for a case's value in Cook’s Distance to determine influence?
A value greater than 1 indicates that a case is likely to be influential on the model.
99
The differences between observed values and the values predicted by the regression model; they represent the error in prediction.
Residuals
100
What does it mean if predictors exhibit high correlation in a regression model?
It indicates the presence of multicollinearity, which can obstruct the estimation of individual predictor effects.
101
An assumption that the error terms in a regression analysis should be uncorrelated for any pair of observations.
Independent Errors
102
Why is it essential to check for no multicollinearity in multiple regression?
High multicollinearity can inflate standard errors and make it difficult to assess the individual contribution of predictors
103
What is regression analysis used for?
To examine the relationship between one dependent variable and one or more independent variables.
104
What is the difference between simple and multiple regression?
Simple regression involves one independent variable, while multiple regression includes two or more independent variables.
105
What does the regression coefficient (B) represent?
The amount of change in the dependent variable for a one-unit change in the independent variable, holding other variables constant.
106
What is the R-squared (R²) value?
It indicates the proportion of variance in the dependent variable explained by the independent variables.
107
What is multicollinearity, and why is it problematic in regression?
Multicollinearity occurs when independent variables are highly correlated, making it difficult to determine their individual effects on the dependent variable.
108
What assumptions must be met for linear regression?
Linear relationship between variables.
Homoscedasticity (constant variance of errors).
No multicollinearity.
Normally distributed residuals.
Independence of observations.
109
What is the purpose of standardizing variables in regression?
To compare variables with different units and scales by converting them to z-scores, making the regression coefficients comparable.
110
What does a p-value in regression output indicate?
Whether the relationship between the independent variable and the dependent variable is statistically significant.
111
What is the intercept in a regression equation?
The predicted value of the dependent variable when all independent variables are zero.
112
How is a regression line calculated?
Using the equation
𝑌
=
𝑎
+
𝑏
𝑋
+
𝜖
Y=a+bX+ϵ, where
𝑌
Y is the dependent variable,
𝑎
a is the intercept,
𝑏
b is the regression coefficient,
𝑋
X is the independent variable, and
𝜖
ϵ is the error term.
113
What is a residual in regression analysis?
The difference between the observed and predicted values of the dependent variable.
114
What is adjusted R-squared, and how does it differ from R-squared?
Adjusted R-squared accounts for the number of predictors in the model, providing a more accurate measure of model fit.
115
What diagnostic tests can be used to check regression assumptions?
Scatterplots for linearity.
Residual plots for homoscedasticity.
Variance Inflation Factor (VIF) for multicollinearity.
Histogram or Q-Q plot for normality of residuals.
116
What is the purpose of hierarchical regression?
To determine the incremental variance explained by adding new predictors to the model.
117
What is a dummy variable in regression?
A variable coded as 0 or 1 to represent categorical data.
118
What is the difference between standardized and unstandardized coefficients in regression?
Unstandardized coefficients (B): Represent the change in the dependent variable for a one-unit change in the independent variable, measured in the original units.
Standardized coefficients (β): Represent the change in the dependent variable in standard deviations, making them comparable across variables.
119
What is stepwise regression?
A method of adding or removing predictors in a regression model based on statistical criteria, such as p-values or changes in R-squared.
120
What is a regression interaction term?
A variable created by multiplying two predictors to test whether the effect of one predictor on the dependent variable depends on the level of another predictor.
121
How do you test for multicollinearity in regression?
Use the Variance Inflation Factor (VIF):
VIF > 10 suggests high multicollinearity.
Tolerance < 0.1 also indicates multicollinearity.
122
What is logistic regression, and how is it different from linear regression?
Logistic regression: Used for a binary dependent variable (e.g., yes/no, 0/1).
The outcome is predicted as a probability using a logistic function, not a straight line.
123
What is the Akaike Information Criterion (AIC) in regression?
A metric used to compare models, balancing goodness-of-fit and model complexity. Lower AIC values indicate a better model.
124
What is overfitting in regression models?
When a model performs well on the training data but poorly on new, unseen data due to being overly complex.
125
How do you interpret an odds ratio in logistic regression?
The odds ratio indicates the change in odds of the outcome occurring for a one-unit increase in the predictor variable.
126
What is the difference between hierarchical and stepwise regression?
Hierarchical regression: Predictors are added in a specific, theory-driven order.
Stepwise regression: Predictors are added/removed based on statistical criteria without theoretical guidance.
127
What is cross-validation in regression?
A technique to evaluate the predictive performance of a model by splitting the dataset into training and testing subsets.
128
How do you interpret a negative coefficient in regression?
A negative coefficient indicates that as the predictor variable increases, the dependent variable decreases.
129
What is the Durbin-Watson statistic?
A test statistic used to detect autocorrelation (independence) in the residuals of a regression model. Values close to 2 indicate no autocorrelation.
130
What is ridge regression?
A type of regression that adds a penalty term to the regression equation to reduce multicollinearity and prevent overfitting.
131
When would you use a polynomial regression model?
When the relationship between the independent and dependent variables is non-linear.
132
What is bootstrapping in regression analysis?
resampling method used to estimate the sampling distribution of a statistic and to improve the reliability of regression estimates.
133
What is the purpose of interaction plots in regression?
To visualize how the relationship between one predictor and the dependent variable changes at different levels of another predictor.
134
What is the F-statistic in regression?
A measure of the overall significance of the regression model, testing whether the predictors collectively explain a significant portion of the variance in the dependent variable.
135
What does a confidence interval for a regression coefficient tell you?
The range within which the true value of the coefficient is likely to fall, with a specified level of confidence (e.g., 95%).
136
What is heteroscedasticity, and how can it be detected?
Unequal variance of residuals across levels of a predictor. It can be detected using residual plots or statistical tests like the Breusch-Pagan test.
137
How do you handle missing data in regression analysis?
Listwise deletion (removing rows with missing data).
Imputation (filling in missing values with mean, median, or model-based estimates).
Multiple imputation (generating multiple plausible values for missing data).
138
