Model Selection

Question 1

What is the traditional strategy?

Answer 1

No model selection. You run one model, and assume every continuous variable has a linear relationship. Include all interactions

Question 2

Problems with traditional strategy?

Answer 2

No multicollinearity assessed, overfiting data and not all relationships are linear.

Question 3

Historical strategy

Answer 3

Based on past models that ran just fine. One advantage is that with this, you are able to compare data from past models because you include same predictors and same variables

Question 4

Problems with historical designs

Answer 4

You may ignore variables that were not initially considered. Design may be insufficient eventually

Question 5

The exploratory approach

Answer 5

Here a wide range of models are run and then usually the one that worked is the one reported.

Question 6

Problems with the exploratory strategy?

Answer 6

A lot of experimenter degrees of freedom. Results could be too flexible and could lead to replication problems. Use a more conservative model selection approach (AIC and BIC are always safe bets)

Question 7

Don’t do this in calling analyses

Answer 7

Call exploratory analysis confirmatory

Question 8

Theoretically driven approach

Answer 8

Only a limited number of models are run here, 3 tops has to be theoretically driven. Is very systematic and highly localized. Less prone to overfitting

Question 9

Problems with theoretically driven?

Answer 9

Can miss the analysis of variables already collected. Is okay to include both models just report everything. But for confirming that you should run an additional experiment and run it as confirmatory.

Question 10

What is the mixed strategy?

Answer 10

Report which decisions models were exploratory, theoretically driven, etc. in its appropriate sections. Can segment the analysis into different parts, lessens the overfitting issue

Question 11

Disadvantages of mixed strategy?

Answer 11

Researcher degrees of freedom

Question 12

P- focused strategy

Answer 12

Choose the model that has your key variables significant.

Question 13

Interocular Test

Answer 13

You look at the results, and it hits you between the eyes. Plot your data,a which says a lot. Statistics are more like an afterthought. Worry if the graph shows no differences and the analysis is telling you otherwise

Question 14

What happens with the R square

Answer 14

The proportion of variance accounted for. To calculate this you have the mean somewhere in the formula. Mean is a measure of central tendency in a NORMAL DISTRIBUTION. Which is why in models where this is not met, R square is not a good model fit indicator

Question 15

RMSE

Answer 15

Is the square root of the variance squared or the ss formula. The squared errors are averaged. In other words this is a measure of the average distance any point is from the mean. But this does not by any means deal with complexity

Question 16

Mallow CP

Answer 16

In JMP, when you try to do a stepwise regression. Another inde,x butit is not common

Question 17

The AIC and BIC

Answer 17

The Akaike and Bayesian Information criteria. One penalizes for complexity. BIC favors simpler models over complex models

Question 18

Likelihood Ratios

Answer 18

When you are comparing one model to the other we talk about how much likely one model is to fit the data versus another model. It gives a ratio because… is 10 times more likely, is 35 times more likely.

Question 19

AIC / BIC / Likelihood ratios

Answer 19

Since AIC and BIC are based on Log likelihoods, you can include them in the model fit inspection, or you can transform AICs and BICs to the log likelihood scale to make them more interpretable

Question 20

Cross validation

Answer 20

Uses two types of data. Sample data to develop a model. Test the dataset to see if that model also applies to more data. This is an empirical way to determine if the model is going to perform well out of that sample. If it does, it cross-validates.

Question 21

LOO

Answer 21

Leave-one-out cross-validation. It holds one line out of your data out and after the model is done and created, see how well it does with that one out, and repeats for every single row in your dataset

Question 22

Bootstrapping

Answer 22

Sampling with replacement, and treat the distribution as the population. It gives you an empirically estimated distribution Some people use it to make up for a small sample size.

Question 23

When do you use bootstrapping

Answer 23

When you have bad, very bad distributional properties that there is no statistical test to determine the differences. This happens in something called interportal ranges.

Question 24

Where could you use this?

Answer 24

In multilevel repeated meassures designs when you have imbalances but slight ones like in each condition some values were missing but does not apply if the whole level is missing

Question 25

Another application of Bootstrapping

Answer 25

You can bootstrap your R square values which is great because now you add the uncertainty estimate of the R square to make more informed decisions

Question 26

Weakness of R-squared

Answer 26

It isn’t really appropriate when the errors aren’t normally distributed. R-squared and RMSE do not control for model flexibility

Question 27

Weaknesses of AIC and BIC

Answer 27

Can only compare models using the same scale in both models (log-log) Very sensitive to missing data for comparison, be careful if a variable has missing data, as it makes the model no longer comparable. You cannot compare a not-transformed vs a transformed model

Question 28

Overfitting

Answer 28

Too small sample sizes. Robustness over p-values.

Question 29

Bias-variance tradeoff

Answer 29

With cross valitadion we are stimating how big the gap is. Too small an be underfitting, too large can be overfitting

Question 30

What is bias?

Answer 30

Model does not fit the training data... makes sense

Question 31

What is variance?

Answer 31

The size of the difference between testing and training systems

Question 32

Ideal model

Answer 32

The one that minimizes both variance and bias

Question 33

Cross validation vs BIC, AIC

Answer 33

When you don't have enough data you can opt for a AIC/BIC

Question 34

Statistical comparissons

Answer 34

Statistical comparisson of both models. chi square the most common and tests for fit as a function of increased flexibility. But models have to be nested within the other

Question 35

What does nested mean?

Answer 35

A model is nested when it includes a subset of the predictors that is in the model one

Question 36

Check model is a lifesaver

Answer 36

Because when AICs are not okay because the two scales of y are different, when your models are not nested, when you don't have enough to cross validate... Check model is enough to compare between models