Lecture 8

Question 1

What are Randomization tests?

Answer 1

Permutation tests; exact tests; test that enumerates ALL of the possible outcomes that could occur in some reference besides the outcome that was actually observed.

Question 2

What are 2 characteristics of a Randomization test?

Answer 2

1. Small data IS NOT poor data.

2. If data sets are too large for exact tests, then typically MC methods or a classical parametric test will be used.

Question 3

Name 2 characteristics of Fisher’s exact test.

Answer 3

1. It had a one-sided Alternative Hypothesis.

2. It does not require large samples.

Question 4

What is the Confusion Matrix?

Answer 4

One dichotomous variable represents Reality/truth. The other dichotomous variable represents a measurement/test/claim.

Question 5

Under which 3 conditions will a Hypergeometric distribution exist?

Answer 5

1. Total number of items (population) is fixed.
2. Sample size (number of trials) is a portion of the population.
3. Probability of success changes after each trial.

Question 6

What is overfitting?

Answer 6

Something that occurs when we fit our model “too perfectly” to the data at hand (zero bias), and will thus perform poorly and unpredictably on “new” data, across different samples.

Question 7

To what leads overfitting?

Answer 7

To large variance of parameters when the model is applied to new data/samples. There is zero/low bias and high variance.

Question 8

What can we introduce when the model fits the data “too well”?

Answer 8

A penalty term to introduce bias to the traditional estimation.

Question 9

Which 2 things can avoid overfitting?

Answer 9

1. Implenting the Bias Variance Trade-off.

2. Implenting train-test paradigm principles.

Question 10

What is a Bias Variance Trade-off?

Answer 10

To avoid overfitting we should introduce a little bias in the model built on the data at hand (no perfect fit anymore), it will perform better on “new” data with less variance.

Question 11

Which 2 regularisation techniques are based on the Bias Variance Trade-off principle? What do they do?

Answer 11

1. Ridge regression.
2. Lasso regression.
   These techniques slightly change the slope of the regression line in order to create bias.

Question 12

What is a Tran-Test paradigm?

Answer 12

We should use training data to build the model and test data to assess the model. Both training and testing data are created (from the dataset/sample that we started with) by sophisticated partitioning and using the data.

Question 13

What is Cross-Validation?

Answer 13

It is the most famous technique to implement these ideas, more particularly by sophisticated partitioning the data.