Predictive Analytics Problem Definition (10-20%)

Question 1

Define descriptive analytics.

Answer 1

Focuses on insights from the past and answers the question “what happened?”

Question 2

Define predictive analytics.

Answer 2

Focuses on future insights and addresses “what will happen next?”

Question 3

Define prescriptive analytics.

Answer 3

Suggests decision options ie. “what would happen if i do this?” or “what is the best course of action?”

Question 4

What are the characteristics of predictive modeling problems?

Answer 4

1. Identify and define business issues that needs to be addressed
   
   • clarify the business issue
   • develop hypotheses, a testable hypothesis to guide the project
   • an ability to assess the outcome, gain clarity, and measurable key
     performance indicators (KPIs)
2. Address the issue with a few well-defined questions - what you need, what’s the target outcome and how well it is representing the problem, how will the model performance be evaluated
3. Having good/useful data that can answer these questions - high quality, permission to use data. amd identify info gaps
4. Knowing the predictions will drive actions or increase understanding - is the approach practical, understandable, adn implementable
5. Knowing the approach is better than the existing one - easier approach
6. Monitor and update the model as new data comes in

Question 5

What is a Field Test?

Answer 5

When the model is implemented in the exact way it will be used, but it is not yet used for decision making, only observed on its potential to help users make good decisions. Particularly when the problem, data, or type of model is new

Question 6

Explain the Bia-Variance Trade-off.

Answer 6

• Bias: the expected loss arising from the model not being complex/flexible enough to capture the underlying signal
• Variance: the expected loss arising from the model being too complex and overfitting to a specific instance of the data
• High bias means that the model won’t be accurate because it doesn’t have the capacity to capture the signal in the data, whereas high variance means that the model won’t be accurate because it overfit to the data it was trained on and thus won’t generalize well to new, unseen data

Question 7

For y=f(x) + Ɛ, what are the assumptions for the error term (Ɛ)?

Answer 7

1. The errors are independent - with observed data, we know the form and parameters of distribution of Ɛ, but for a new observation, the past tells us nothing about the value of Ɛ that will go with it.
2. The errors have a mean of zero - if f(x) is known, on average, predictions will be correct.
3. The errors have a constant variance - variance doesn’t depend on x.
4. The errors have a Gaussian (normal) distribution.

Question 8

Define Loss Function.

Answer 8

A function of two variables - the prediction and a single observed new value that measures error ie. squared error loss function.

Note: the loss is a random variable since many items in the expression are random variables. Because the loss function is a random variable, it may not be useful as a measure of model quality so we typically look at its expected value (average loss over predictions we might make and over all training sets we might use).

Question 9

Define bias-variance decomposition.

Answer 9

A useful theoretical window into how model complexity affects the quality of models and helps demonstrate the important insight that more complex models aren’t always better models.