Machine Learning with Viya® 3.4® Lesson 6: Model Assessment and Deployment Flashcards
How would you add a challenger model to a pipeline comparison in Model Studio?
Select the model in its pipeline.
Which assessment measure should be used to determine the champion model for predicting an interval target?
Average Square Error
How would you identify which model has the best classification accuracy using C-statistic values?
The model with highest C-statistic value has the best performance.
Which dataset partition will be used to select the champion model when using the default settings in Model Studio?
Validation
What type of data is used during champion-challenger testing to compare the performance of the currently deployed model and a challenger model during the model deployment phase?
Champion-challenger testing compares performance on historic data during model deployment.
What are the primary considerations for choosing an appropriate model selection statistic?
- business needs
- the prediction type
- the measurement level
The confusion matrix is the foundation of which assessment plot?
the ROC chart
A confusion matrix helps you classify which type of target?
Binary
Which validation method would you recommend for a small dataset?
Cross-validation
A cumulative lift chart shows that a machine learning model has a lift of 2.6 at a depth of 10%. What does this mean?
For the top 10% of cases, the machine learning model captures 2.6 times more primary outcome cases than a random model.
What model fit statistics are recommended for a decision prediction?
accuracy or misclassification
KS
What are two commonly used performance statistics for estimate predictions?
Schwarz’s Bayesian Criterion (SBC)
Weighted Square Error?
What assessment measure would you use to assess the probability of a customer responding to a targeted ad campaign?
the Gains chart
What is another term for gains chart?
Cumulative Percentile Hits chart
What is another term for a Cumulative Lift chart?
a Gains chart
What is a cumulative lift chart?
A lift chart indicates how well the model did as compared to no model. The lift is the ratio between the result predicted by the model and the result using no model.
cumulative lift plotted as a percent on the vertical axis
What’s the calculation for the lift for a given percentile when evaluating a Cumulative Captured Response (Gains) Chart?
Divide the Model Response Rate by the Random Response Rate:
Lift = (P,M) = CPH(P,M) / P
where P is a given percentile
What are two things you can investigate in an ICE plot?
Subgroups and interactions among model variables.
What do level differences in an ICE plot suggest?
group effects
What does an intersecting slope in an ICE plot indicate?
Interactions between the plot variable and one or more additional model variables
What are two things to look for in an ICE plot?
- intersecting slopes
- level differences
Which machine learning model is the easiest to interpret?
Decision trees are highly interpretable because they are based on English rules, which are rules that use Boolean logic.
Which dataset partition assists in comparing possible models?
Validation
