Project Questions

Question 1

What is the purpose of the “set.seed()” function in R?

Answer 1

It ensures that output from randomness will be the same given same inputs.

I.e., omitting the seed function, running the same code will lead to different results every time due to different points of departure

Question 2

What is the utility of the “caret” package?

Question 3

What is the utility of the “corrplot” package?

Question 4

What is the utility of the “tidyverse” package?

Question 5

What is the utility of the “pROC” package?

Question 6

What is the utility of the “rpart.plot” package?

Question 7

What is a factor?

Answer 7

In R, a factor is a categorical variable that represents distinct levels or categories

Question 8

What is an integer?

Answer 8

An integer is a whole number that does not have any fractional or decimal part.

Question 9

What is a character?

Answer 9

A character is a data type in programming that represents individual letters, numbers, symbols, or spaces

Question 10

To incorporate interaction effect with one or both variables being categorical, dummy variables are used for this purpose. Why?

Answer 10

In logistic regression, Dummy variables provide a way to represent categorical variables numerically. Each dummy variable represents one category of the categorical variable and takes on values of 0 or 1, indicating the absence or presence of the category. If absent, the entire term becomes 0. If present, the model multiples the corresponding coefficient with 1 (present).

Question 11

Which of the following additional use cases does BDA have in hospitality?
A) facilitates service innovation
B) insights into customer satisfaction through e.g., big data text analysis of customer reviews
C) creating client profiles and enhance customer relationship management
D) all of the abovw

Answer 11

D) All of the above

Question 12

What is the goal of data transformation? It involves modifying the structure or content of a dataset.

Answer 12

The goal is to ensure appropriate fit between the type of data and chosen statistical method

Question 13

Which data transformation steps did you perform?

Answer 13

1) Transformation from character to factor
2) Transformation from integer to factor
3) Combining #kids and #babies

Question 14

Why did you transform characters to factors?

Answer 14

Factors allows the model to interpret and utilise categorical variables efficiently by providing a finite number of options that the value can take

Question 15

Why did you transform integers to factors?

Answer 15

Factors allows the model to distinguish between numerical variables that represent categories and numerical variables that represent continuous quantities

Question 16

Give me an example of which integer you you transformed into a factor, and explain why it made sense

Answer 16

Is_repeat_guest and is_cancelled: in reality, these variable can take one of two values: yes (1) or no (0). Thus, it made sense to transform them from integers to factors to ensure that the model interpreted them as categorical variables

Question 17

Give me an example of which character you you transformed into a factor, and explain why it made sense

Answer 17

Meal: there are 4 options of meal types, all each by two capital letters and came as characters. We wanted the model to interpret the variable as categorical to allow it to map any associations between this and the output.

Question 18

As part of data cleaning, you removed some observations based on their deposit type. Elaborate on this

Answer 18

Removed the observations with deposit type = non-refundable: EDA showed that 99% of these bookings were cancelled - highly counter-intuitive

Question 19

As part of data cleaning, you removed observations with no adults recorded. Why? Is this necessarily an error?

Answer 19

Not necessarily an error. However, the rooms with no adults were expected to be perfectly associated with another booking. Thus, these rooms are not representative as a singular observation to be considered in the model

Question 20

Why did you keep the remaining two deposit types rather than deleting the entire variable altogether?

Answer 20

The observations with refundable deposit and no deposit paid behaved intuitively, and no abnormality was detected. E.g., cancellation rate is slightly higher for no deposits, which makes sense

Question 21

Why did you remove observations with “undefined” meal and distribution channel? Were they missing completely at random?

Answer 21

This resulted in deletion of ~1000 rows, which we evaluated not to impact the model significantly (w. +100k observations remaining after all cleaning)
MCAR:

Question 22

How did you seek to tackle the bias-variance trade-off?

Answer 22

To ensure that at right balance was striked between generalisation (bias) and overfitting (variance), we partitioned the the data into a training set (60%) and test set (40%). Then, we applied 5-fold cross-validation

Question 23

What does it mean that you applied 5-fold cross validation?

Answer 23

5-fold cross validation is done within the TRAINING set, where the data is divided into 5 portions (folds). In each fold, 4/5 of the training data is used for training, and 1 is used for testing. This is repeated 5 times until all training data has been used for training AND testing

Question 24

Why did you choose 5 folds?

Answer 24

It came down to a trade-off between the degree of cross-validation and computational time - each fold takes a long time.
Meanwhile, with the significant size of the dataset, each fold still contains a large amount of data - suggesting that the bias-variance tradeoff was sufficiently handled

Question 25

What did you do to deal with dataset imbalance?

Answer 25

We applied stratified random sampling, which ensures that the distribution between 0 and 1 are similar in the test and training sets. This mitigates the risk of drawing a “lucky” or “unlucky” test/train set, affecting the predictive performance

Question 26

How does the LR model estimate coefficients?

Answer 26

By maximum likelihood estimation, where the model applies an iterative process to estimate the coefficients that maximises the likelihood of the given outcome being equal to the observed outcome

Question 27

Which of the following statements are TRUE about DT?
A) it operates by recursive partitioning a dataset into subsets based on the values and input features, which creates a hierarchical structure of decision nodes
B) Each intermediary node represents a decision point, wherein a given feature is evaluated
C) Each terminal leaf node reflects a final outcome or prediction
D) The recursive partitioning divides the dataset into subsets that minimises entropy and Gini impurity
E) all of the above

Answer 27

E) all of the above

Question 28

Which of the following options are advantages of LR relative to DT?
A) if offers a precise interpretation of the association between each variable and the output through coefficients
B) it is more flexible in adapting to non-linear relations
C) it provides a visual, intuitive mapping of the decision process, revealing the interplay of various predictors
D) all of the above

Answer 28

LR > DT in following aspects
A) if offers a precise interpretation of the association between each variable and the output through coefficients

DT < LR
B) it is more flexible in adapting to non-linear relations
C) it provides a visual, intuitive mapping of the decision process, revealing the interplay of various predictors

Question 29

Explain the process employed in your selection of variables in the mode.

How did you determine the importance of numerical variables?

How did you determine the importance of
categorical variables?

How did you determine the importance of
different interactions?

Answer 29

For numerical variables, the importance of each predictors was evaluated based on their respective correlation with cancellation rate

For categorical variables, the importance of each predictor was evaluated through graphical illustrations to illuminate any meaningful associations with cancellation rate

For interactions, a series of heatmaps were constructed to explore any association across categorical predictors. For numerical variables, those with highest correlation were chosen

Question 30

You excluded following explanatory variables. Why?
A) room type
B) agent ID
C) hotel type
D) reservation status
E) company
F) days_waiting_list and arrival_date_day_of_month

Answer 30

A) room type: denoted by letters -> unable to interpret
B) agent ID: denoted by numbers –> unable to interpret
C) hotel type: to enhance generalisability beyond the two hotel types
D) reservation status: for predictive purposes on future bookings, you don’t have this observation prior to expected guest arrival
E) company: 94% were NULL observations
F) days_waiting_list and arrival_date_day_of_month: negligible correlation with cancellation

Question 31

AUC is a threshold independent metric
TRUE/FALSE

Answer 31

TRUE
AUC provides a general evaluation of a model’s performance across its sensitivity and 1-specificity

Question 32

Confusion-matrix-derived metrics are threshold-dependent
TRUE/FALSE

Answer 32

TRUE

Question 33

Proportion of correct predictions in all records is measure by____
A) accuracy
B) sensitivity
C) specificity
D) precision
E) negative predictive value

Answer 33

Proportion of correct predictions in all records is measure by ACCURACY

Question 34

Ability to correctly predict the positive class is measure by____
A) accuracy
B) sensitivity
C) specificity
D) precision
E) negative predictive value

Answer 34

Ability to correctly predict the positive class is measure by SENSITIVITY (=RECALL)

Sensitivity = TP/(TP+FN)

Question 35

Ability to correctly predict the negative class is measure by____
A) accuracy
B) sensitivity
C) specificity
D) precision
E) negative predictive value

Answer 35

Ability to correctly predict the negative class is measure by SPECIFICITY

Specificity = TN/(TN+FP)

Question 36

Proportion of correct predictions in those the classifier predicted as positives is measure by_____
A) accuracy
B) sensitivity
C) specificity
D) precision
E) negative predictive value

Answer 36

Proportion of correct predictions in those the classifier predicted as positives is measure by PRECISION

Precision = TP/(TP+FP)

Question 37

Proportion of correct predictions in those the classifier predicted as negatives is measure by_____
A) accuracy
B) sensitivity
C) specificity
D) precision
E) negative predictive value

Answer 37

Proportion of correct predictions in those the classifier predicted as negatives is measure by NEGATIVE PREDICTIVE VALUE

NPV= TN/(FN+TN)

Question 38

In the context of the core business objective being to improve operational performance, prioritising a slight overestimation of cancellations (with some false positives) is preferable to risking revenue loss due to underestimating cancellations (false negatives). Therefore, the primary focus in the model comparison centres on _____ vs. _____

Answer 38

In the context of the core business objective being to improve operational performance, prioritising a slight overestimation of cancellations (with some false positives) is preferable to risking revenue loss due to underestimating cancellations (false negatives). Therefore, the primary focus in the model comparison centres on SENSITIVITY vs. PRECISION

Question 39

½Hyperparameters play a crucial role in the domain of machine learning, setting themselves apart from model parameters as they operate _______ to the learning process.
Fill in the bland

Answer 39

Hyperparameters play a crucial role in the domain of machine learning, setting themselves apart from model parameters as they operate EXTERNALLY to the learning process.

Question 40

What role does the cost complexity parameter play in decision tree modeling, particularly in relation to overfitting?

Answer 40

The cost complexity parameter in decision tree modeling balances the trade-off between the complexity of the tree and its predictive performance, preventing overfitting

Question 41

What is the purpose of the tune-length hyperparameter in the systematic tuning process for pruning decision trees?

Answer 41

The tune-length defines the number of complexity parameters to be evaluated, and in this study, a tune-length of 15 is chosen to strike a balance between performance, risk of overfitting, and computational demand.

Question 42

Why did you set the threshold at below 0.5?
A) asymmetric costs associated to overprediction vs. underprediction
B) the study defines a preference for slight overestimation of cancellations, leading to lower threshold choice
C) to ensure that the model overestimates cancellations rather than underestimating it
D) all of the above

Answer 42

D) all of the above

Question 43

You set different thresholds for each of the model versions. Why?
A) to ensure an unbiased evaluation of predictive performance across models that allows for comparability
B) each model possesses unique characteristics that impact its performance at various threshold levels
C) it ensures a more impartial performance assessment of each model rather than setting an “unfair” threshold for some of the models
D) all of the above

Answer 43

D) all of the above

Question 44

How did you set the individual thresholds?

Answer 44

An iterative process. Every model starts with a threshold of 0.25. Then, it is gradually adjusted upward using two decimal points to a number that maximises the accuracy of the model BUT under the condition that it must predict more false positives than false negatives

Question 45

A Sensitivity of 0.59 in Model 3 indicates
that the model correctly predicts 59.% of the actual positives, whereas a Precision of 0.57 indicates that 57% of the positive predictions are actually positive.

TRUE/FALSE

Answer 45

TRUE

Question 46

What does a no information rate of 0.7151 mean?

Answer 46

The NIR tells us the accuracy (predictive performance) of a naive benchmark.
The naive consistently predicts the outcome based on the largest proportion of the output being positive or negative.

I.e., if NIR is 0.71, simply means that 29% of the dataset has a is cancelled outcome (1). By predicting “not cancelled” every time, the naive model will then be right in 71% of the time.

Question 47

Which of the DT models was optimal? Why?

Answer 47

DT 4 was rendered the most optimal DT model.
The accuracy of 3 and 4 are the only two that exceeds NIR. DT 3 had higher precision, but DT 4 had higher sensitivit –> DT 4 is best based on business goal (preference for overprediction at the expense of higher false positives)

Question 48

Which of the LR models was optimal? Why?

Answer 48

LR 3 was rendered the most optimal RL model.
Both sensitivity and precision increase from LR 1-3.
Despite Precision being highest for model 4, sensitivity is highest for LR 3 –> LR 3 best based on business goal (preference for overprediction at the expense of higher false positives)

Question 49

Best DT model (4) outperforms the best LR model (3) across all evaluated metrics.
This leads to the conclusion that this DT emerges as the better choice based
on predictive performance - both sensitivity and precision is higher for DT 4 than LR 3

TRUE/FALSE

Answer 49

TRUE

Question 50

Which advantages does LR have over DT wrt. practical adoption?

Answer 50

1) Lower computational demand and faster update
2) Easier to interpret with coefficients indicating the association to predictive features - easier to communicate

Question 51

LR provides a percentage likelihood for a particular booking being cancelled (probability) - do DTs give the same output?

Answer 51

No - decision trees tells you whether the given booking is most likely to be cancelled or not, not the distinct probability. It does tell you however that based on the training set, x% of the total data was captured in a given leaf node, and 0.Y of those observations turned out to be cancelled/not cancelled

Question 52

What is the business objective of the study?

Answer 52

Based on booking cancellation prediction, hotels can leverage the insight to increase operational performance through more informed revenue management initiatives (overbooking) and minimise cost base (staffing)