A-kassen

Question 1

What is a label?

Answer 1

What we want to predict.

Question 2

What is the label in the paper?

Answer 2

Churn/loyal members.

Question 3

Why did you use Jupiter?

Answer 3

Jupiter is like a notebook that comprise different code languages and python code is one of the languages. This is a powerful tool that enabled us to do the data mining through specify, program our own process steps.

Question 4

What was the main purpose of the data preparation?

Answer 4

To take a critical look on the dataset. Make it so clean and small as possible because this gives more prediction power.

Question 5

What is the first step in the data preparation?

Answer 5

Exploration phase

Question 6

What does the exploration phase entail?

Answer 6

Three-step process. Missing data, Data types, Outliers

Question 7

Why was it important to make predictions about customers intentions or tendencies to churn?

Answer 7

1. Enable efficiency in regards to planning of retaining customers - for example through specific promotional activities
2. Comparison between predictions and actual results allows to spot meaningful indicators, useful for improving performance

Question 8

How is predictive modelling and data mining useful for companies?

Answer 8

Instruments for improving the decision making process within companies.
The more accurate and timely the knowledge is, the increases the likeliness for the company to improve their business performance and industry position.

Question 9

Why do companies use machine learning algorithms and big data?

Answer 9

To reduce uncertainty of for example unforeseen market changes or customers behaviour.

Question 10

What did you do in your paper?

Answer 10

We analysed customer data from Akademikernes A-kasse in an effort to create a predictive model of which attributes are applicable when customers chose to churn. However the model we created does not predict how likely the customer is to churn but gives a binary churn/no churn result.

Question 11

What do you mean about a binary churn/no churn result?

Answer 11

It means that we classified the two elements of a given set, churn/no churn into two on the basis of a classification rule. And we classified them on the basis of attribute selection in the predictive model.

Question 12

What could be the next step for Akademikernes A-kasse?

Answer 12

To use regression to calculate the likelihood of churn within the predicted group of churners we have identified. A heat map for example, could visualise which customers and what attributes where more likely to churn in the future.

Question 13

Why did you chose to use the CRISP model in your paper?

Answer 13

We decided to use it as a guiding principle for the paper, as we saw correlations of what the model present and what we wanted to do in our paper. However we were aware that the model is an exploratory tool that emphasise approaches and strategy rather than software design.

Question 14

What are the six stages in the data mining process that the CRISP model defines?

Answer 14

Business understanding, data understanding, data preparation, modeling, evaluation, and deployment.

Question 15

What does the business understanding stage present?

Answer 15

Concern the practical business goals that the organisation wants to achieve. This goal were converted into a problem that the data mining seek to solve. AK seek to uncover the reasoning behind the churn of their members and in turn ultimately reduce the number of churners.

Question 16

Could you mention some critical perceptions of your paper?

Answer 16

To begin with, we thought we would use decision trees as they are high quality models generate simple rules and make it easy to understand the impact.
Decision trees will be very complex and big if there are many attributes so we were really harsh in the attribute selection as we wanted few attributes. If we did not have this initial plan the number of attributes might be higher .

Question 17

How can AK use your results?

Answer 17

Knowledge of the customers. Attributes that are linked previous churners, and therefore these tendencies can be used to spot current members that are linked to the same attributes.

Question 18

What is data mining?

Answer 18

A process used for discovering meaningful relationships, trends, and patterns between large amounts of data collected in a dataset.

Question 19

What is clustering?

Answer 19

Clustering is gathering groups of people with common attributes in a K number of clusters. Clustering (also called unsupervised learning) is the process of dividing a dataset into groups such that the members of each group are as similar (close) as possible to one another, and different groups are as dissimilar (far) as possible from one another.

Question 20

How can clustering create value for AK?

Answer 20

Goal is to group together similar instances using some metric of similarity - so create groupings where the members of a given group are similar to each other. For example group similar customers together and design different campaigns.
It is light classification but the groupings are not predefined.
Could find a way to group similar customers together. May or may not relate to the churn question.

Question 21

What can you do to present results from data mining as informative as possible?

Answer 21

You can sacrifice details - subjective decision.
Switching from ROC (receiver operating characteristics) with AUC (area under curve) and lift curves/cumulative response curve.

ROC curves are not the most intuitive visualisation for many business stakeholders who really ought to understand the results. One of the most common examples of the use of an alternate visualisation is the use of the “cumulative response curve, which is more intuitive

Question 22

What are lift curves?

Answer 22

Visualisation framework that might not have all of the nice properties of ROC curves, but are more intuitive. So, conceptually as we move down the list of instances ranked by the model, we target increasingly larger proportions of all the instances.

Question 23

Why is your data affected by selection bias?

Answer 23

Only access to Akademikernes A-kasse dataset, and not other A-kasse organisations. Therefore the dataset does only comprise information about AK´s members and is not a representable representation of the entire population.

Question 24

What makes machine learning algorithms supervised?

Answer 24

We know the target class, and more specifically what we are looking at. The opposite is unsupervised which does not provide a purpose or target information.

Question 25

Why did you chose to solely look at supervised machine learning algorithms?

Answer 25

We knew the target class, and what we wanted to look at which is churner/no churner

Question 26

How do you measure the performance of the classification?

Answer 26

Through classification accuracy. However accuracy does not capture what is important for the problem at hand.

Question 27

Why did you chose classification?

Answer 27

As we had a label, classification was appropriate as classification involves selecting which, out of a set of labels should be assigned to some data, according to some spot meaningful indicators.

Question 28

What is binary classification?

Answer 28

It means that we are working with two options.

Question 29

Why did you disregard some attributes?

Answer 29

Because we found some of them to be irrelevant to predictions related to the target "will churn" and "will not churn". We observed distinctiveness degree in which the attributes we have chosen affect the target in the attribute selection section of the paper.

Question 30

Why is it not enough to look at the accuracy?

Answer 30

Because a dataset can be imbalances, which was the case for our dataset, and in this case the accuracy can be misleading. In our example we had 71,5% non-churners.

Question 31

What did you do to cope with the accuracy "problem"?

Answer 31

We decided to use stratified cross validation in order to evaluate how generalisable our model was.

Question 32

What is stratified cross validation?

Answer 32

Stratified cross validation is a way of splitting or portion the entire dataset into "k" bin of equal size to get a fair share of data points in both training data and test data.

Question 33

Why is it efficient to use stratified cross validation?

Answer 33

Utilise the dataset more. By using stratified cross validation, you will get more out of the training and test data, that is, the best validation and learning results as possible. If we were to only use training data (hold out data), we would only train on parts of the dataset and leave the rest behind for test.

Question 34

Why is your dataset imbalanced?

Answer 34

Because the class churn_loyal has a high concentration 71,5% - and as the concentration is not equal it can be argued that it is imbalanced

Question 35

What do you need to take into consideration when your dataset is imbalance?

Answer 35

It is a very normal thing - but we knew that we need to be aware that the accuracy could be misleading and therefore we chose not to focus much upon the accuracy.

Question 36

What did you do when you were checking data types?

Answer 36

We ensured that the machine learning algorithms were able to get the right data types.

Question 37

Why did we look for outliers?

Answer 37

To make the dataset as clean as possible, and then we could either remove or adjust these.

Question 38

Why was your dataset poor?

Answer 38

We spotted a lot of duplicates, null-values, inconsistent casing, and danish special characters.

Question 39

Why did we look for outliers?

Answer 39

To make the dataset as clean as possible, and then we could either remove or adjust these.

Question 40

How did you drop values?

Answer 40

Defined a rule to decide wether and attribute brings value into predicting churn/no churn.

Question 41

How did you drop values?

Answer 41

Used coding to create som rules to decide wether and attribute brings value into predicting churn/no churn. For example we cleaned some of the specific instances, and categorised education names and commune into broader groups.

Question 42

What is an example of an unique value?

Answer 42

Education name which has values as for example Ha. IT and Ha.it.

Question 43

Why did we have to clean each attribute in some cases?

Answer 43

Because the machine cannot distinguish between for example Ha.IT and Ha.it. After doing this unique values were dropped from 841 to 796.

Question 44

Explain how you handled registration date and churn month.

Answer 44

It was not possible to use churn month because it was null for most/all members in the dataset, since they have not churned - and it is not possible to make calculating time range of null values. We converted the registration date and churn month into a new attribute: days as member. And we could calculate how many days the instance had been a member. The data is only representable within the given timeframe but this is easy to change. Recommend to change every day - so AK has an accurate overview.

Question 45

How did you handle outliers?

Answer 45

Introduced a new attribute - years_correct. age*365-days as member. So we calculated the critical range, and if a persons number was below 6570 it meant that they had been member before turning 18 or the data was not correct.

Question 46

What did the general clean up include?

Answer 46

1. Map danish names to machine computer friendly names 2. Fill null values with meaningful values such as other etc 3. Drop low quality instances based on our rule based attribute selection

Question 47

Which five evaluation metrics did you use?

Answer 47

Accuracy, variance and standard deviation, confusion matrix metrics, ROC/AUC, base rate

Question 48

What is accuracy?

Answer 48

A measure of how many instances the model has correctly guessed in total.

Question 49

What is accuracy paradox?

Answer 49

Often we can see this in dataset that are skewed, which is the case for us. There is accuracy paradox when models with lower accuracy are preferred because it is important, on the basis of our business understanding, that who can be churners on the basis of attributes.

Question 50

What is variance and standard deviation?

Answer 50

Tells us how much we can expect our model to vary when adding new datasets. If we see a big variance between each fold we could think that the dataset is not generalisable which is preferred if using the same model on another dataset.

Question 51

What is confusion matrix?

Answer 51

Gives a quick overview of results. Many metrics can be calculated on behalf of the matrix and one of them that we used is TPR or recall.

Question 52

What is confusion matrix?

Answer 52

Gives a quick overview of results. Many metrics can be calculated on behalf of the matrix for example FPR and TPR.

Question 53

Why is base rate used?

Answer 53

For model performance. To set a baseline for the performance of the model.

Question 54

Why is a naive classifier used?

Answer 54

To get a median over the population

Question 55

How do you limit the amount of overfitting?

Answer 55

We used stratified cross validation and fold 5 times so

Question 56

How do you limit the amount of overfitting?

Answer 56

We used stratified cross validation and fold 5 times so we could train each fold and measure the mean and variance, which can be used to estimate the expectation upon future datasets.

Question 57

Why did we chose 5 folds instead of 10?

Answer 57

x

Question 58

Why did we want to be above the random graph which is the diagonal line from 0,0 to 1,1?

Answer 58

If it is below the line the model are not performing well and anything avove is an advantage from being random.

Question 59

What does 0,0 and 1,1 represent?

Answer 59

0,0 means that you never classify an instance as positive. 1,1 means that you classify all as positive.

Question 60

Why is feature engineering used?

Answer 60

Feature engineering can lead to increased performance by refining the dataset with knowledge about the attributes. An example is days as member where we creates a new feature since we saw bigger potential this way. So instead of introducing models that are more complex we refined our dataset and improve performance in that way.

Question 61

What is domain knowledge?

Answer 61

x

Question 62

Why is feature scaling used?

Answer 62

To limit the influence from numeric features with broader ranges. As for example days as member had a very high range this could affect other numeric features. Therefore we used feature scaling to secure that our attributes contribute approximately proportionally.

Question 63

What is a feature scaling method?

Answer 63

Standardscaler which standardized the numeric features

Question 64

What is standardization?

Answer 64

Is done in the dataset to reduce variance in order to not have an imbalance in the estimator influence and dominate the function

Question 65

What is information gain attribute evaluation?

Answer 65

How much gain each attribute provides to the label. The closer to 1 the more information the attribute provides.

Question 66

Why did we run information gain an correlation threshold in comparison?

Answer 66

To choose which attributes are most important in regards to the label we wanted to predict. We defined a rule that would exclude the attributes that did not meet a high enough contribution.

Question 67

What is correlation attribute evaluation?

Answer 67

The correlation coefficient spans from -1 to 1 whereas the coefficient is denoting how correlated a feature is to the label. If the coefficient states 0 the feature is no way correlating to the label meaning no prediction power.

Question 68

Why is stratified k-fold and why is it useful?

Answer 68

Very similar to cross validation, but SFK ensures the class is approximately the same size in each fold. Each of the folds will be tested and trained k number of times.

Question 69

What are the two categories models can be divided into?

Answer 69

Parametric and nonparametric

Question 70

Why did you use both parametric and nonparametric models?

Answer 70

In order to utilise a broader toolset of classifiers on our dataset, it made sense to include methods from both categories.

Question 71

You tested Linear SVC, SVC Poly kernel, SVC RBF kernel and SVC Linear kernel. Why did you use Linear SVC?

Answer 71

The accuracy, TPR and TNR are identical so we followed the advice from scikit learner and used only Linear SVC

Question 72

Why did we use KNN?

Answer 72

To explore where our instances fit in regards to our target value.

Question 73

The goal with KNN is to find neighbours with the shortest distance. How can the distance be calculated?

Answer 73

With euclidean distance

Question 74

Why did we decide K nearest neighbour with k = 4?

Answer 74

Based on error rate. But how? IN regards to recall we chose 5? (67%)

Question 75

Why did you chose to work with KNN instead of Linear SVC?

Answer 75

Because Linear SVC predicted only 14% TPR and KNN predicted 69%

Question 76

What have you achieved through data mining efforts?

Answer 76

Determined which attributes are contributing factor when a member choose to churn.

Question 77

How can our results be implemented in the business?

Answer 77

Our model can look at any given customer in AK and predict with 87% certainty wether the customer will churn or not.

Question 78

What are our recommendations to AK?

Answer 78

Run the model agains customer database every day to identify groups of customer who will stay members but also identify a group that are likely to churn. With this AK can engage in several efforts to retain customers.

Question 79

Does the model predict how likely the customer is to churn?

Answer 79

No. It only gives a binary churn/no churn result.

Question 80

What did we state in our integrim progress report?

Answer 80

x

Question 81

What did we state in our workplan?

Answer 81

x

Question 82

What was the three main thing you did in your paper?

Answer 82

- We defined the data set - We acquired and prepared the data - We used data mining techniques to extract value from data

Question 83

Why did you use cross validation? And what type did you use?

Answer 83

On average is the best way to evaluate how generalizable our model was. It is always working on held-out data (training data). We used stratified cross validation

Question 84

Why did you use stratified k-fold cross validation?

Answer 84

Stratified cross validation is a way of splitting or portion the entire dataset into “k” bin of equal size in order to get a fair share of data points in both the training data and test data

Question 85

What did we think was challenging? Were we too ambitious?

Answer 85

Large dataset. Time consuming to clean.

Question 86

Did you use any other approaches to big data that were not covered explicitly in the curriculum?

Answer 86

Python - ten line python script to transform dates, understand something, make the instances more clean and therefore more intuitive and usable

Question 87

What is Linux?

Answer 87

Basis for serious data science

Question 88

Why didnt you use the results from the training data?

Answer 88

Because the important results are on the test data because then you have trained the model and ....