Machine Learning + Supervised

Question 1

Who first defined machine learning and when?

Answer 1

Arthur Samuel 1959

Question 2

Define machine learning.

Answer 2

Field of study that gives computers the ability to learn without being explicitly programmed.

Question 3

How did Arthur Samuel implement machine learning?

Answer 3

Automated checkers programmed that learnt positions so could re-use them instead of searching manually each time.

Question 4

How does machine learning operate in a nutshell?

Answer 4

Machine receives answers and data as input and outputs rules.

Question 5

Name the steps of machine learning. x6

Answer 5

1. Data Gathering
2. Data Pre-Processing
3. Choose a Model
4. Train the Model
5. Fine-Tune the Model
6. Apply the Model

Question 6

Give some more example applications of machine learning.

Answer 6

Any one of:
Building smart robots
Text understanding
Computer vision,
Medical Informatics
Database Mining

Question 7

Distinguish between supervised and unsupervised learning with examples.

Answer 7

Supervised: Has training data with known answers and produces model to predict for new data. E.g. Classify tumors

Unsupervised: Data provided but with no answers and algorithm finds structure or interesting patterns in data. E.g. Market Segmentation Research

Question 8

Distinguish between eager and lazy learning.

Answer 8

Eager: Constructs a general, input-independent target function during training. Model constructed before any queries.

Lazy: Generalization beyond the training data is delayed until a query is made. Stores training data with only minor processing until query.

Question 9

T or F. Lazy learners take less time in training and less time in predicting than eager learners.

Answer 9

False. Lazy learners take less time in training and more time in predicting.

Question 10

List relevant algorithms studied for eager / lazy respectively.

Answer 10

Eager:
- Decision Trees
- NaiveBayes
- Neural Networks

Lazy:
- K-nearest neighbour

Question 11

Timing of Model Building: Difference between Lazy and Eager Learning.

Answer 11

Lazy: During prediction
Eager: Before prediction.

Question 12

Data Dependency: Difference between Lazy and Eager Learning.

Answer 12

Lazy: Relies heavily on training data during prediction.

Eager: Less dependent on training data during prediction.

Question 13

Computational Efficiency: Difference between Lazy and Eager Learning.

Answer 13

Lazy: Faster in training, slower in prediction due to model building.

Eager: Slower in training, faster during prediction due to pre-built model.

Question 14

Memory Usage: Difference between Lazy and Eager Learning.

Answer 14

Lazy: Less memory usage in training, more during prediction.

Eager: More memory in training, but less during prediction.

Question 15

Other than supervised, unsupervised, what is the third type of machine learning?

What drives it?
Give one use.
Give one example algorithm.

Answer 15

Reinforcement Learning
Learning from mistakes
Gaming
Q-Learning

Question 16

What drives supervised and unsupervised learning respectively.

Answer 16

Supervised: Task driven
Unsupervised: Data driven

Question 17

What is regression?

Answer 17

Technique used in supervised learning where historical data is used to predict future.

Question 18

What is a decision tree?

Answer 18

Each node has a question, each branch possible answers, leaf nodes have decisions. E.g. Series of q’s to determine user BMI

Question 19

Define the following terms as parts of decision tree.
1. attribute
2. attribute value
3. classification
4. conjunctive term

Answer 19

1. Internal Node
2. branch
3. Leaf (External) Node
4. Path to a leaf node

Question 20

Outline the DT learning algorithm.

Answer 20

Choose best attribute
Add new node for this attribute and a new branch for each attribute value.
Sort training examples through node to current leaves.
If training examples are unambiguously classified then stop.
Otherwise repeat.

Question 21

T or F. Finding an optimal decision tree is tractable.

Answer 21

False. It is intractable.

Question 22

T or F. Finding a near optimal tree is tractable.

Answer 22

True.

Question 23

What is the ideal goal when making a decision tree?

Answer 23

That it be as small as possible.

Question 24

T or F. If an attribute leads to an immediate classification then it is a good attribute.

Answer 24

True

Question 25

Define entropy. What is the measurement of entropy?

Answer 25

Computational measure of impurity / uncertainty. Bits.

Question 26

What is the relationship between probability and entropy?

Answer 26

Add up the result of - p log2 p for all different outcomes with probabilities p_i and this is H(X). Large probability = low entropy Small probability = large entropy

Question 27

Formula for Gain(S, D).

Answer 27

Gain(S, D) = H(S) - (sum for all attribute-values of D) (p of attribute value) * entropy of attribute value

Question 28

When do we stop decision tree making process?

Answer 28

When entropy = 0

Question 29

What is the problem of noise? How can we fix it?

Answer 29

They are no attributes left, but there are both positive and negative examples remaining. Have leaf nodes report majority or probabilistic classifications.

Question 30

What is the problem of overfitting? How can we fix it?

Answer 30

Algorithm may use irrelevant attributes to make spurious distinctions among the examples. Use statistical significance to determine whether a Gain is large enough to proceed.

Question 31

What is the problem of missing data? How can we fix it?

Answer 31

Not all new examples will have data for every attribute. Assign average value for empty attribute or try all branches.

Question 32

What is the problem of multi-valued data? How can we fix it?

Answer 32

Attributes with large number of possible values can non-representative gain results. E.g. Id Penalize broad uniform splits.

Question 33

What is the problem of continuous variables? How can we fix it?

Answer 33

Building tree with continuous values from training data leads to problems as new examples are unlikely to have exact same value. Use ranges, rather than exact values.

Question 34

What is the problem of costly attributes? How can we fix it?

Answer 34

Testing certain attributes may carry a substantial cost. Low cost might be more desirable than small tree. Solution: Incorporate cost. E.g. Gain^2 / Cost

Question 35

On what are decision trees, naive bayes and KNN based on?

Answer 35

D: Information-Based N: Probability-Based K: Similarity Based

Question 36

T or F. Naive Bayes is eager.

Answer 36

True

Question 37

T or F. Decision Trees is eager and naive bayes is not.

Answer 37

False

Question 38

T or F. Decision Trees is eager.

Answer 38

True

Question 39

T or F. KNN is eager.

Answer 39

False

Question 40

What are lazy learners also known as and why?

Answer 40

Instance-based learners because they store training points or instances.

Question 41

What is feature-space?

Answer 41

An n-dimensional way to describe a dataset which n attributes (one for each dimension). All datapoints can be positioned at a unique point in this depiction.

Question 42

What does a nearest neighbor algorithm do?

Answer 42

Seeks to identify the most similar instance in the known training set to a given query.

Question 43

What is the general formula for similarity used in nearest neighbor algorithm. Assume m dimensions. What is the name of the general formula? How can Manhattan and Euclidean distance be derived from it?

Answer 43

Let a, b be two points in the m-dimensional feature-space. (sum_{i=1}^{m} abs(a[i] - b[i])^{p}))^{1/p} Minkowski distance M: p = 1 E: p = 2

Question 44

What is a voroni tessellation?

Answer 44

- Shows how input space divides into classes - "local neighborhoods" across the feature space with each space define by a subset of the training data. - Each line segment is equidistant between two points of opposite classes.

Question 45

Difference between 1NN and 3NN.

Answer 45

In 1NN, if there is an outlier, anything closest to it will be misclassified. We smooth this out by having k (e.g. 3) nearest neighbors vote.

Question 46

What are the three primary requirements of knn?

Answer 46

Set of training instances Distance metric to measure between distances The value of k, number of nearest neighbors to retrieve.

Question 47

What is the general idea of knn?

Answer 47

Use class labels of k nearest neighbors to determine the class label of an unknown instance.

Question 48

Name for the process of choosing the right k in knn. Discuss choosing small k vs big k.

Answer 48

Parameter Tuning k too small -> disrupted by noise k too big -> less precise

Question 49

Give two sample methods of choosing k that are good rules of thumb.

Answer 49

Choose odd k when even number of instances. Choose k < sqrt(n) for n instances.

Question 50

Outline KNN basic classification algorithm.

Answer 50

- Calculate the similarity of new point to all the instances in feature-space. - Rank instances in feature-space by their similarity to new point. - Identify the set of k nearest neighbors. - Report the "majority vote" of this set.

Question 51

Outline some common problems with KNN.

Answer 51

Scaling Issues & Missing Values: Attributes may have to be scaled to not dominate and missing values leads to misclassification. Outliers: Individual spaces which are separate from the class to which they belong. Noise. Especially for low k values causes misclassification.

Question 52

T or F. Knn is supervised and lazy.

Answer 52

True

Question 53

T or F. Decision Trees are unsupervised and eager.

Answer 53

False. They are supervised.

Question 54

State Bayes Theorem.

Answer 54

P(c | x) = P(x | c)P(c) / P(x)

Question 55

What are the two assumptions when dealing with multiple attributes and bayes theorem? Are these assumptions real-word and what does this mean?

Answer 55

1. Attributes are conditionally independent. 2. Attributes are weighted equally towards the outcome. No in all likelihood, means it is not perfectly accurate but yet it still yields very good results.

Question 56

T or F. Naive Bayes performs better on categorical data compare to numerical data/

Answer 56

True

Question 57

T or F. Naive Bayes is O(N).

Answer 57

True

Question 58

Outline how Naive Bayes is performed.

Answer 58

Construct a frequency table and than a likelihood table for each attribute against the target class. Then apply formula P(c | X) = P(x1 | c) x P(x2 | c) x ... x P(xn | c) x P(c) To all cases for the target class.

Question 59

How do I standardize naive bayes to have values between 0 and 1?

Answer 59

P(Yes) / P(No) + P(Yes) = Yes standardized. With No standardized being the opposite.

Question 60

Give an application for naive bayes.

Answer 60

Credit scoring.