4 - Introduction to probability distributions

Question 1

What is the difference between supervised and unsupervised learning?

Answer 1

Supervised learning involves training a model on labeled data (input-output pairs), while unsupervised learning uses data without labels to find structure or patterns.

Question 2

Define the bias-variance tradeoff in machine learning.

Answer 2

The bias-variance tradeoff is the balance between a model’s ability to generalize (low variance) and its accuracy on training data (low bias). High bias leads to underfitting, high variance to overfitting.

Question 3

What is overfitting?

Answer 3

Overfitting happens when a model learns noise in the training data and performs poorly on unseen data.

Question 4

What are decision trees?

Answer 4

Decision trees are predictive models that split data into branches to predict outcomes, using rules learned from features.

Question 5

Define entropy in the context of decision trees.

Answer 5

Entropy measures the impurity or uncertainty in a dataset. In decision trees, it’s used to decide how to split the data.

Question 6

What does a decision tree algorithm aim to do at each node?

Answer 6

It selects the attribute that provides the highest information gain to split the data.

Question 7

What is the purpose of pruning in decision trees?

Answer 7

Pruning reduces the size of the decision tree by removing nodes that provide little predictive power, helping to prevent overfitting.

Question 8

What are the two main types of pruning?

Answer 8

Pre-pruning (early stopping): Stop tree growth early based on criteria like max depth or minimum samples per node.
Post-pruning (reduced error pruning): Grow the full tree first, then remove nodes that do not improve performance on validation data.

Question 9

What is the ID3 algorithm?

Answer 9

ID3 (Iterative Dichotomiser 3) builds decision trees by selecting attributes that maximize information gain at each node.

Question 10

How does ID3 handle numeric attributes?

Answer 10

ID3 requires discretization of numeric attributes — converting them into categorical bins or thresholds.

Question 11

What’s a major limitation of ID3?

Answer 11

It can be overfit on noisy data and does not handle continuous attributes or pruning natively.