Lecture 1

Question 1

what is the definition of a classifier?

Answer 1

it is a function or an algorithm that maps every possible input (from a legal set of inputs) to a finite set of decisions

Question 2

what are discriminating features?

Answer 2

they are features that should help the system to discriminate between inputs from different classes

Question 3

what is Classification error (total error)?

Answer 3

the ratio between the number of test objects and the objects that were classified incorrectly

for example, if there are 50 test objects, and 17 of them were classified incorrectly, the classification error is 17/50

Question 4

True / False ?

It is better to use one discriminating feature, because it is simpler

Answer 4

False!!
usually one feature is poor alone, and it results in relatively high total error

Question 5

what is overfitting the data?

Answer 5

using complicated boundaries that are too “tuned” to the particular
training data

lesson learned: avoid using too complicated boundaries

example of overfitting: using a boundary that match EVERY datum of the training data, even if it is an outlier

Question 6

True / False?

overfitting don’t generalize well

Answer 6

True

Question 7

True / False?

If decision boundary 1 perform better on the training data than decision boundary 2, it will likely perform better on new data too

Answer 7

False!

One of the examples is overfitting; overfitting decision boundary perform ideally on the training data but usually perform poorer on testing data than a simpler decision boundary

Question 8

True / False ?

A lot is known easier
A little is known harder

Answer 8

True

Question 9

True / False?

training data is needed when the probability distribution of the categories is known

Answer 9

False

Question 10

What do we need to do when the shape of probability distribution is known?

Answer 10

we need to estimate parameters of probability
distribution from the training data

Question 11

What should we do when no probability distribution (no shape or
parameters are known) but the shape of discriminant functions is
known

Answer 11

Need to estimate parameters of the
discriminant function

Question 12

Which is considered “easier”?
1.The shape of the probability distribution is known
2.The shape of the discrimination function is known

Answer 12

1

Question 13

What to do when neither probability distribution nor discriminant function is known (all we have is labled data)?

Answer 13

Estimate the probability distribution from the labeled data

Question 14

What to do when NOTHING is known (even the data is not labled!)?

Answer 14

1. Estimate the probability distribution from the unlabeled data
2. Cluster the data

Question 15

To which type of problems we can design optimal classifier?

Answer 15

Know probability distribution of the categories (Also called Bayesian Decision theory)

But it is rare in the real world

Question 16

How proir probabilities are determined and what they reflect?

Answer 16

They reflect our prior knowledge, and they are determined accordingly

Question 17

What event we choose when using ML classifier

Answer 17

The event which maximize the likelihood function; the event that is most likely to happen

Question 18

Is likelihood function:
1. A density
2. A probability distribution
3. Non of the above

Answer 18

3

Question 19

True or False?

Posterior is proportional to likelihood * prior

Answer 19

True

according to bayes, without the normalizing factor, cause it doesn’t depend on the class

Question 20

True or False?

MAP classifier minimizes the probability of error

Answer 20

True

Question 21

What is loss function?

Answer 21

loss functions lamda(alpha_i,c_j) describes the loss occurred when taking action alpha_i when the true
class is c_j

Question 22

True or False?

Action is just another word for class

Answer 22

False

For example, in some cases we may want to refuse to make a decision, so the number of actions is larger than the number of classes

Question 23

Why we want to have a loss function?

Answer 23

Usually some mistakes are more costly than others

For example: classifying a benign tumor as
cancer is not as bad as classifying cancer
as benign tumor

Question 24

What is conditional risk?

Answer 24

It is the expected loss associated with taking
action

It is a function of an action alpha and data x

sum of (for each class): (the penalty to decide to take action alpha when the class of x is c) * (the probability of class x being c)

Question 26

What is a zero-one loss function?

Answer 26

lambda(alpha_i,c_j) = 0 if i=j otherwise 1 in other words: 0 if we chose the right class, and 1 otherwise

Question 27

# True / False? MAP classifier is Bayes decision rule under zero-one loss function

Answer 27

True

Question 28

What is a decision rule?

Answer 28

Decision rule is a function (x) which for every x specifies action out of {alpha_1,alpha_2,...,alpha_k}

Question 29

Write the equation of average risk for alpha(x)

Answer 29

R(alpha) = integral(R(alpha(x)|x)p(x)) dx ## Footnote note that to make this as small as possible (out goal) we should minimize R(alpha(x)|x)

Question 30

How to make the average risk as small as possible

Answer 30

from the equation of the average risk we can notice that to minimize it we should minimize R(alpha(x)|x). | R(alpha) = integral(R(alpha(x)|x)p(x)) dx

Question 31

# continue the sentence at observation x choose class iff

Answer 31

g_i(x) > g_j(x) for all j != i

Question 32

What are the 3 decision rules?

Answer 32

1.ML decision rule: gi(x) = P(x|ci) 2.MAP decision rule: gi(x) = P(ci|x) 3.Bayes decision rule: gi(x) = -R(ci|x)