test set bounds

Question 1

what does successful learning mean

Answer 1

resulting classifier ha rediction ability better then what we have seen at training
can not expect any prediction ability in general

Question 2

why study prediction theory

Answer 2

to gain insight into how the mechine brains work (insights into learning)
better methods for learning
better methods for verifying machines predictive ability

Question 3

standard technoque to verify learning succeeded

Answer 3

divide sample into training set and test set
we train on the training set
we test on the test set

Question 4

what must be avoided by algorithm design

Answer 4

overfitting

Question 5

why is the true error quantity unknown

Answer 5

the true distrbution in unknown

Question 6

when is cd (true error) a random variable

Answer 6

if C (classifier ) is trained on sample set S then cd is a random variable of the function of C

Question 7

when is cd (true error) deterministic

Answer 7

if c (classifier) is a fixed function then cd is deterministic

Question 8

what does the pivot of the cumulative find

Answer 8

it ifnds the largest true error such that probability of observing K or fewer “heads” (errors) in n tirals (test points ) is at least δ(confidence level)

Question 9

what does the text set bound tell us

Answer 9

• certify the extent to which the classifier has learnt correctly
• The guarantee is probabilistic - it only implies that the bound will not be wrong for a 1−δ fraction of its applications / i.e. training sets
• bound tigthtens with large sample sizes n
• relies on i.i.d samples
• used for both theorectical insights and practical applications
• simple and validity is well understood