Safety and Reliability

Question 1

Erroneous Behaviour of a Classifier

Answer 1

Given a trained classifier f : R_n -> R_k (from n features to k values) and a target function h : R_n -> R_k, an erroneous behavior of the classifier f is demonstrated by a legitimate input x which exists in R_n such that
arg max_j f(x) != arg max_j h(x)

Question 2

Loss function

Answer 2

L(y, f(x)) ; loss between prediction f(x) and ground truth y.

Question 3

Empirical Loss

Answer 3

Average loss over a set.

Question 4

Expected Loss

Answer 4

The estimated loss (loss of accuracy) before being tested.

Question 5

Generalisation Loss

Answer 5

Empirical loss - Expected loss. Too big of this value is a result of overfitting.

Question 6

Overfitting

Answer 6

A machine learning model is overfitted if it performs well on training data but not on test data samples.

Question 7

Adversarial Examples

Answer 7

Represent erroneous behaviours which introduce safety implications.

Question 8

Measurements of adversarial examples

Answer 8

• Magnitude of perturbation -> ||x-x’||
• Probability gap between and after the perturbation -> |f_y(x) - f_y(x’)|
  (With f(x) being the regular example and f(x’) being the adjusted example.)

Question 9

Data Poisoning

Answer 9

The injection of malicious data into a training process, making the algorithm perform something it should not.

Question 10

Model Stealing

Answer 10

Given model f, a model stealing agent reconstructs another model f’ (etc querying model f).

Question 11

Membership Inference

Answer 11

Identifies the training data via shadow models for the training model by observing the models behaviour and the outcomes.