AI2 lecture 1 -> Uncertainty Flashcards
(24 cards)
what is uncertainty in AI
Uncertainty refers to situations where the outcome or state
of a system is not fully known.
what are the different sources of uncertainty in AI
Data Uncertainty
Model Uncertainty
Environmental Uncertainty
what is data uncertainty
Noisy or incomplete data.
what is model uncertainty
Limited or incorrect model assumptions.
what is environmental uncertainty
Dynamic, unpredictable environments.
what is predictive uncertainty
Suppose we trained a model fˆw(x), where ˆw denotes the parameters
of the model obtained on the training set, and (x, y ) is a random
test example.
What is the distribution of the residual y − fˆw(x)?
Predictive uncertainty refers to this distribution.
what are the 2 main types of uncertainty
Aleatoric Uncertainty
Epistemic Uncertainty
what is aleastoric uncertainty
Intrinsic randomness in data or processes.
Example: Sensor noise in autonomous vehicles.
what is epistemic uncertainty
▶ Uncertainty due to lack of knowledge; reducible with more data or improved models.
▶ Example: Insufficient training data for a machine learning model.
example of aleastoric uncertainty
▶ Example: Sensor noise in autonomous vehicles.
example of epistemic uncertainty
▶ Example: Insufficient training data for a machine learning model.
what is the residual error
y − fˆw(x) = y − gw∗ (x) + gw∗ (x) − fˆw(x)
residual y − fˆw(x) = aleatoric + epistemic
To simplify, let’s assume that we know the model family, so g = f . Then,
y − fˆw(x) = y − fw∗ (x) + fw∗ (x) − fˆw(x)
y − gw∗ (x)
what does this mean in term of aleastoric uncertainty
Error of the best possible model gw ∗ .
(Typically, neither the ”true model” g nor its ”true parameters” w∗
are known to us.)
gw∗ (x) − fˆw(x)
what does this mean in term of epistemic uncertainty
Error of our model fˆw vs. gw∗
what is the bayesian perspective of handling uncertainty
Uncertainty is represented as a probability distribution (prior belief,
which incorporates prior knowledge) over a set of parameter values of
the model, which are updated based on new data.
Parameters are treated as random variables, and data is seen as
non-random.
how are parameters and data treated in bayesian perspective
Parameters are treated as random variables, and data is seen as
non-random.
what is the frequentist perspective of handling uncertainty
▶ Uncertainty is based on long-run frequencies of outcomes.
▶ Data is treated as a random draw from some (unknown) distribution,
and the model parameters are non-random.
▶ Data is used to estimate (fit) parameters
how are parameters and data treated in frequentist perspective
Frequentist Perspective:
▶ Uncertainty is based on long-run frequencies of outcomes.
▶ Data is treated as a random draw from some (unknown) distribution,
and the model parameters are non-random.
▶ Data is used to estimate (fit) parameters
different between bayesian and frequentist in terms of interpretation of probabaility
Bayesian: Probability as a degree of belief.
Frequentist: Probability as the long-run frequency of events.
different between bayesian and frequentist in terms of handling of uncertainty
▶ Bayesian: Can quantify and update epistemic uncertainty.
▶ Frequentist: Uncertainty is captured by confidence intervals or
p-values, not belief.
different between bayesian and frequentist in terms use of data
▶ Bayesian: Incorporates prior knowledge and continuously updates.
▶ Frequentist: Relies on observed data to estimate fixed model
parameters
whay handle uncertainty in AI
Improved Decision-Making: Accounts for possible errors or
unknowns. Examples:
▶ if uncertain, then patient should see a doctor
▶ if uncertain, robot should avoid obstacle, defer action, move closer,
collect more data, etc.
▶ if uncertain, raise a query to the humans (active learning).
▶ Better Model Generalization: Avoids overfitting to noisy data.
▶ Robustness and Reliability:
▶ Reducing epistemic uncertainty facilitates performance under
unpredictable conditions.
▶ Quantifying the confidence and accuracy of your model helps the
human user take appropriate decisions.
reak world applications of AI
Autonomous Vehicles: Predicting traffic behavior under varying
conditions.
▶ Medical Diagnostics: Handling noisy data and ambiguous
symptoms.
▶ Finance: Forecasting under market volatility.
▶ Computer Vision and Robotics
▶ Many vision and robotic systems still operate with deterministic
algorithms.
▶ Huge scope to incorporate uncertainty handling in object detection,
dynamic models, parameters, etc
