Lecture 3 - Optimisation and Hypothesis Space Flashcards
Map Example of Optimisation
REFER TO SLIDES
But essentially:
- Attempting to find the shortest apth using nodes, then calculating the minimum path
Define Hypothesis Space
The set of all models or hypotheses that can be represented using the selected language or representation.
What are the key characteristics of Hypothesis Space?
Key characteristics:
* It is determined by your choice of language - language defined later.
* It defines what’s possible to describe in your AI system.
* It includes all theoretical candidate solutions, regardless of whether they are “good” or “bad”.
REFER TO SLIDES FOR EXAMPLE
Define Candidate Solution
Definition: A single model within the hypothesis space; a potential solution to the problem.
Think of it as:
* An individual point inside the hypothesis space that is being tested or evaluated.
Define Solution Space
Definition: This is often used synonymously with hypothesis space, but in some contexts it refers to the space of all possible outputs or behaviours of the system based on the hypothesis space.
What are the three ingredients of optimisation?
As part of optimisation , there are three requirements that are used to ensure proper optimisation takes place, these are:
- Language (Solution Space/Hypothesis Space)
- Model (Candidate Solution)
- Metric (How good is the model?)
Define Language
Language: The formal system or structure used to describe possible solutions or hypothesis.
- Examples include:
○ Mathematical equations
○ Matural languages
○ Grammars
○ Logics
○ Finite automata/finite-state machines
○ Computer programs
○ Logic programs
○ Gantt charts
○ PERT charts
○ Simulation languages
○ Popsticks and glue
Why is Language considered important?
If you can’t describe it - you can’t model it, language allow you to do this
Generation VS Parsing OR Testing Vs Generate
○ Parsing (Testing):
○ Determines if a solution is valid within a language (efficient).
○ Such as the example above
○ Generating: Enumerates all valid solutions (inefficient, often infinite).
What is the Expressiveness of a Language?
Expressiveness: Some languages can express more complex ideas or solutions than others.
- When everything in one language (B) can be also be describe in the other language (A), we say A subsumes B
- If something can be be in one language (A) but not in another language (B), we say B does not subsume A
What is Chompsky Hierarchy?
Demonstrates a layered structure of language complexity (Type 3 < Type 2 < Type 1 < Type 0), in terms of expressiveness
What does the Chompsky Hierarchy look like
Its build on four types Type 3 < Type 2 < Type 1 < Type 0, with Type 3 being the least powerful and Type 0 being the most powerful
Typically it follows this level:
Type 3 are Regular Languages such as strings or regex
Type 2 are Context Free Languages such as matching parenthesis
Type 1 are Context Sensitive Languages such as symbol matching
Type 0 are Recursively Enumerable Languages which is any lanaguage that can be understood by a computer program
Why is Chompsky Hierarchy important (why does it matter)?
It gives us an idea of:
○ What kinds of models can be described
○ How complex the models can be
○ What computational resources are needed to test or parse them
Define Model
Model: A specific instance of a hypothesis described in the chosen language.
- Essentially this is an abstraction or approximation of the real world
- An instance of all the possible things that can be described in the language
- Where in the context of AI, it is a candidate solution to a problem.
Why are Models important - what does it represent?
The model is the subject of evaluation. It represents our best attempt at mimicking or understanding the target system or data.
What are some key characteristics of Models (Key Ideas)
- Models may not be perfect representations; they approximate.
- All real-world systems can be represented as functions.
- The model space is sometimes too large to search exhaustively.
Define Metric (Evaluation)
Definition: A function used to assess how good a model (hypothesis) is in comparison to the target (or real-world phenomenon).
Also called: Error function, Cost function, Fitness function, Objective function, Penalty function, Utility function
Breakdown of Metric and Types of Metrics
Formally:
* A function from hypothesis space to real numbers: f : H -> ℝ
○ f is a function.
○ It takes an input from the set H — the hypothesis space.
○ It outputs a real number (ℝ), which represents a metric or evaluation score (e.g., error, cost, fitness).
=====
Types of Metrics:
- Numerical: e.g., Mean Squared Error (MSE)
Why are Metrics important?
We need a way to measure which solution is better. This is critical in optimisation.
What is the issues with Metrics, what can be done instead?
- Usually we are happy if we can determine relative closeness
- Doesn’t need to be meaningful in an absolute sense, only relative to another
- Sometimes we don’t know much about the “real” thing
○ Can just assume its infinitely ‘good’
○ Seek the best hypothesis
What is Ideal 1 Defintion of Optimisation?
Find a model within the hypothesis space that is indistinguishable from the target (zero error).
- This is the theoretical best-case scenario.
- You’re aiming to find a model that perfectly replicates the real-world target.
- That means: when evaluated by the metric, the error is exactly zero.
Why is Ideal 1 Defintion of Optimisation important?
It gives us a goalpost—a target to aim for in optimisation.
Useful for evaluating how expressive your language is: if your representation can’t describe the perfect model, ideal optimisation is impossible.
What are the limitations of Ideal 1 Defintion of Optimisation?
- In real-world problems:
○ The hypothesis space might not contain the exact real-world model.
○ Real-world data is often noisy or incomplete.
○ Models are approximations, not exact replicas.
○ Ideal optimisation becomes intractable when the space is too large or the model is too complex.
What is Ideal 2 Definition of Optimisation?
Find a model in the hypothesis space that is closest (minimal error) to the target.
- You’re no longer aiming for zero error, but the smallest possible error that can be achieved given your hypothesis space.
- Still assumes perfect knowledge of the metric and ability to search the space effectively.
