Post-Midterm

Question 1

Define discontinuity-based segmentation

Answer 1

A segmentation technique that identifies abrupt changes in intensity (where values jump significantly compared to their neighbours)

Question 2

What is the goal of discontinuity-based segmentation

Answer 2

To separate objects from background by finding boundary pixels (edges/lines/points) that mark transitions between regions

Question 3

In order to detect edges, an important mathematical foundation needed is _________________

Answer 3

The derivative

Question 4

A positive derivative means the intensity is __________ as x increases. Vice versa for a negative derivative.

Answer 4

Increasing

(dark –> bright when moving to the right)

Question 5

How does a first-order derivative (gradient) behave in an edge detection algorithm

Answer 5

Gradients highlight where large changes occur, but also indicate which side is darker or lighter (via sign)

Question 6

How does a second-order derivative (Laplacian) behave in an edge detection algorithm

Answer 6

Laplacians reinforce fine details and help locate edges precisely. The sign of the second derivative often reveals whether an edge is going from dark-to-light or light-to-dark

Question 7

What are the three different types of discontinuity

Answer 7

Points, lines, & edges

Question 8

Describe a point discontinuity

Answer 8

Single pixel that differs sharply from neighbours (detected by the Laplacian)

Question 9

Describe a line discontinuity

Answer 9

1-2 pixel wide structures differing in intensity from surroundings (detected by directional masks or second derivatives)

Question 10

Describe an edge discontinuity

Answer 10

Transition zones (ideal: step edges; real: blurred or ramp edges)

Question 11

What are the advantages of discontinuity segmentation

Answer 11

Can directly locate boundaries. It’s good for images where objects exhibit strong contrast against the background

Question 12

What are the challenges of discontinuity segmentation

Answer 12

It can be sensitive to noise (derivatives amplify noise). Images often require smoothing (pre-filtering) and careful threshold selection to avoid false edges or fragmented edges

Question 13

Describe what a ‘point’ is in the context of point detection

Answer 13

An isolated pixel whose intensity differs significantly from its immediate neighbours. It typically appears as a bright or dark ‘spot’ in a relatively uniform background

Question 14

What are the steps involved to implement a point detection algorithm

Answer 14

Step 1: Apply second-order derivative (Laplacian) filter

Step 2: Take the absolute value of the response

Step 3: Threshold the absolute response

Step 4: Label isolated points

Question 15

Describe how to accomplish the first step, ‘Apply Laplacian Filter’, when implementing a point detection algorithm

Answer 15

Convolve the kernel below with an image to obtain a filter response

Second-Order 3x3 Laplacian kernel:

0 1 0
1 -4 1
0 1 0

Question 16

Describe how to accomplish the second step, ‘Take the absolute value of the response’, when implementing a point detection algorithm following the application of the Laplacian filter

Answer 16

After the first step, the Laplacian response can be positive or negative. By taking the absolute value of the response, we get a magnitude that indicates how large the change is, regardless of sign

Question 17

Describe how to accomplish the third step, ‘Threshold the absolute response’, when implementing a point detection algorithm after taking the absolute value of the Laplacian response

Answer 17

Test or take a percentage of the maximum magnitude such that only prominent ‘spikes’ get labelled

Question 18

Describe how to accomplish the fourth and final step, ‘Label isolated points’, when implementing a point detection algorithm after thresholding the absolute response

Answer 18

If the magnitude of Z(x,y) is greater than the threshold, declare (x,y) an isolated point. Store this point as a 1 (or ‘white’) in an output binary image, while others are stored as 0 (or ‘black’)

Question 19

Describe a ‘line’ in the context of line detection

Answer 19

A set of connected pixels with similar intensity, often just 1-2 pixels in thickness, differing in intensity from its background

Question 20

Briefly describe the process to apply line detection to an image

Answer 20

Convolve the image with a second-order derivative filter or with specialized directional filters, as shown below. After convolution, threshold the filter response to isolate line pixels.

Vertical kernel:
-1 2 -1
-1 2 -1
-1 2 -1

Horizontal kernel:
-1 -1 -1
2 2 2
-1 -1 -1

45-degree kernel:
2 -1 -1
-1 2 -1
-1 -1 2

Question 21

Define ‘edge’ in the context of edge detection

Answer 21

A boundary between two distinct regions of intensity or texture

Question 22

What are the different types of edges?

Answer 22

Step edge - Sudden transition in intensity (ideal)

Ramp edge - Gradual transition (common)

Roof edge - Transition to one intensity from another, then quickly back to the original (typical in thin lines or object ridges)

Question 23

True or False: Image noise/blur do not cause step edges to turn into ramp edges

Answer 23

False

Question 24

What is ‘clustering’ in clustering segmentation

Answer 24

The clustering approach in segmentation involves grouping pixels based on intensity, colour, or feature similarity without requiring labelled data

Question 25

What is 'K-Means' clustering?

Answer 25

An approach to clustering segmentation that involves the following steps: 1. Choose k (# of clusters) 2. Initialize cluster centres (random or heuristics) 3. Assign each pixel to nearest cluster centre (Euclidean distance in intensity space) 4. Update cluster centres as mean of assigned pixels 5. Iterate until convergence The output of this process will be an image with each pixel labelled with a cluster index (segmented image with k segments)

Question 26

What is 'Fuzzy C-Means' clustering?

Answer 26

An approach to clustering segmentation where each pixel has partial memberships to clusters (helps with ambiguous boundaries/smooth transitions) and is most useful when cluster boundaries are not sharp. This method builds clusters step-by-step (agglomerative) or splits them (divisive) and is more computationally intensive than K-Means clustering.

Question 27

What is watershed segmentation

Answer 27

A gradient-based segmentation method that interprets the gradient magnitude as a topographic surface (low intensities = valleys; high intensities = ridges)

Question 28

What are active contours (or snakes)

Answer 28

A gradient-based segmentation method that 'evolves' a curve such that it locks onto region boundaries. The curve evolves under internal smoothness constraints and external 'image forces' derived from the gradient. This means that the contours are minimized through a combination of internal energy (smoothness) and external energy (pulls curve toward edges)

Question 29

What are some common metrics used to evaluate the effectiveness of a segmentation algorithm?

Answer 29

# of correct pixels / # of total pixels Pixel Accuracy: qty. of correct pixels / qty. of total pixels IoU (Intersection over Union): intersection(A,B) / union(A,B) Dice Coefficient: 2 x [ intersection(A,B) / ( |A| + |B| ) ]

Question 30

What are common challenges involved with image segmentation?

Answer 30

- Noise sensitivity (derivatives amplify noise) - Parameter tuning (thresholds, cluster sizes, kernel sizes) - Complex scenes (overlapping objects, illumination changes, low contrast)

Question 31

Define structuring element (SE)

Answer 31

A small, predefined shape (or set of pixels) used in morphological image processing. It acts like a 'probe' that scans over an image to analyze or modify its shapes

Question 32

When processing an image using mathematical morphology, a _____________ is moved across the image to modify object shapes based on specific rules

Answer 32

Structuring element (SE)

Question 33

What are the most common morphological operations

Answer 33

- Erosion (remove pixels from objects) - Dilation (expands pixels from objects) - Opening & Closing (combinations of erosion and dilation)

Question 34

Define reflection and translation in the context of image morphology

Answer 34

Reflection: Flipping an SE by 180 degrees Translation: Moving an SE across an image

Question 35

Define the erosion morphological operation

Answer 35

This operation shrinks foreground objects in an image. This is done by translating an SE over all possible positions in an image, and marking the origin of the SE as a foreground pixel (1) if the SE fits entirely inside the image at each position. Otherwise, all other pixels are marked as a background pixel (0)

Question 36

What are the effects of the erosion morphological operation?

Answer 36

- Thins/Shrinks objects - Removes small noise - Separates connected components in an image - Shapes objects based on structuring element (e.g. elongated SE can reduce objects to a line)

Question 37

Define the dilation morphological operation

Answer 37

This operation expands/thickens foreground objects in an image. This is done by translating an SE over all possible positions in an image, and marking the origin of the SE as a foreground pixel (1) if the SE overlaps at least one foreground pixel at each position in the image. Otherwise, all other pixels are marked as a background pixel (0)

Question 38

What are the effects of the dilation morphological operation?

Answer 38

- Grows objects - Small gaps/holes filled - Shapes expand based on SE size and shape

Question 39

Describe the concept of 'morphological duality' in the context of erosion and dilation

Answer 39

Duality describes the relationship between dilation and erosion through complementation (one operation can be derived from the other by working with the image's background instead of the foreground)

Question 40

What is the erosion-dilation duality

Answer 40

The complement of the erosion of A by B is equal to the dilation of the complement of A using the reflected structuring element, Br

Question 41

What is the dilation-erosion duality

Answer 41

The complement of the dilation of A by B is equal to the erosion of the complement of A using the reflected structuring element, Br

Question 42

Define the opening morphological operation

Answer 42

This operation removes small objects or noise while preserving the general shape of larger objects. This is done by first eroding A by B (shrinking it), then dilating it back using B, partially restoring the main structure.

Question 43

Define the closing morphological operation

Answer 43

This operation fills small gaps or holes, smooths object contours, and fuses narrow breaks. This is done by first dilating A by B (expanding it), then eroding it back using B, smoothing object boundaries and filling in small gaps.

Question 44

Define 'Hit-Or-Miss' (HMT) Transform

Answer 44

A morphological tool used for shape detection in binary images. It relies on two SEs rather than one (one for foreground, other for background)

Question 45

Why would one use a hit-or-miss transform

Answer 45

HMT allows detection of small features (e.g. corners, endpoints) precisely by combining erosion with a specially designed pair of SEs

Question 46

Define boundary extraction

Answer 46

A basic morphological algorithm that isolates the edges of a foreground object using erosion and set difference

Question 47

What is the process involved in a boundary extraction algorithm?

Answer 47

Erode the object within the image using a structuring element. Then, subtract the eroded image from the original image to leave only the boundary pixels

Question 48

Define hole filling algorithm

Answer 48

A basic morphological algorithm that fills background regions enclosed by a connected foreground boundary using dilation, complementation, and intersection

Question 49

What is the process involved in a hole filling algorithm?

Answer 49

Create an array of 0's the same size as the image and set 1's at known hole locations. Then, apply dilation to to the newly created array using a symmetric SE. Intersect the result with the complement of the image to limit expansion inside the hole, and repeat this process until no further changes occur. The union of this array and the original image fills all holes while preserving object boundaries

Question 50

Define connected component extraction

Answer 50

A basic morphological algorithm that identifies and isolates groups of connected foreground pixels in a binary image

Question 51

What is the process involved in a connected component extraction algorithm?

Answer 51

Create an array of 0's the same size as the image and set 1's at known points within each connected component. Apply dilation to this array using a symmetric SE. Intersect this array with the original image to restrict growth, and repeat this process until no further changes occur. The newly created array contains all connected components of the original image

Question 52

Define convex hull

Answer 52

The convex hull of a set A is the smallest convex set that fully contains A. This basically means that many of the same type of morphological operation is performed on an image, but each with different SEs. The intersection of all these outputs is the convex hull

Question 53

Define thinning

Answer 53

A morphological operation that reduces a binary object to a skeleton-like shape while preserving its connectivity. It is defined using HMT and an iterative process involving SEs (see lecture slides for SE structures)

Question 54

Define thickening

Answer 54

A morphological operation that expands the foreground structure in a controlled way. It is the 'morphological dual' of thinning

Question 55

Define skeleton

Answer 55

A thin, central representation of a set, preserving its topology and shape while reducing redundancy

Question 56

Define morphological reconstruction

Answer 56

A powerful transformation that uses two images (marker and mask) and an SE to extract or restore objects

Question 57

What is a marker image and what is a mask image?

Answer 57

A marker image defines starting points for morphological reconstruction. A mask image restricts growth (conditions the reconstruction)

Question 58

Define geodesic dilation

Answer 58

A morphological operation that expands the marker image while limiting growth by using a mask

Question 59

Define geodesic erosion

Answer 59

A morphological operation that shrinks the marker image while staying greater than or equal to a mask

Question 60

What is 'Reconstruction by Dilation'

Answer 60

A type of morphological reconstruction that utilizes geodesic dilation iterated until stability

Question 61

What is 'Reconstruction by Erosion'

Answer 61

A type of morphological reconstruction that utilizes geodesic erosion iterated until stability

Question 62

Describe how dilation would work in a grayscale image rather than a binary image

Answer 62

The origin pixel (centre of SE) is replaced with the maximum value in the SE neighbourhood. This expands bright regions and enhances peaks

Question 63

Describe how erosion would work in a grayscale image rather than a binary image

Answer 63

The origin pixel (centre of SE) is replaced with the minimum value in the SE neighbourhood. This shrinks bright regions and enhances valleys

Question 64

Define Large Language Models (LLMs)

Answer 64

Neural networks trained on vast amounts of text data that can predict/generate human-like text, often using the transformer architecture

Question 65

What are the three different variations of transformer architecture

Answer 65

- Encoder-Decoder Structure - Encoder-Only Structure - Decoder-Only Structure

Question 66

Describe an encoder-decoder transformer architecture

Answer 66

The encoder processes input (e.g. text tokens) and produces hidden representations, decoder generates output (e.g. translated text or next token) by attending to encoder outputs and previously generated tokens. It's best applied to text summarization or question answering applications

Question 67

Describe an encoder-only transformer architecture

Answer 67

This architecture focuses only on understanding or embedding text and not typically used for text generation. It's best applied to sentiment analysis and text classification applications

Question 68

Describe a decoder-only transformer architecture

Answer 68

This architecture generates text by using past context in a single transformer block without an encoder (ChatGPT does this). It operates in an autoregressive manner (generates one token at a time while using previous outputs as context). It's best applied to story/article generation, chatbots, and code generation applications

Question 69

Define scaled dot-product attention

Answer 69

A core component of Transformer models that calculates 'attention' weights by taking the dot product of query and key vectors, scaling the result, and applying a softmax function to obtain normalized weights, which are then used to weight the value vectors.

Question 70

What are the three inputs that scaled dot-product attention takes

Answer 70

Queries (Q), Keys (K), and Values (V)

Question 71

Describe the Key (K) input to scaled dot-product attention

Answer 71

An input that represents the “memory” or reference that other tokens compare themselves to. Basically, keys define what each token has to offer

Question 72

Describe the Query (Q) input to scaled dot-product attention

Answer 72

An input that indicates what the current token is looking for in the other tokens. We compare this query against all the keys to figure out how relevant each key is to our query

Question 73

Describe the Value (V) input to scaled dot-product attention

Answer 73

Holds the actual content that can be retrieved if the token is deemed relevant (based on the query-key comparison). Once we determine how much attention (weight) to assign to each token (via the query-key matching), we use the corresponding values to form the output representation

Question 74

Define multi-head attention

Answer 74

A key mechanism of transformers that allows the model to look at different parts of the sequence. It splits the hidden representations into multiple “heads” to attend to different positions or features in parallel. This allows the model to capture more nuanced relationships within text

Question 75

Define vision transformers (ViT)

Answer 75

The same as a LLM transformer, but images are treated as tokens instead of words. Images are divided into fixed-size patches, each acting as a 'token'

Question 76

Define Contrastive Language-Image Pre-training (CLIP)

Answer 76

A model developed by OpenAI that learns to connect images and text by training on a massive dataset of image-text pairs. It creates a shared latent space where visual and textual concepts are aligned

Question 77

What are the architecture components of CLIP

Answer 77

Image encoder (converts input image into feature vector) and text encoder (converts text description into a corresponding feature vector)

Question 78

What are the two components that allow the CLIP training mechanism to function

Answer 78

Contrastive loss and joint representation space

Question 79

Describe contrastive loss

Answer 79

A component in the CLIP training mechanism where the training objective is to bring the representations of matching image-text pairs closer while pushing apart non-matching pairs. This is achieved using a loss function that considers all pairs in a batch

Question 80

Describe joint representation space

Answer 80

A component in the CLIP training mechanism where both encoders are optimized so that semantically related images and texts have similar representations

Question 81

Define diffusion models

Answer 81

A class of generative models that create images by iteratively denoising random noise

Question 82

Fill in the Blank: The diffusion process has two components, _________ and ________

Answer 82

Forward, reverse

Question 83

Describe the forward process in diffusion

Answer 83

The forward process is the actual 'diffusion' portion. Gaussian noise is added gradually over many time steps such that the image slowly degrades until it becomes nearly pure noise. The purpose of this process is to define a known probabilistic path from data to noise, which the model learns to reverse

Question 84

Describe the backward process in diffusion

Answer 84

The backward process is the 'denoising' portion. Starting from pure noise, the model performs a series of denoising steps by predicting and subtracting the noise component. This progressively refines the image and, once complete, the model reconstructs a high-quality image

Question 85

What is the data space of traditional diffusion models

Answer 85

Traditional diffusion models operate directly on pixel space

Question 86

Describe the architecture of traditional diffusion models

Answer 86

Consists of two components: Forward (diffusion) and Reverse (denoising) Process

Question 87

True or False: Traditional diffusion models have low computational cost due to their data space

Answer 87

False, computational cost is high due to the high dimensionality of pixel space

Question 88

True or False: Traditional diffusion models require many denoising steps for high-quality image reconstruction

Answer 88

True

Question 89

What is the data space of latent diffusion models

Answer 89

The data space of latent diffusion models is a compressed, lower-dimensional latent space rather than directly on pixels

Question 90

Describe the architecture of latent diffusion models

Answer 90

Consists of an encoder, decoder, and a diffusion process. The encoder is a pre-trained autoencoder or VAE that converts images into a latent representation. The decoder takes the latent representation and transforms it back to a high resolution image. Among this process, a U-Net or similar network applies the diffusion process within the latent space (prior to decoding)

Question 91

True or False: Latent diffusion is more efficient in terms of computation and memory than traditional diffusion

Answer 91

True

Question 92

What is the data space of stable diffusion models

Answer 92

The data space of a stable diffusion model incorporates a text encoder (e.g., from CLIP) to guide the image generation process based on textual prompts

Question 93

Describe the architecture of stable diffusion models

Answer 93

Consists of an autoencoder, U-Net diffusion model, and cross-attention. The autoencoder encodes images into latent space and decodes them back, as in latent diffusion. The U-Net diffusion model applies the diffusion process in latent space, but enhanced with cross-attention layers to integrate text embeddings. Cross-attention merges the text conditioning with the image latent features to steer the generation towards the desired content

Question 94

True or False: Latent diffusion models can produce high-quality, detailed images guided by natural language inputs

Answer 94

False, a stable diffusion model does that

Question 95

When integrating vision and language models, what are the three fusion strategies for accomplishing this?

Answer 95

Early Fusion: Combining modalities early in the network (less common due to different data structures) Late Fusion: Independent processing followed by alignment in a joint embedding space Cross-Attention: Integrates features from both modalities during processing for deeper interaction