Deep Learning: Introduction and Mathematical Foundations

Question 1

What is the key idea of Machine Learning?

Introduction

Answer 1

“Machines that think”. It refers to the idea of creating machines that can perform tasks requiring intelligence, like learning, reasoning, or problem-solving.

Question 2

Why are abstract and formal tasks easy for machines but hard for humans?

Introduction

Answer 2

Abstract tasks, like playing chess, involve well-defined rules and patterns that machines can follow easily, whereas humans struggle with the computational aspects.

Question 3

What is machine learning?

Introduction

Answer 3

Machine learning is the acquisition of knowledge from raw data by identifying patterns and learning from them.

Question 4

Why is data representation important in machine learning?

Introduction

Answer 4

Data is formatted into representations and features to make pattern recognition easier, though identifying the most beneficial features can be challenging.

Introduction

Question 5

What is representation learning?

Introduction

Answer 5

Representation learning uses machine learning to map raw data into meaningful representations, separating factors of variation. An example is an autoencoder (Encoder + Decoder).

Introduction

Question 6

What is the role of an encoder and decoder in an autoencoder?

Introduction

Answer 6

The encoder converts input data into a representation, and the decoder reconstructs the input from that representation.

Introduction

Question 7

What is deep learning, and how does it differ from other types of machine learning?

Introduction

Answer 7

Deep learning is a subfield of machine learning that extracts abstract features through multiple layers, including visible (simple) and hidden (abstract) layers.

Introduction

Question 8

What historical concepts influenced deep learning architectures?

Introduction

Answer 8

Concepts like cybernetics, connectionism, and artificial neural networks were inspired by neuroscience.

Introduction

Question 9

What does “model depth” mean in deep learning?

Introduction

Answer 9

Model depth is the number of sequential instructions or computations required to evaluate an output.

Introduction

Question 10

Why can’t linear models learn the XOR function?

Introduction

Answer 10

Linear models cannot capture the non-linearity required to separate XOR inputs, as they rely on straight-line decision boundaries.

Introduction

Question 11

How has the understanding of biological neurons influenced modern neural networks?

Introduction

Answer 11

Modern neural networks adopt the idea that increased interconnections between neurons lead to more intelligent systems, though they diverge from actual biological neurons.

Introduction

Question 12

What is the modern neuron architecture used in neural networks?

Introduction

Answer 12

The Rectified Linear Unit (ReLU) is the standard modern neuron architecture for neural networks.

Introduction

Question 13

What is the significance of deep belief networks?

Introduction

Answer 13

The Deep Belief Network (DBN), introduced in 2006 by Geoffrey Hinton, revolutionized deep learning by using greedy layer-wise pretraining with Restricted Boltzmann Machines (RBMs) to address vanishing gradients, enable unsupervised feature learning, and make deep networks trainable, sparking the modern era of deep learning.

Introduction

Question 14

How did big data impact machine learning?

Introduction

Answer 14

Big data provided vast amounts of information, making it easier for machine learning models to learn and improve performance.

Introduction

Question 15

How are model size and performance related to computer capabilities?

Introduction

Answer 15

Model size and performance are directly proportional to the computational power and performance of modern computers.

Introduction

Question 16

What is a logical inference machine?

Introduction

Answer 16

A system that reasons automatically about formal statements using logical inference rules.

Introduction

Question 17

What is a knowledge base in AI?

Introduction

Answer 17

A database of formally defined facts and rules used for reasoning in AI systems.

Introduction

Question 18

What is logistic regression?

Introduction

Answer 18

A simple machine learning algorithm used for binary classification problems.

Introduction

Question 19

What is Naive Bayes?

Introduction

Answer 19

A probabilistic classifier based on Bayes’ theorem, assuming independence between features.

Introduction

Question 20

What is a multilayer perceptron (MLP)?

Introduction

Answer 20

A feedforward neural network consisting of multiple layers of neurons with activation functions, enabling hierarchical learning.

Introduction

Question 21

How is model depth measured in neural networks?

Introduction

Answer 21

By counting the number of sequential computations or layers in the network.

Introduction

Question 22

What is the McCulloch-Pitts neuron?

Introduction

Answer 22

A simple mathematical model of a biological neuron used in early neural network research.

Introduction

Question 23

What is ADALINE?

Answer 23

Adaptive Linear Neuron, an early machine learning model using linear activation and adaptive weights.

Question 24

What is stochastic gradient descent (SGD)?

Answer 24

An optimization algorithm that updates model parameters using the gradient of a randomly selected subset of data points.

Question 25

What is the Neocognitron?

Answer 25

A hierarchical, convolutional neural network model inspired by the human visual system.

Question 26

What is parallel distributed processing (connectionism)?

Answer 26

A perspective in AI emphasizing the distributed representation of knowledge across interconnected units.

Question 27

What are distributed representations in deep learning?

Answer 27

Representations where information is encoded across multiple neurons rather than a single unit.

Question 28

What is an LSTM model?

Answer 28

Long Short-Term Memory model, a type of recurrent neural network designed to capture long-term dependencies in sequential data.

Question 29

What is reinforcement learning?

Answer 29

A machine learning paradigm where an agent learns to maximize rewards through interactions with its environment.

Question 30

What are the different components of a tensor space?

Answer 30

A scalar is a singular value, representing a single numerical quantity. A vector is a one-dimensional collection of values, typically representing magnitude and direction in space. A matrix is both a one-dimensional collection of vectors and a two-dimensional collection of values, organized in rows and columns. A tensor is an 𝑛-dimensional collection of values, generalizing scalars, vectors, and matrices to higher dimensions.

Question 31

What are some important matrix properties? (Part 1)

Answer 31

Transpose: Flips rows and columns. Inverse: A matrix 𝐴 such that 𝐴𝐴⁻¹ =𝐼 where 𝐼 is the identity matrix. Square Matrix: Equal number of rows and columns. Singular: A matrix without an inverse (determinant = 0). Diagonal Matrix: Non-diagonal elements are zero.

Question 32

What are some important matrix properties? (Part 2)

Answer 32

Symmetric: 𝐴=𝐴^𝑇 Orthogonal: 𝐴𝐴^𝑇=𝐼 Orthonormal: Vectors in 𝐴 are unit length and orthogonal Trace: Sum of diagonal elements Unit Vector: A vector of magnitude 1

Question 33

What are positive and negative definite matrices ?

Answer 33

A matrix 𝐴 is positive definite if: 1. All its eigenvalues are strictly positive (𝜆_𝑖0) 2. For any non-zero vector 𝑥, the quadratic form 𝑥^𝑇𝐴𝑥=0 A matrix 𝐴 is negative definite for the vice versa of the above. Positive definite matrices often appear in optimization, where they ensure that a function has a unique minimum. Negative definite matrices indicate that a quadratic function is concave and has a unique maximum.

Question 34

What are positive and negative semi-definite matrices ?

Answer 34

A matrix 𝐴 is positive semi-definite if: 1. All its eigenvalues are non-negative (𝜆_𝑖≥0). 2. For any non-zero vector 𝑥, the quadratic form 𝑥^𝑇𝐴𝑥≥0 A matrix 𝐴 is negative semi-definite for the vice versa of the above. Positive semi-definite matrices represent systems where the quadratic form cannot be negative but might equal zero for some 𝑥 Negative definite matrices indicate that a quadratic function is concave and has a unique maximum.

Question 35

How does a linear combination operator relate to solving systems of equations?

Answer 35

A linear combination operator spans all values obtainable through linear combinations of vectors. It helps determine whether a system has infinite or no solutions. z = αx + (1 - α)y for a 2 unknown system.

Question 36

How is a linear system represented mathematically?

Answer 36

A linear system can be written as 𝐴_𝑛×𝑚𝑥=𝑏, where 𝑏 ∈ 𝑅^𝑚, 𝑛≥𝑚, and solutions are linearly independent.

Question 37

What is a norm in linear algebra?

Answer 37

A norm measures the "size" or "span" of a vector in space. Common norms include: 1. L₂ (Euclidean Norm): ∥𝑥∥₂= √ (∑x_i² 2. L₁ (Manhattan Norm): ∥𝑥∥₂= ∑x_i ^{3. L_inf (Max Norm): ∥𝑥∥_inf= max(x_i) 4. Frobenius Norm: The Euclidean Norm for a matrix}

Question 38

What conditions must a norm function satisfy?

Answer 38

A norm function must: 1. Satisfy 𝑓(0)=0, 2. Obey the triangle inequality, and 3. Be homogeneous under scaling.

Question 39

Why are orthogonal matrices important?

Answer 39

Orthogonal matrices are computationally efficient because their determinant is easy to compute, and their inverse is simply their transpose.

Question 40

What are eigenpairs, and how are they represented?

Answer 40

Eigenpairs consist of: 1. Eigenvalue (𝜆): Indicates the stretching or scaling factor. 2. Eigenvector (𝑣): Specifies the direction. The relationship is expressed as 𝐴𝑣=𝜆𝑣.

Question 41

What does matrix diagonalizability mean?

Answer 41

A matrix 𝐴 is diagonalizable if it can be written as 𝐴=𝑉diag(𝜆)𝑉⁻¹, where 𝑉 is the eigenvector matrix and 𝜆 are eigenvalues. This representation helps assess system stability.

Question 42

What is Singular Value Decomposition (SVD)?

Answer 42

U: Left singular vectors, Σ: Diagonal matrix of singular values, 𝑉^𝑇: Right singular vectors. It generalizes matrix inversion and applies even when eigenvalue decomposition fails.

Question 43

What is the Moore-Penrose Pseudoinverse?

Answer 43

The Moore-Penrose Pseudoinverse is used for matrices that are: 1. Non-square, 2. Singular, or 3. Represent systems with no or infinite solutions. It provides a generalized inverse to solve 𝐴𝑥=𝑏.

Question 44

What is PCA, and why is it important?

Answer 44

Principal Component Analysis (PCA) is a dimensionality reduction technique that: 1. Extracts features by maximizing variance, 2. Reduces data dimensions while preserving variance, and 3. Simplifies data visualization and computation.

Question 45

How does PCA work?

Answer 45

PCA involves: 1. Preparing the data, 2. Computing the covariance matrix, 3. Finding eigenpairs (eigenvalues and eigenvectors), 4. Sorting and selecting principal components, and 5. Transforming the data into the new basis.

Question 46

What are the three main sources of uncertainty in probability?

Answer 46

Inherent Stochasticity – Events have randomness built into them. Incomplete Observability – We lack knowledge of all variables in the system. Incomplete Modeling – Our models either lack information or deliberately discard it.

Question 47

What is the "degree of belief" in probability?

Answer 47

The degree of belief is a fuzzy logic truth value between 0 (completely uncertain) and 1 (completely certain), representing confidence in an event occurring.

Question 48

How does frequentist probability differ from Bayesian probability?

Answer 48

Frequentist Probability focuses on event occurrence rates over many trials. Bayesian Probability quantifies certainty in an event occurring, incorporating prior knowledge. Deep Learning typically uses the former while the use of the latter is growing

Question 49

What are random variables, and how are they classified?

Answer 49

Discrete Random Variables take finite or countable values. Continuous Random Variables take values in an uncountable real range.

Question 50

What is the Probability Mass Function (PMF), and what properties does it satisfy?

Answer 50

The PMF gives the probability distribution of discrete random variables. It must satisfy: 1. 0≤𝑃(𝑋=𝑥)≤1for all 𝑥 2. ∑_𝑥𝑃(𝑋=𝑥)=1(it normalizes to 1). It can also describe joint probability distributions across multiple variables.

Question 51

What is a uniform distribution?

Answer 51

A distribution where all events have equal probability.

Question 52

What is the Probability Density Function (PDF), and how does it differ from the PMF?

Answer 52

The PDF describes the probability distribution of continuous random variables. Properties: 1. It integrates to 1 over the domain: ∫ _−∞^∞𝑝(𝑥)𝑑𝑥=1 2. It satisfies 𝑝(𝑥)≥0 for all 𝑥. Unlike PMFs, PDF values are not probabilities themselves but density values.

Question 53

What is marginal probability, and how is it computed?

Answer 53

Marginal probability is the probability distribution of a subset of variables when the joint probability over all variables is known. It is computed by summing (for discrete variables) or integrating (for continuous variables) over the unwanted variables. For discrete variables (summing over unwanted variable 𝑦): 𝑃(𝑥)=∑_𝑦𝑃(𝑥,𝑦) For continuous variables (integrating over unwanted variable 𝑦): 𝑃(𝑥)=∫𝑃(𝑥,𝑦)𝑑𝑦 This process effectively eliminates the variables that are not of interest.

Question 54

What is expectation?

Answer 54

The expected value (average) of a function over a probability distribution. 1.Discrete case: 𝐸[𝑓(𝑋)]=∑_𝑥𝑓(𝑥)𝑃(𝑋=) 2. Continuous case: 𝐸[𝑓(𝑋)]=∫𝑓(𝑥)𝑝(𝑥)𝑑𝑥 Expectation is linear and homogeneous: E[aX+b]=aE[X]+b

Question 55

What is conditional probability?

Answer 55

The probability of an event given another event has occurre: 𝑃(𝐴∣𝐵)=𝑃(𝐴∩𝐵)/𝑃(𝐵), where 𝑃(𝐵)≠0, since conditioning on an impossible event is undefined.

Question 56

What is variance, and how is it computed?

Answer 56

Variance measures the spread of a random variable: Var(𝑋)=𝐸[(𝑋−𝐸[𝑋]²)] A low variance means values cluster around E[X], while high variance means values are widely spread.

Question 57

How is standard deviation related to variance?

Answer 57

The standard deviation is the square root of variance: 𝜎=√Var(𝑋)

Question 58

What is covariance?

Answer 58

A measure of how two variables are linearly related: Cov(𝑋,𝑌)=𝐸[(𝑋−𝐸[𝑋])(𝑌−𝐸[𝑌])] High covariance means variables vary together. The diagonal of the covariance matrix is a vector of variance values.

Question 59

What is correlation, and how is it different from covariance?

Answer 59

Correlation normalizes covariance: 𝜌(𝑋,𝑌)=Cov(𝑋,𝑌)/𝜎_𝑋𝜎_𝑌 It is bounded between -1 and 1, making it easier to compare relationships across different scales.

Question 60

What is a Bernoulli distribution?

Answer 60

A binary probability distribution: 𝑃(𝑋=𝑥)=𝜙^𝑥(1−𝜙)^(1−𝑥) where 𝑥∈{0,1}. Expectation: 𝐸[𝑋]=𝜙 Variance: Var(X)=ϕ(1−ϕ).

Question 61

What is the precision of a distribution?

Answer 61

Precision is the inverse of variance: 𝜆=1/𝜎² It simplifies calculations in Gaussian distributions.

Question 62

# ``` ``` What is a Standard Normal Distribution?

Answer 62

A Normal Distribution with Expectation: 𝐸[𝑋]=0 and Standard deviation: 𝜎=1

Question 63

What is the Central Limit Theorem?

Answer 63

The sum of many independent random variables is approximately normally distributed, regardless of their original distribution.

Question 64

What is an indicator function?

Answer 64

A Boolean function representing whether a condition is satisfied: 1_{condition}. It takes values: 1 if the condition is true 0 otherwise.

Question 65

# ``` ``` What is a Dirac distribution?

Answer 65

A distribution concentrated at a single point: p(x)=δ(x−μ), where δ is the Dirac delta function.

Question 66

What is the Dirac Delta Function?

Answer 66

A generalized function: δ(x)=0 everywhere except at x=0. ∫ _−∞^∞ δ(x)dx=1. It is used in empirical distributions.

Question 67

What is a Mixture Distribution?

Answer 67

A distribution formed from multiple component distributions, often using latent variables (unobservable random variables). It allows for more flexible modeling of complex data. Mathematically, a mixture distribution is expressed as: P(x)= ∑_k=1 ^K π_k P(x∣z=k), where: K is the number of component distributions. 𝜋_𝑘is the mixing coefficient, representing the probability of selecting the k-th component. 𝑃(𝑥∣𝑧=) is the component probability distribution given that the latent variable z is in state k. The mixing coefficients satisfy : ∑_k=1 ^K π_k =1

Question 68

What is the Gaussian Mixture Model (GMM)?

Answer 68

A mixture model using Gaussians, acting as a universal density approximator. It estimates prior and posterior probabilities.

Question 69

What is the logistic sigmoid function?

Answer 69

A function that transforms values into probabilities: σ(x)=1/(1+e^−x) It is used to generate 𝜙 values for Bernoulli Distributions It Saturates at extreme values. It stays flat/stable for extremely small changes The logit function is commonly used and is its inverse

Question 70

What is the Softplus function?

Answer 70

A smooth approximation of max(0, x): Softplus(x)=log(1+e^x) It is used to generate 𝜎 values for Gaussian Distributions

Question 71

What are independent variables in probability?

Answer 71

Two random variables 𝑋 and 𝑌 are independent if knowing one does not change the probability distribution of the other. This is mathematically defined as: P(X,Y)=P(X)P(Y) which means their joint probability equals the product of their individual probabilities.

Question 72

What are dependent variables in probability?

Answer 72

Two variables are dependent if knowledge of one changes the probability of the other, meaning: P(X,Y)=P(X=x)P(Y=y|X=x) In this case, their probabilities influence each other.

Question 73

What is conditional independence, and how is it defined mathematically?

Answer 73

Two events A and B are conditionally independent given a third event C if knowing C makes A and B independent. Mathematically, this is written as: P(A,B∣C)=P(A∣C)P(B∣C) This means that once we account for C, the occurrence of A does not affect the probability of B.

Question 74

What is an isotropic Gaussian distribution?

Answer 74

An isotropic Gaussian distribution is a multivariate normal distribution where the covariance matrix is a scaled identity matrix, meaning the variables have equal variance and are uncorrelated. Mathematically, it is given by: 𝑃(𝑥)=1/(2𝜋𝜎²)^𝑛/2exp(-∥𝑥−𝜇∥²/2𝜎²) where: μ is the mean vector. σ ²I is the covariance matrix (where is the identity matrix). The distribution is "isotropic" because it has the same variance in all directions.

Question 75

What is an empirical distribution

Answer 75

The empirical distribution represents observed data points and provides an estimate of the true underlying distribution. It is defined as: 𝑃_𝑛(𝑥)=1/𝑛∑_i=1 ⁿ=1/𝑛∑𝛿(𝑥−𝑥_𝑖) where: 𝑥_𝑖 are observed data points. 𝛿(𝑥−𝑥_𝑖) is the Dirac delta function, which ensures that each observation contributes a probability mass of 1/𝑛 The empirical distribution converges to the true distribution as the number of observations increases.

Question 76

What is Bayes’ Rule, and how is it applied?

Answer 76

Bayes' Rule is a fundamental theorem in probability that describes how to update beliefs in light of new evidence. It is given by: 𝑃(𝐴∣𝐵)=𝑃(𝐵∣𝐴)𝑃(𝐴)𝑃(𝐵) where: 𝑃(𝐴∣𝐵) is the posterior probability (updated belief after observing B). 𝑃(𝐵∣𝐴) is the likelihood (probability of observing B given A). 𝑃(𝐴) is the prior probability (initial belief about A). 𝑃(𝐵) is the evidence (normalization factor).