Week 10 - Kalman Filter, Recurrent Neural Networks

Question 1

Define Kalman Filter

Answer 1

It is similar to HMMs, but it is for Kalman filter is for continuous hidden state (𝒁𝒕)

Question 2

What does each hidden state (𝒁𝒕) depend on?

Answer 2

It depends on the one before it in time 𝑍𝑡−1, for all 𝑡 = 1, 2, … , 𝑇

Question 3

What are the observations depends on in Kalman filter?

Answer 3

It only depend on the associated hidden state (𝒁𝒕)

Question 5

What is Kalman filter often used for?

Answer 5

It is often used for target tracking or motion smoothing with noisy observations applications in automated airplane or ship guidance

Question 6

What is the linear- Gaussian recurrence relation?

Answer 6

LGRR is the simplest model. The form is:

Question 7

What is the main focus of reinforcement learning (RL) in the context of decision-making?

Answer 7

RL focuses on finding optimal policies to maximize rewards by making decisions based on observed effects of those choices, rather than predicting sequences from observed data.

Question 8

How can dynamic programming be applied in reinforcement learning problems?

Answer 8

Dynamic programming is used in reinforcement learning when the problem is modeled as a Markov Decision Process (MDP), a special kind of sequential graphical model.

Question 9

What components define the progression in a Markov Decision Process (MDP)?

Answer 9

An MDP consists of a sequence of input actions Zt that lead to observed states St where each state has an associated deterministic reward R(St).

Question 10

How are rewards and utility defined in a Markov Decision Process?

Answer 10

Each state St has a deterministic reward R(St). The utility U of a sequence of states (S1,…,St) is also a deterministic function of these states

Question 11

What limitation do standard deep networks have when dealing with sequential data?

Answer 11

Standard deep networks do not take the ordering of data into account.

Question 12

How do Recurrent Neural Networks (RNNs) differ from standard deep networks?

Answer 12

It treats the hidden layer from the previous stage as an additional input to the current stage, enabling them to process sequential data

Question 14

What role does the hidden state Zt-1 play in RNNs?

Answer 14

The value of the previous hidden state Zt-1 is included in computing the current hidden state Zt, allowing the network to retain memory of past inputs.

Question 15

How do RNNS handle the order of inputs?

Answer 15

RNNS explicitly incorporate order dependence into their structure, allowing them to model temporal or sequential relationships in data.

Question 16

What an example of RNN?

Answer 16

Stock prediction

Question 17

What is the goal of using an RNN for stock prediction?

Answer 17

The goal is to predict the next day’s stock price given the prices from the previous N days (e.g N = 3)

Question 18

What are the inputs and key parameters in an RNN for stock prediction?

Answer 18

Input are daily stock prices (plus a bias). The parameters include weight w0,w1, recurrent weight u1, and output weight v0,v1, which are shared across time steps.

Question 19

How are the hidden and output values computed in a stock prediction RNN?

Answer 19

Each hidden state 𝑧𝑡 = 𝑓(𝑤0 + 𝑤1𝑥𝑡 + 𝑢1𝑧𝑡−1), and the output y = 𝑓(v0 + v1Zt, using the same weights across time steps

Question 20

What is a major challenge when training RNNs and what are the solutions?

Answer 20

RNNs suffer from the vanishing or exploding gradient problem, Solution include LSTM networks, and Tranformers