11 - Rational Decisions Over Time

Question 1

Difference between search problems and MDPs

Answer 1

Search problems aim to find an OPTIMAL SEQUENCE. MDPs aim to find an OPTIMAL POLICY.

Question 2

Optimal policy maximizes the ________.

Answer 2

EXPECTED UTILITY

Question 3

Types of utility functions

Answer 3

Additive & Discounted utility functions

Question 4

What is a Markov Decision Process (MDP)?

Answer 4

MDP = Markov Chain + Actions + Rewards

Question 5

What is a Partially Observable Markov Decision Process (POMDP)?

Answer 5

POMDP = Hidden Markov Model + Actions + Rewards

Question 6

POMDPs are generalizations of a ________ without ________.

Answer 6

POMDPs are generalizations of a MDP without DIRECT STATE OBSERVATIONS.

Question 7

Advantage of MDPs

Answer 7

Easy to compute and specify

Question 8

Disadvantage of MDPs

Answer 8

Assumes perfect knowledge of the state

Question 9

Advantage of POMDPs

Answer 9

Allows for learning and uncertainty

Question 10

Disadvantage of POMDPs

Answer 10

Computationally expensive

Question 11

What problems can be modeled as Markov Decision Processes?

Answer 11

Sequential Decision Problems in uncertain discrete environments

Question 12

Utility of a state sequence is the sum of all ________.

Answer 12

REWARDS OVER THE SEQUENCE

Question 13

MDP vs. POMDP - What is more difficult to solve?

Answer 13

POMDPs

Question 14

How are POMDPs solved?

Answer 14

Solved by conversion to an MDP

Question 15

Value iteration vs. policy iteration - What converges faster and why?

Answer 15

Policy iteration since the policy might be optimal without knowing the exact utilities of each state.