W8 Hierachical RL

Question 1

1) Why can hierarchical reinforcement learning (HRL) be faster?
2) Why can hierarchical reinforcement learning be slower?

Answer 1

1) HRL simplifies problems through abstraction, the temporal abstractions increase sample efficiency
2) domain knowledge needs to be available, algorithmic complexity cost time, marco-actions

Question 2

Why may hierarchical reinforcement learning give an answer of lesser quality?

Answer 2

The macro-actions may skip over possible shorter routes, that the primitive actions would have found.

Question 3

Is hierachical reinforcement more general or less general?

Answer 3

More general
subtasks reduce brittleness due to overspecialization of policies. Policies become more general, and are able to adapt to changes in the environment more easily.

Question 4

What is the options framework?

Answer 4

Whenever a state is reached that is a subgoal, then, apart from following a primitive action (main policy), you can follow the option policy, a macro action consisting of a different subpolicy specially aimed at satisfying the subgoal in one large step. In this way macros are incorporated into the reinforcement learning framework.

Question 5

What is an option?

Answer 5

An option is a group of actions with a termination
condition.
Options take in environment observations and output actions until a termination condition is met.

Question 6

What are the three elements that an option consists of?

Answer 6

The initiation set 𝐼 ⊆ 𝑆 are the states that the option can start from
The subpolicy 𝜋 : 𝑆 × 𝐴 → [0, 1] internal to this particular option
The termination condition 𝛽 : 𝑆 → [0, 1] tells us if 𝜔 terminates in 𝑠

Question 7

What is a macro?

Answer 7

Macros are combinations of primitive actions, and their use can greatly improve the performance of the policy.

Question 8

What is intrinsic motivation?

Answer 8

An inner drive to explore.
Named so to contrast it with classic extrinsic motivation (the conventional RL reward signal).
Often related to model curiosity.

Question 9

what are the three elements of an option 𝜔?

Answer 9

initialization set I_𝜔: the states that the option can start from
subpolicy 𝜋𝜔 (𝑎|𝑠): internal to this particular option
terminal condition 𝛽𝜔 (𝑠): tells us if 𝜔 terminates in s

Question 10

How do multi agent and hierarchical reinforcement learning fit together?

Answer 10

agents often work together in teams or other hierarchical structure

Question 11

What is so special about Montezuma’s Revenge?

Answer 11

The reward signal is little. The agent needs to walk without the reward changing