W7 Multi Agent RL

Question 1

Why is there so much interest in multi-agent reinforcement learning?

Answer 1

multi-agent is much more realistic because in the real world we also have multiple agents

Question 2

What is one of the main challenges of multi-agent reinforcement learning?

Answer 2

1.partial observability
2.nonstationary environments
3.large state space

Question 3

What is a Nash strategy?

Answer 3

when the agent is guaranteed to do no worse than tie against any other opponent strategy

Question 4

What is the NAsh equilibrium?

Answer 4

a situation where no agent has anything to gain by changing its own strategy (minimax). The agent does not try to exploit the opponent strategy’s flaws, it just wins when the opponent makes mistakes

Question 5

What is a Pareto Optimum?

Answer 5

the best possible outcome for us where we do not hurt others, and others do not hurt us. It is a cooperative strategy.

Question 6

what is the Pareto efficient solution?

Answer 6

the situation where no cooperative agent can be better off without making at least one other agent worse off

Question 7

In a competitive multi-agent system, what algorithm can be used to calculate a
Nash strategy?

Answer 7

Counterfactual Regret Minimization (CFR)

Question 8

What makes it diffcult to calculate the solution for a game of imperfect information?

Answer 8

it increases the size of the state space, and computing the unknown outcomes quickly becomes unfeasible

Question 9

Describe the 1) Prisoner’s dilemma and the 2) iterated Prisoner’s dilemma.

Answer 9

1) if both prisoners confess they both get 5 years in prison
if both stay silent they both get 2 years in prison
if I confess and the other stays silent, I walk free
if I stay silent and the other confesses, I get 10 years
this is an example of mixed behaviour

2) for multiple rounds of the Prisoner’s dilemma a tit for tat strategy works best: in the first round you play cooperative, after that you play whatever the opponent did in the previous round

Question 10

Name two multi-agent card games of imperfect information

Answer 10

poker, blackjack, bridge

Question 11

What is the setting with a heterogeneous reward function usually called?

Question 12

What is regret?

Answer 12

the regret of an action is the amount of reward that is missed by an agent for not choosing the actions with the highest payoff

Question 13

Name three kinds of strategies that can occur a multi-agent reinforcement learning.

Answer 13

CFR
evolutionary strategies
cooperative strategies

Question 14

Name two solution methods that are appropriate for solving mixed strategy games.

Answer 14

evolutionary methods and cooperative methods

Question 15

What AI method is named after ant colonies, bee swarms, bird blocks, or fish schools? How does it work in general terms?

Answer 15

swarm computing

focuses on emerging behavior in decentralized, collective, self-organized systems. Introduces forms of communication between agents

cooperation and survival of the group (Pareto)

Question 16

Name two solution methods that are appropriate for solving mixed strategy games

Answer 16

evolutionary methods and cooperative methods

Question 17

Describe the main steps of an evolutionary algorithm.

Answer 17

inspired by bio-genetic processes of reproduction: mutation, recombination, selection. repeat:
1. Evaluate the fitness of each individual of the population
2. Select the fittest individuals for reproduction
3. Through crossover and mutation generate new individuals
4. Replace the least fit individuals by the new individuals

focus on competition and survival of the fittest (Nash)

Question 18

Describe the general form of Hide and Seek and three strategies that emerged from the interactions of the hiders or seekers.

Answer 18

Collaboration

Question 19

what is population-based training?

Answer 19

it combines evolutionary ideas with RL ideas

teams compete against each other and if a team learns good behaviour they survive: team-based learning of better and better behaviour