2.6 Side Effects and Reward Hacking

Question 1

Side effects and reward hacking

Answer 1

Can result in AI-based systems generating unexpected, and even harmful results

Question 2

Negative side effects can result when

Answer 2

The designer of an AI-based system specifies a goal that ‘focuses on accomplishing some specific tasks in the environment but ignores other aspects of the (potentially very large) environment, and thus implicitly expresses indifference over environmental variables that might actually be harmful to change’

Question 3

Example of negative side effect

Answer 3

A self-driving car aiming for ‘fuel-efficient and safe’ travel may achieve the goal, but passengers become annoyed at excessive time taken

Question 4

Reward hacking can result from

Answer 4

An AI-based system achieving a specified goal by using a ‘clever’ or ‘easy’ solution that ‘perverts the spirit of the designer’s intent’

Question 5

Reward hacking effectively means

Answer 5

The goal can be gamed

Question 6

Widely used example of reward hacking

Answer 6

An AI-based system teaching itself to play an arcade computer game with the goal of ‘highest score’ hacks the data record that stores the highest score, rather than playing the game to achieve it