8 - Recurrent Neural Networks

Question 1

What is the problem with using a time as space model?

Answer 1

-expensive
-how far back in time should you go?
-semantically not correct

Question 2

How to deal with problems in time as space model?

Answer 2

allow activity to reverberate through the network by using RECURRENT CONNECTIONS -> dynamic network activity

Question 3

What analogy can you use to explain reverberating activity?

Answer 3

throwing stone in puddle and wave of puddle can tell you where input(stone) came from

Question 4

In Elman Networks specifically, what is the purpose of feedback weights?

Answer 4

connect the hidden layer to the context layer

Question 5

Elman Network - What sort of activity is seen between hidden layer and context layer?

Answer 5

reverberating activity

Question 6

Why cant a perceptron (single rate neuron model) perform an XOR function?

Answer 6

output is not linearly separable

Question 7

What can solve the XOR problem?

Answer 7

with added layers of neurons !!

Question 8

Are there any feedback connections in the feed forward model ? (outputs which feedback into itself)

Answer 8

noooooooooooooo (called feedforward rate networks for a reason)

Question 9

What is the added additional layer in the Elman Network compared to a Feedforward Network?

Answer 9

context layer

Question 10

What tool does learning sequence information use to predict the sequential XOR INPUT?

Answer 10

uses backpropagation

Question 11

When the error is reduced does the ability of the network to predict the next letter/number in the sequence increase or decrease?

Answer 11

increases

Question 12

Word with similar semantic have similar or different pattern of activity? (with predicting letter and level or error)

Answer 12

similar semantics and similar patterns

Question 13

What does adding recurrent connections between layers (reverberation activity) allow?

Answer 13

-introduces a new kind of ‘memory’ capacity which enables networks to represent temporal structure in the sequences of training patterns

Question 14

Why can you read a text which has letters omitted in the words?
What sort of memory is this called?
What network is this memory seen in?

Answer 14

because there is a pattern in language which the brain has stored (Elman)
-associative memory
-Hopfield Network

Question 15

What is cell assembly in memory?
What is a cell assembly able to do?

Answer 15

-a selection of neurons connected with increased synaptic weights
-this increase in synaptic weights of connections between neurons allows you to store an item in memory

Question 16

How do cell assemblies contribute to associate memory? (Hebbian)

Answer 16

-items stored via creation of cell assembly
-associated items can be recalled due to activation of cell assembly (this is due to the increase in synaptic weights between neurons in Hebbian Network)

Question 17

What learning is this: neurons that fire together, wire together?
What does this mean?

Answer 17

Hebbian Learning: neurons that are active together results in strengthened connections between them

Question 18

In ferromagnets, is the direction of the atom’s spin independent on the neighbouring atom’s spin?

Answer 18

no they are not independent - they can have strong or weak interactions which influence eachother

Question 19

If a ferromagnet atom gets excited and spins so to point in the opposite direction, what happens afterwards if you apply the Hebbian dynamics eqn.?
How does this relate to Hebbian Learning/Network?

Answer 19

-atom flips back to original orientation
-an example of recalling memory is when the atom flips back as the original orientation is the ‘stored’ memory

Question 20

What is the definition of a Hopfield Network?

Answer 20

a recurrent network with symmetric weights and no self-connections

Question 21

In the definition of a Hopfield Network, what does it mean when the weights are symmetric?

Answer 21

the connection from neuron j to i is equal to the connection between i to j

Question 21

In a Hopfield Network, what is form is the output?
What is the output in a Hopfield Network?

Answer 21

-either active +1 or inactive -1
-output OF EACH UNIT is weighted sum of the inputs pushed through a step function to generate +1 or -1

Question 21

Do individual neurons interact with themselves in a Hopfield network?

Answer 21

non there are no self-connections

Question 22

How are memories stored as in a Hopfield Network?

Answer 22

stored as low-energy states of the dynamics

Question 23

Finish the sentence: When an axon of cell j repeatedly or persistently takes part in firing cell i, -What networks is this rule seen in?

Answer 23

then j’s efficiency as one of the cells firing i is increased -it is the basis of Hebbian Learning but it is also used in Hopfield Network

Question 24

What does yi/yj represent in Hopfield Network? " xi/xj " wij

Answer 24

-the activity of the network -the pattern we want to store -the weights

Question 25

What is the dynamics equation for the Hopfield Network? What does the 𝑗≠𝑖 mean in this eqn.?

Answer 25

𝑦𝑖 (𝑡+1)=𝑠𝑖𝑔𝑛(∑(𝑗≠𝑖) 𝑤𝑖𝑗 𝑦𝑗 (𝑡) ) weighted sum of all neurons connections apart from connecting to itself so 𝑗≠𝑖

Question 26

What is the weight between two neurons in Hopfield Network equal to?

Answer 26

Energy: 𝐸𝑖𝑗 = −𝑤𝑖𝑗 𝑦𝑖 𝑦𝑗

Question 27

Why do we set the weights wij in a Hopfield Network?

Answer 27

-to store a pattern x (memorisation)

Question 28

How do we store and recall a pattern in a Hopfield Network? (two steps)

Answer 28

-set the weights to wij=xixj to store a pattern 𝒙 (memorisation) -start the activity at 𝒚 that is similar to 𝒙, the dynamics moves 𝒚 toward 𝒙 (recall)

Question 29

How do you set the weights in memorisation/pattern storage in a Hopfield Network?

Answer 29

1) Start with 𝑤_𝑖𝑗=0 2) To memorise pattern 𝒙^𝑝, update weights: 𝑤𝑖𝑗←𝑤𝑖𝑗+𝑥𝑖^𝑝 𝑥𝑗^𝑝 3) Repeat step (2) for every other pattern

Question 30

In a Hopfield Network, what does it mean when connections enforce correlation?

Question 31

How do you recall a pattern in a Hopfield Network? (with associative memory eg. you see an incomplete letter/pattern M which is similar/associated with the stored memory of M (xp)

Answer 31

-start with network activity at y(1) -For each neuron 𝑖 update activity by 𝑦𝑖(𝑡+1)=𝑠𝑖𝑔𝑛(∑(𝑗≠i) 𝑤𝑖𝑗𝑦𝑗 (𝑡)) until 𝑦𝑖 (𝑡+1)=𝑦𝑖 (𝑡) for every 𝑖 -This “stable” activity will match the pattern 𝑥 that 𝑦(1) is most similar to.

Question 32

Hopfield Network: Why are memories stored as patterns which are random/abstract concepts in the brain? (instead of as images)

Answer 32

to avoid interference as images would be too similar

Question 33

What are attractor states in Hopfield Networks? How are the involved in recall? What are the similar to?

Answer 33

-where the pattern is stored -during recall the system is push towards the attractor state to get original stored pattern -fixed points

Question 34

What do Hopfield Dynamics do to energy? What is the eqn. for this?

Answer 34

-minimises energy as you go toward attractor states (in a well, like a fixed point) 𝐸𝑖𝑗 = − 𝑤𝑖𝑗 𝑦𝑖 𝑦𝑗

Question 35

When do spurious attractors occur in Hopfield Networks? What are they?

Answer 35

when a network is trying to store too many memories, spurious attractors will appear -attractors which are very hard to reach compared to desired attractors (have shallow energy wells)

Question 36

What can Hopfield Networks be trained to do?

Answer 36

trained to store memories as fixed-point attractor states

Question 36

If the neuron is in a learned pattern state what is the energy equal to? " not is a learned pattern state? Why is it this way around?

Answer 36

E = -1 E = +1 -because the energy always is minimised when moving in the direction of the attractor state

Question 37

What type of learning do attractors in Hopfield Networks employ?

Answer 37

Hebbian Learning

Question 38

How does Elman introduce networks that introduce time?

Answer 38

by introducing recurrent connections from the hidden layer to the context layer

Question 39

How are memories stored in a Hopfield Network?

Answer 39

As fixed-point attractor states

Question 40

How are neurons connected in a Hopfield network?

Answer 40

Neurons are connected to all other neurons in the network but not themselves, and these connections are symmetric.

Question 41

What is the output of a Hopfield Network defined as? ( include what it generates)

Answer 41

output of each unit (yi) = is a weighted sum of the inputs pushed through a step function to generate +1 or -1

Question 42

In the Hopfield model, how are memories stored?

Answer 42

random patterns (M) of binary weights corresponding to a stored concept -not pictures

Question 43

In terms of attractor states, what is the difference between memorisation and recall in the Hopfield Model?

Answer 43

Memorisation is creating the attractor states, recall is reaching an attractor state

Question 44

In the Hopfield Network Model, how are memories recalled? What do x and y represent?

Answer 44

-Getting y to iterate until it matches the pattern x correctly -y is the process of recall (activity of the network) x is the stored pattern in the brain/memory (the correct pattern that y is trying to reach)

Question 45

Hopfield model, in memory recall, when is a an attractor state (fixed point of model) reached?

Answer 45

when the pattern of y activity is the same as x (after all the iterations)