HNN

Question 1

What is a Hopfield Neural Network? Primary purpose?

Answer 1

A type of recurrent neural network (RNN) where connections are symmetric.
Primary purpose: Autoassociative Memory (content-addressable memory).
It stores a set of “fundamental memories” (patterns). When presented with a noisy or incomplete version of a stored pattern, it can retrieve (converge to) the closest stored pattern.

Question 2

Hopfield Network: Key architectural features?

Answer 2

Single layer of neurons.
Neurons are fully interconnected (each neuron connected to every other neuron), but no self-connections (w_ii = 0).
Connections are symmetric (w_ij = w_ji).
Neurons are typically binary (0,1) or bipolar (-1,1).
It is a recurrent ANN.

Question 3

Hopfield Network: How are patterns stored

Answer 3

Patterns are stored by setting the connection weights (w_ij).
Hebbian Learning Rule (generalized): If two neurons are often active together in the patterns to be stored, their connection weight is strengthened.
For bipolar patterns s of length N: w_ij = (1/N) * Σ (s_p_i * s_p_j) (sum over all patterns p). If i=j, then w_ii = 0. (Simplified concept: w_ij = Σ (pattern_i * pattern_j) for bipolar, then set w_ii = 0).
The lecture notes (pg 19) use wij = Σ (2pi-1)(2pj-1) for binary training data (converting binary 0/1 to bipolar -1/1 implicitly during weight calculation) and wij = Σ pipj for bipolar training data. Both with wii=0.

Question 4

How are states updated to retrieve a pattern?

Answer 4

Given an initial (possibly noisy) input pattern.
Neurons update their states iteratively.
Update methods:
Asynchronous: Neurons update one at a time (randomly or sequentially). This is generally preferred as it guarantees convergence to a stable state (local minimum of energy function). (Test II 2023 Q3 correct answer)
Synchronous: All neurons update their states simultaneously based on the previous state of the network. Can lead to oscillations for some patterns.
Update Rule (for neuron i):
Calculate weighted sum of inputs: net_i = Σ (w_ij * s_j) (sum over all other neurons j).
Apply activation function: s_i(new) = 1 if net_i > θ_i (threshold), s_i(new) = 0 (or -1) if net_i < θ_i. (If net_i = θ_i, state often remains unchanged). Threshold θ_i is often 0.

Question 5

Hopfield Network: What is the concept of an “Energy Function”?

Answer 5

A Hopfield network has an associated “energy function” (Lyapunov function).
During the state update process (especially asynchronous), the network dynamics tend to decrease this energy until it reaches a stable state (a local minimum of the energy landscape).
Stored patterns correspond to these local minima.

Question 6

Hopfield Network: Key properties and limitations?

Answer 6

Properties: Associative memory, pattern completion, noise reduction.
Limitations:
Storage Capacity: Limited (approx. 0.14N patterns for N neurons). Overloading leads to poor recall and spurious states.
Spurious States: Can converge to states that are not among the originally stored patterns.
Can get stuck in local minima if the initial pattern is too far from a stored one or if patterns are too similar.
It is NOT a sparsely connected network