Week 10 - Neural Networks, Artificial Life & Cellular Automata

Question 1

How can neural networks be adapted for dynamic/temporal/sequential data?

Answer 1

Using a shifting time window to predict the future from recent past - recurrent neural networks

Question 2

Describe the structure of a recurrent neural network (RNN). Give an example of an RNN.

Answer 2

Similar to MLPs but also have connections from activity in a previous time step, so have a short term memory

Many possible types, e.g. Hopfield and Jordan-Elman

Question 3

What might backpropagation though time lead to?

Answer 3

No convergence and developing chaotic behaviour

Question 4

What are the aims of supervised learning and unsupervised learning?

Answer 4

Supervised - find a function fitting examples
Unsupervised - find regularities in inputs

Question 5

Name 3 types of unsupervised learning.

Answer 5

Clustering, density estimation, dimensionality reduction

Question 6

What do SOMs (Self-Organising Maps) aim to do?

Answer 6

Learn to map points from a high-dimensional space to a low-dimensional (discrete) space (2D or 3D) in a way to preserve topological properties (spatial relations) via unsupervised learning

Question 7

What do SOMs (Self-Organising Maps) aim to do?

Answer 7

Learn to map points from a high-dimensional space to a low-dimensional (discrete) space (2D or 3D) in a way to preserve topological properties (spatial relations) via unsupervised learning

Question 8

What are some working assumptions of SOMs?

Answer 8

Input data that belongs to the same class shares some common features

SOM will possibly be able to identify these key features across a number of data points

SOM will be able to organise/order meaningfully the input data according to a given 2D/3D structure

Question 9

How does SOM work?

Answer 9

There are as many input nodes as “features” in the data

Input nodes are connected to a “map” of interconnected nodes

Every node in the output map is connected to every input node via weighted edges

Every input pattern is a point in a high-dimensional space

Every input is made to correspond to a node in an output map via a competitive process among nodes on the output space

The winner is the node whose weights have the smallest (Euclidean) distance to input pattern

Once the network has been trained, nodes next to each other in output space will be related

The “clusters” then have to be labelled (often manually) through observation of the input data and output node

Question 10

What are some applications of SOM?

Answer 10

Speech recognition (originally used)

Data visualisation

Pattern recognition

Speech analysis

Industrial and medical diagnostics

Data mining

Question 11

What is artificial life (a-life)?

Answer 11

The study of man-made systems that exhibit behaviour characteristics of natural living systems

Question 12

What does a-life’s bottom-up approach mean?

Answer 12

The opposite approach to biology - works from molecules up to organisms

Question 13

What are the 3 levels a-life is conducted at?

Answer 13

Wetware (using bits from biology (e.g. RNA, DNA) to build new types of organisms (Synthetic Biology)

Hardware (for instance, autonomous and collective robots (Swarm Robotics)

Software (simulating biological systems)

Question 14

What are the 2 philosophies of a-life?

Answer 14

Weak a-life: computer simulations are just that, simulations and investigations of life

Strong a-life: life is not just restricted to a carbon-based chemical process - life can be “created” in silico

Question 15

Who and when was cellular automata originally devised by?

Answer 15

Stan Ulam and John von Neumann in the late 1940s

Question 16

What is cellular automata?

Answer 16

Originally devised as a method of representing a stylised universe, with rules (e.g. laws of thermodynamics) acting over the entire universe

In terms of a-life, it’s a demonstration of emergence: complex global behaviour emerges spontaneously from local interactions following simple rules

Question 17

What does cellular automata consist of?

Answer 17

An automaton consists of a grid/lattice of cells each of which can be in a (small and finite, normally binary) number of states

Question 18

What can an automaton be?

Answer 18

1D (just a line of cells)

2D (a square of cells)

3D+ (there is no limit to dimension numbers)

Toroidal (cells “wrap around” to the other side)

Question 19

Describe the execution of cellular automata (CA).

Answer 19

The CA “runs” by changing the states of the cells by the state transition rules

These state transition rules depend on the state of the cell and its neighbours

Every cell in the automaton has its rules applied before the automaton is updated

Each timestep of the automaton can be seen as a system configuration for that particular snapshot in time

Question 20

What are 2 popular CA neighbourhoods?

Answer 20

von Neumann (1 square to the top, left, right and down)

Moore (1 square to diagonals too)

Question 21

What are the rules of Conway’s Game of Life?

Answer 21

If a cell is off (state 0), and exactly 3 of its neighbours are on (state 1), then that cells becomes on (state 1) in the next timestep, otherwise it remains off

If a cell is on and either two or three of its neighbours are on then the next timestep, that cell remains on, otherwise it is turned off

Question 22

Name some stationary results of Conway’s Game of Life.

Answer 22

Block, beehive, loaf, boat

Question 23

Name some oscillating results of Conway’s Game of Life.

Answer 23

Blinker, toad, beacon, glider, lightweight spaceship

Question 24

What are 3 important properties of CA?

Answer 24

Localism (states are updated based on the properties of the neighbourhood)

Parallelism (the state of every cell is updated in parallel)

Homogeneity (the same set of rules is applied across the automaton)

Question 25

What is Wolfram’s proposed classification scheme criteria for CAs?

Answer 25

1. Evolution leads to a homogeneous state
2. Evolution leads to a set of separated simple stable or periodic structures
3. Evolution leads to a chaotic pattern
4. Evolution leads to complex localised structures, sometimes long lived

Question 26

What are some applications of CAs?

Answer 26

Modelling spatial processes (e.g. forest fires, disease spread)

Modelling physical processes (e.g. crystal formation, thermodynamics)

Modelling biological processes (e.g. pattern formation, self-replication, enzyme kinetics)

Solving computational problems (e.g. random number generators, ciphers)

Parallel processing architectures (e.g. systolic arrays, connection machine)

Question 27

What is the learning algorithm for SOMs?

Answer 27

Initialise network (set weights to small random values, define (large) neighbourhood size)

Sample random input vector (x(t)) from training set and present it to the network

Compute Euclidean distance of each output node j from the current input

Find the node closest to the input (BMU - Best Matching Unit)

Adjust the weights of BMU and its neighbours

Repeat until output map stops changing

Question 28

What are the variables that change over time in a SOM? How do they work?

Answer 28

Learning rate - determines the magnitude of weight updates during learning. It controls the speed at which the network adapts to the input patterns. Typically, the learning rate starts high and gradually decreases over time. This reduction helps the network converge to a stable and refined representation of the input data.

Neighbourhood radius - defines the extent of influence that the BMU has on the neighbouring neurons during weight updates. Initially, the neighbourhood radius is set to encompass the entire map, allowing global exploration. As the learning progresses, the radius gradually decreases, focusing on smaller local regions of the map. This reduction in neighbourhood radius promotes fine-tuning and better differentiation of clusters or patterns within the input data.