Lecture 2: The world of parallelism

Question 1

What is the trend in GPU and CPU usage?

Answer 1

The number of cores is increasing

Question 2

What is Flynn’s Taxonomy?

Answer 2

Classification system for computer architectures based on number of instruction and data streams they can process simultaneously.

Question 3

What are the categories of Flynn’s Taxonomy?

Answer 3

SISD: Single Instruction, Single Data
SIMD: Single Instruction, Multiple Data
MISD: Multiple Instruction, Single Data
MIMD: Multiple Instruction, Multiple Data (Chip MPs)

Question 4

Define the four categories of Flynn’s Taxonomy

Answer 4

SISD: One instruction executed at a time
processing one data element at a time. e.g. Traditional single processors

SIMD: One instruction executed at a time
operating on multiple independent streams of data e.g. Vector processors (1970s), Vector units (MMX, SSE, AVX), GPUs

MIMD: Multiple sequences of instructions executed independently
each one operating on a different stream of data e.g. Chip Multiprocessors

SIMD: Multiple instruction streams but with the same code on
multiple independent streams of data e.g. Data Parallel machines built from independent processors

Question 5

Describe ineterconnections and communication in parallelism

Answer 5

1. Between cores
2. Between cores and memory

The way the connections are affects the type of computations

Question 6

Inerconnections in Chip MPs Advantages and Disadvantages

Answer 6

Advantages: Faster than traditional interconnects so lower cost
Disadvantages: Limited silicon and power for network

Question 7

What is a grid?

Answer 7

1. Direct link to neighbours
2. Private on-chip memory
3. Staged communication with non-neighbours
4. NxN grid worst case: 2*(N-1) steps

Question 8

What is a torus?

Answer 8

1. Direct link to neighbours
2. Private on-chip memory
3. More symmetrical, more paths, shorter paths
4. NxN grid worst case: N steps
5. More wires, complex routing

Question 9

What is the smallest kind of on-chip interconnect?

Answer 9

A grid

Question 10

In what interconnect do all cores connect four neighbors?

Answer 10

Torus

Question 11

Which interconnects are suitable for smaller and which for larger systems?

Answer 11

Grid -> Smaller
Torus -> Larger
Bus -> Smaller

Question 12

Whats a key property of Torus?

Answer 12

They can be generalized further to multiple dimensions:
2D torus 2D grid →
folded 4 neighbours →
3D torus 3D grid →
folded 6 neighbours →
4D torus 4D grid →
folded 8 neighbours →
CMPs rarely go above 2D

Question 13

Which inerconnect is relied on by many multiprocessors?

Answer 13

A bus, partially or fully

Question 14

What is a bus?

Answer 14

1. All cores to all cores
2. Simple to build
3. Constant latency
4. Memory can be oranized in any way: private to each core or shared between cores
5. Time-shared bus (disadvantage)
   → complexity, lesser bandwidth (fraction of that of grid)
6. Very long wires (to connect all these cores)
   → area, routing, power, slow

Question 15

For a large number of cores what would be the main bottleneck?

Answer 15

The bus

Question 16

Which topologies are more suitable for larger systems (scalable)?

Answer 16

1.Trees
2. Hierarchical (Crossbars,
Hypercubes, Rings,
MIN, etc)

Important for high core count

Question 17

What type of switching is in scalable topologies? Desrcibe it.

Answer 17

Packet switching: Dividing data into small packets for efficient transmission across a network. Can follow different paths to the destination and arrive out of order.

Question 18

What is a reason for discontuity in core count growth?

Answer 18

Interconnection. Larger counts require more complex network topologies.

Question 19

What is shared memory? Describe its hardware and software view.

Answer 19

Accessible from every part of the computation.

Hardware: Memory connected to all cores

Software: Global. Accessible from all threads (Reads/Writes)

Question 20

What is distributed memory? Describe its hardware and software view.

Answer 20

Accessible from only one part of the computation.

Hardware: Memory connected to only one core

Software: Local. Accessible only by the owning thread (Message passing)

Question 21

What is the software view referred to?

Answer 21

Programming model

Question 22

What are the types of programming model?

Answer 22

Serial globally accessible memory →
(sw restrictions may apply)

Data Sharing globally accessible memory →
(sw equivalent of Shared Memory)

Message Passing (distributed) thread-owned memory →
(sw equivalent of Distributed Memory)

Question 23

True or False

Programming model = Memory organisation

Answer 23

False

Question 24

Match the following memoery with programming model

1. Shared memory
2. Distributed memory

A. Message passing
B. Data sharing

Answer 24

1. B
2. A

Question 25

How efficient is simulating data sharing on distributed memory?

Answer 25

Slow

Question 26

How efficient is message passing on shared memory?

Answer 26

1. Fast but slower than Data Sharing 2. Extra traffic might impact bandwidth

Question 27

Which memory is better for HW perspective?

Answer 27

Distributed Memory. 1. Easier implementation 2. Higher Bandwidth 3. Scales better (E.g. Super computers!)

Question 28

Which memory is better for SW perspective?

Answer 28

Memory Sharing. 1. Easier programming 2. Works with irregular communication

Question 29

What is one of the central conflicts of contemporary architecture?

Answer 29

Hardware is complex

Question 30

What are the software issues of complex hardware?

Answer 30

SW exposed to it (distributed) →Higher SW cost →Complicated code →Wasted energy & cycles

Question 31

What are the hardware issues of complex hardware?

Answer 31

HW hides it (shared memory) →Higher HW cost →Complicated design →Wasted energy & cycles

Question 32

What type of memory for chip mp?

Answer 32

Shared

Question 33

Where is distributed memory used?

Answer 33

Supercomputers

Question 34

What is the NxN worst case for torus and grid?

Answer 34

Torus -> N Grid -> 2*(N-1)