Distributed Machine Learning

Question 1

What are the 7 different communication patterns?

Answer 1

1. Push
2. Pull
3. Broadcast
4. Reduce
5. All-reduce
6. Wait
7. Barrier

Question 2

What is the push communication pattern?

Answer 2

Machine A send data to Machine B

Question 3

What is the pull communication pattern?

Answer 3

Machine B requests data from Machine A

Question 4

What is the broadcast communication pattern?

Answer 4

Machine A sends data to many machines.

Question 5

What is the reduce communication pattern?

Answer 5

Compute some reduction of data from multiple machines and materialise the result on one machine.

Question 6

What is the all reduce communication pattern?

Answer 6

Compute come reduction of data on multiple machines and materialise the results on all those machines.

Question 7

What is the wait communication pattern?

Answer 7

One machine pauses its computation and wait on a signal from another machine.

Question 8

What is the barrier communication pattern?

Answer 8

Many machines wait until all those machines reach a point in their execution, then continue from there

Question 9

What is the key principle in distributed computing?

Answer 9

Overlapping computation and communication.

Question 10

How does Stochastic Gradient Decent work with All reduce?

Answer 10

1. The SGD equation is split into M sub equations
2. Each of M machines is assigned a summation
3. After all gradients are computed, the outer sum can be performed using all-reduce
   4.After the all-reduce the whole sum is present on all machines and can be used to update model parameters

Question 11

What is the advantage of SGD with All-reduce?

Answer 11

The algorithm is easy to reason about as it’s equivalent to minibatch SGD. The same hyperparameters can be used. The algorithm is easy to implement.

Question 12

What is the disadvantage of SGD with All-reduce?

Answer 12

While the communication for the all-reduce is happening, the workers are idle. We are not overlapping computation and communication.

Question 13

How is parallel k-means implemented?

Answer 13

Parallel k-means is split into 3 phases:
1. Map
2. Combine
3. Reduce

Question 14

How is the map step done in parallel k-means?

Answer 14

1. Compute distances between the points and k centroids
2. Assign points to the clusters
3. send the intermediary data to combiner

Question 15

How is the combine step done in parallel k-means?

Answer 15

1. Compute centroid of each cluster
2. Send local sums of the value across dimensions and the number of samples

Question 16

How is the reduce step done in parallel k-means?

Answer 16

1. Test convergence
2. Update the cluster centroids
3. Return to Map step until convergence reache