Map-Reduce

Question 1

Objective

Answer 1

Sequentially read a lot of data

Question 2

Phases

Answer 2

Map phase
Group by key
Reduce phase
Write the result

Question 3

Map

Answer 3

read the input and produce key,value pairs
For each work output its count

Question 4

Sort & Shuffle

Answer 4

performed by the system

Question 5

Reduce

Answer 5

collect values with the same key and produce

Question 6

What is programmer responsible for

Answer 6

• Map function
• Reduce function

Question 7

What is the MapReduce system responsible for

Answer 7

• Partitioning the input data
• Scheduling the program’s execution across a set of machines
• Performing the sort by key & shuffle step
• Handling machine failures
• Managing required inter-machine communication

Question 8

Master node

Answer 8

Master node coordinates the
execution:
Task status: (idle, in-progress,
completed)
Idle tasks get scheduled as
workers become available
When a map task completes, it
sends the master the location and
sizes of its intermediate files, one
for each reducer
Master pushes this info to
reducers
Master pings workers periodically
to detect failures

Question 9

Worker node

Answer 9

Worker node performs
map or reduce tasks, as
requested by the
coordinator.

Question 10

Map worker failure

Answer 10

Upon detection of the failure
of a worker, map tasks
restarted in different worker

Question 11

Reduce worker failure

Answer 11

Reduce task is restarted in
other worker

Question 12

Stragglers (slow workers)

Answer 12

If a task is taking too long to
complete, it is launched in
other worker. First result used.

Question 13

Master failure

Answer 13

MapReduce task is aborted
and client is notified