Batch - Yarn and MapReduce

Question 1

YARN (Yet Another Resource Negotiator)

Answer 1

Resource management system designed to handle distributed computing

Question 2

YARN APIs

Answer 2

Request and work with cluster resources (not made by user code, but by framework itself!)

Question 3

Fundamental idea of YARN

Answer 3

Split functionalities of resource management and job scheduling.

Question 4

What makes Yarn scheduler a “pure scheduler”?

Answer 4

Doesn’t monitor application/job status
Doesn’t restart application/job on failure

Question 5

Application

Answer 5

Single job or DAG of jobs

Question 6

Applications Manager job

Answer 6

Accept job submissions, negotiate container for executing AMP, provide service for restarting if AMP fails

Question 7

FIFO scheduler

Answer 7

No configuration necessary, bad for clusters

Question 8

Capacity Scheduler

Answer 8

Fixed amount of capacity to each job

Question 9

Fair Scheduler

Answer 9

Balances available resources between running jobs

Question 10

Resource Manager (RM) (def) (2)

Answer 10

Ultimate authority allocating containers,
1. Accept job submissions from
client
2. Set up ApplicationsMaster (w/ initial container)

Question 11

Node Manager (NM)

Answer 11

A per-machine agent monitoring resource usage of containers and reports it to RM

Question 12

ApplicationsMaster (2)

Answer 12

Manage job lifecycle and request containers from RM

Question 13

Upon request from client, RM finds a NM to launch ______ in a ___________.

Answer 13

Application Master Process; container

Question 14

Container

Answer 14

Slice of computing resources, reports job status to AMP

Question 15

Data Locality (YARN)

Answer 15

Ensuring tasks are run as close to the data as possible

Question 16

4 Levels of Data Locality

Answer 16

1. Node-level
2. Rack-level
3. Data Center-level
4. Inter-data center

Question 17

MapReduce

Answer 17

A programming model that allows developers to write programs that can process large amounts of data in parallel across a cluster

Question 18

Map Phase

Answer 18

Dataset is partitioned into smaller chunks (input splits) and processed in parallel, turning data into key-value pairs

Question 19

Sort & Shuffle Phase

Answer 19

Data is sorted by key and shuffled (moved) in groups to reducers.

Question 20

Reduce Phase

Answer 20

Intermediate data is aggregated by reduce function

Question 21

Combiner

Answer 21

In some cases, a mini-reduce function used as an optimizer between Map and Reduce phases so less data is transferred.

Question 22

Combiner function must be _______________ and ______________

Answer 22

Associative (a + b) + c = a + (b + c) - grouping

Commutative a + b = b + c - order

Question 23

Map or Reduce task failure

Answer 23

Rescheduled on another node

Question 24

Map or Reduce node failure

Answer 24

All tasks on node rescheduled on another node
worst case: restart entire job

Question 25

5 Steps of MapReduce process

Answer 25

1. Input divided into fixed-size splits 2. User-defined map function for each record in split 3. Key-value pairs sorted by key and stored on disk 4. Sent to reducers that combine values of a given key 5. Results written onto DFS

Question 26

One global reduce task solution

Answer 26

originally, hadoop had one reduce task for an entire job regardless of data size

Question 27

One reduce task per CPU solution

Answer 27

number of reduce tasks based on number of CPU cores, but can cause an imbalance because some keys have more data than others

Question 27

Many reduce tasks per CPU solution

Answer 27

Reduce tasks > CPU cores, each task handling fewer keys for more balanced workload, but more overhead

Question 28

Rule of thumb for picking number of Reduce Tasks

Answer 28

Run for 5 minutes and produce at least one HDFS Block

Question 29

Partition function

Answer 29

Optional MapReduce function to determine how intermediate data is distributed to reduce tasks. Same keys go to same reducers, or aim for balanced workload.

Question 30

How does job get submitted in MapReduce?

Answer 30

Developer launches job on Java Virtual Machine (JVM) which then contacts YARN RM