Apache Hadoop

Question 1

What is Hadoop?

Answer 1

open-source software framework for distributed storage and processing of large data sets, using a clustered network of machines

Question 2

What are the key components of Apache Hadoop? (3)

Answer 2

1. HDFS
2. YARN
3. MapReduce

Question 3

what is a node?

Answer 3

a physical or virtual machine that is part of a Hadoop cluster

Question 4

What is a daemon?

Answer 4

a background process

Question 5

State the daemons related to YARN (computing) (3)

Answer 5

1. NodeManager daemon
2. ResourceManager daemon
3. JobHistoryServer daemon

Question 6

State the daemons related to HDFS (storage) (3)

Answer 6

1. NameNode daemon
2. DataNode daemon
3. SecondaryNameNode daemon

Question 7

Describe the characteristics of the leader in leader-follower architecture (4)

Answer 7

1. Aware of the follower nodes
2. Receives external requests
3. Decides which nodes execute what and when
4. Communicates with follower nodes

Question 8

Describe the characteristics of the follower in leader-follower architecture (2)

Answer 8

1. Acts as a worker node
2. Executes tasks that leader tells it to

Question 9

Which two nodes operate in a leader-follower architecture?

Answer 9

Leader: NameNode
Follower(s): DataNode

Question 10

What is HDFS?

Answer 10

Shared distributed storage among the nodes of the Hadoop cluster, tailored to map reduce jobs

Question 11

Where do daemons run?

Answer 11

On nodes

Question 12

What is the HDFS responsible for storing?

Answer 12

Input and output of MapReduce jobs

Question 13

How is data stored within the HDFS?

Answer 13

In blocks

Question 14

What is the default block size?

Answer 14

128MB

Question 15

How is the minimum parallelisation unit determined?

Answer 15

by the HDFS block size, e.g., mappers will work on a block

Question 16

Why is 128MB the ideal block size?

Answer 16

it balances parallelisation opportunity (favours smaller blocks) with data processing throughput (favours larger blocks)

Question 17

How does a file that is smaller than block size occupy the block?

Answer 17

It only occupies the same amount of disk space as the size of the file, so not the entire 128MB

Question 18

What is the purpose of the NameNode?

Answer 18

to manage the filesystem namespace, filesystem tree and metadata for all files and directories in the tree

Question 19

What is the purpose of the DataNodes?

Answer 19

store and retrieves blocks when instructed and to implement block caching for blocks which are frequently accessed

Question 20

Which node does the DataNode report to?

Answer 20

the NameNode

Question 21

Where is the data for the filesystem tree and the related metadata stored?

Answer 21

persistently on the local disk in the form of two files: the namespace image and the edit log

Question 22

What does the NameNode know about the files in the HDFS?

Answer 22

Which datanodes possess the blocks for a given file and where they are located (but not persistently)

Question 23

How many DataNodes are there per cluster?

Answer 23

at least one

Question 24

How many NameNodes are there per cluster?

Answer 24

only one

Question 25

What is the purpose of the HDFS SecondaryNamenode?

Answer 25

to store a backup copy of index table (communicates periodically with NameNode)

Question 26

What information does the NameNode keep relating to the blocks?

Answer 26

An index table with (all) the locations of each block

Question 27

What would happen if the machine running the NameNode was obliterated?

Answer 27

all the files on the filesystem would be lost since there would be no way of knowing how to reconstruct the files from the blocks on the datanodes

Question 28

How many SecondayNamenodes are there per cluster?

Answer 28

only one

Question 29

What is meant by the “move computation to data” principle with HDFS?

Answer 29

blocks are stored on certain machines, and the mapper and reduce function will both run locally on that machine without needing to move data between map and reduce processes

Question 30

Which feature of HDFS achieves the “move computation to data” principle?

Answer 30

Block replication

Question 31

Why are blocks replicated over the cluster?

Answer 31

for fault-tolerance purposes, spreading replicas among different physical locations to improve reliability

Question 32

What is the default number of replicas for each block?

Answer 32

3

Question 33

What is YARN?

Answer 33

Hadoop’s cluster resource management system

Question 34

what is the relationship between a job and a task?

Answer 34

a job usually consists of multiple tasks

Question 35

What are the Hadoop computation tasks? (3)

Answer 35

1. Resource management 2. Job allocation 3. Job execution/monitoring

Question 36

How would the estimation for the number of map and reduce tasks be calculated? (2)

Answer 36

Based on: 1. input dataset 2. job definition (defined by user)

Question 37

How can you calculate the number of mappers needed?

Answer 37

input size/split size

Question 38

What are the different schedulers available in YARN (3)

Answer 38

FIFO Capacity Fair

Question 39

Why is Hadoop not efficient with I/O? (2)

Answer 39

1. data must be loaded and written from HDFS 2. shuffle and sort incur long latency and produce large network traffic