7. Big Data - Hadoop Ecosystem

Question 1

DataNode function

Answer 1

Store data

Question 2

NameNode function

Answer 2

1. Store metadata 2. Know what DataNode each block is located

Question 3

Qual era o volume de dados no mundo em 2013

Answer 3

4.4 x 10^21

Question 4

Qual é o problema dos discos para processar grande volume de dados?

Answer 4

A velocidade de leitura de um disco de 1TB é de no máximo 100 MB/s

Question 5

O que é Hive?

Answer 5

SQL que roda com dados no HDFS

Question 6

What is Spark?

Answer 6

Processamento interativo na memória

Question 7

Solr

Answer 7

Search de dados no HDFS

Question 8

Qual é o primeiro passo do cientista de dados?

Answer 8

Definir bem a questão, o que busca responder com os dados.

Question 9

Fases do MapReduce

Answer 9

1. Map - filtra o que precisa 2. Reduce - junta os dados em um resumo

Question 10

MapReduce Jobs (components)

Answer 10

Input Data + MapReduce Program + Config Info

Question 11

O que são distributed filesystems?

Answer 11

Filesystems that manage the storage across a network of machines are called distributed filesystems.

Question 12

Quais são os componentes em mente no desenho do HDFS?

Answer 12

1. Very large files
2. Streaming data access
3. Commodity hardware

Question 13

Em quais situações o HDFS não se encaixa bem?

Answer 13

1. Low-latency data access 2. Lots of small files 3. Multiple writers, arbitrary file modifications

Question 14

Qual é o tamanho padrão do “block” do HDFS?

Answer 14

128 MB

Question 15

Para quantos servidores tipicamente é replicado um “block” de dados?

Answer 15

Três servidores

Question 16

Quais são as medidas de proteção do NameNode?

Answer 16

1. Realizar backup dos arquivos de Metadata do NameNode.
2. Rodar um “secondary NameNode”.
3. Instalar no modelo “Active/Standby” disponível a partir da versão 2.

Question 17

Quais são as etapas de inicialização do NameNode?

Answer 17

(i) loaded its namespace image into memory (ii) replayed its edit log, and (iii) received enough block reports from the datanodes to leave safe mode.

Question 18

Quanto tempo leva para o NameNode inicializar em um cluster grande?

Answer 18

On large clusters with many files and blocks, the time it takes for a namenode to start from cold can be 30 minutes or more.

Question 19

Quem é o responsável por gerenciar o processo de failover entre o NameNode ativo e passivo?

Answer 19

Failover controller. The transition from the active namenode to the standby is managed by a new entity in the system called the failover controller. There are various failover controllers, but the default implementation uses ZooKeeper to ensure that only one namenode is active. Each namenode runs a lightweight failover controller process whose job it is to monitor its namenode for failures (using a simple heartbeating mechanism) and trigger a failover should a namenode fail.

Question 20

Qual propriedade define a quantidade de DataNodes que um “block” é replicado?

Answer 20

dfs.replication = x

Question 21

Qual o comando para conseguir ajuda dos comandos do HDFS?

Answer 21

hadoop fs -help

Question 22

Qual a propriedade para habilitar a segurança do HDFS?

Answer 22

dfs.permissions.enabled

Question 23

Como funciona o processo de abertura de um arquivo no HDFS?

Answer 23

The client opens the file it wishes to read by calling open() on the FileSystem object, which for HDFS is an instance of DistributedFileSystem (step 1 in Figure 3-2). DistributedFileSystem calls the namenode, using remote procedure calls (RPCs), to determine the locations of the first few blocks in the file (step 2). For each block, the namenode returns the addresses of the datanodes that have a copy of that block. Furthermore, the datanodes are sorted according to their proximity to the client (according to the topology of the cluster’s network; see Network Topology and Hadoop). If the client is itself a datanode (in the case of a MapReduce task, for instance), the client will read from the local datanode if that datanode hosts a copy of the block (see also Figure 2-2 and Short-circuit local reads).

Question 24

What does it mean for two nodes in a local network to be “close” to each other?

Answer 24

Hadoop takes a simple approach in which the network is represented as a tree and the distance between two nodes is the sum of their distances to their closest common ancestor. Levels in the tree are not predefined, but it is common to have levels that correspond to the data center, the rack, and the node that a process is running on.

Question 25

Proximity factor steps

Answer 25

The idea is that the bandwidth available for each of the following scenarios becomes progressively less: Processes on the same node Different nodes on the same rack Nodes on different racks in the same data center Nodes in different data centers[32]

Question 26

How does the namenode choose which datanodes to store replicas on?

Answer 26

1. Hadoop’s default strategy is to place the **first replica** on the same node as the client (for clients running outside the cluster, a node is chosen at random, although the system tries not to pick nodes that are too full or too busy). 2. The **second replica** is placed on a different rack from the first (off-rack), chosen at random. 3. The **third replica** is placed on the same rack as the second, but on a different node chosen at random. 4. **Further replicas** are placed on random nodes in the cluster, although the system tries to avoid placing too many replicas on the same rack. Overall, this strategy gives a good balance among reliability (blocks are stored on two racks), write bandwidth (writes only have to traverse a single network switch), read performance (there’s a choice of two racks to read from), and block distribution across the cluster (clients only write a single block on the local rack).

Question 27

What is YARN?

Answer 27

Apache YARN (Yet Another Resource Negotiator) is Hadoop’s cluster resource management system. YARN was introduced in Hadoop 2 to improve the MapReduce implementation, but it is general enough to support other distributed computing paradigms as well.

Question 28

What services does YARN provide?

Answer 28

YARN provides its core services via two types of long-running daemon: 1. **resource manager** (one per cluster) to manage the use of resources across the cluster 2. **node managers** running on all the nodes in the cluster to launch and monitor containers.

Question 29

How does YARN run applications?

Answer 29

Question 30

How does HDFS detect file corruption?

Answer 30

1. Each datanode runs a DataBlockScanner in a background thread that periodically verifies all the blocks stored on the datanode. This is to guard against corruption due to “bit rot” in the physical storage media.

Question 31

What are the master daemons in Hadoop?

Answer 31

1. NameNode 2. Secondary NameNode 3. Resource Manager 4. History Server

Question 32

Is it acceptable to run the NameNode and Resource Manager in the same machine?

Answer 32

For a small cluster (on the order of 10 nodes), it is usually acceptable to run the namenode and the resource manager on a single master machine (as long as at least one copy of the namenode’s metadata is stored on a remote filesystem). However, as the cluster gets larger, there are good reasons to separate them.

Question 33

What do you need to do when your DataNode are installed in different racks?

Answer 33

If your cluster runs on a single rack, then there is nothing more to do, since this is the default. However, for multirack clusters, you need to: 1. map nodes to racks. This allows Hadoop to prefer within-rack transfers (where there is more bandwidth available) to off-rack transfers when placing MapReduce tasks on nodes. HDFS will also be able to place replicas more intelligently to trade off performance and resilience.

Question 34

What is the fsimage file in the HDFS?

Answer 34

1. Each fsimage file contains a serialized form of all the **directory** and **file inodes** in the filesystem. 2. Each inode is an internal representation of a file or directory’s metadata and contains such information as the file’s replication level, modification and access times, access permissions, block size, and the blocks the file is made up of. For directories, the modification time, permissions, and quota metadata are stored.

Question 35

List common maintenance procedures

Answer 35

1. Metadata backups (hdfs dfsadmin -fetchImage fsimage.backup) 2. Backup data 3. Filesystem check (It is advisable to run HDFS’s fsck tool regularly (i.e., daily) on the whole filesystem to proactively look for missing or corrupt blocks.) 4. Filesystem balancer (Run the balancer tool (see Balancer) regularly to keep the filesystem datanodes evenly balanced.)

Question 36

Is it necessary to perform backup of DataNode data?

Answer 36

Do not make the mistake of thinking that HDFS replication is a substitute for making backups. Bugs in HDFS can cause replicas to be lost, and so can hardware failures. Although Hadoop is expressly designed so that hardware failure is very unlikely to result in data loss, the possibility can never be completely ruled out, particularly when combined with software bugs or human error.