Ch 3 - class 3 Hadoop FIle System

Question 1

what are two componants of hadooop

Answer 1

mapreduce and hdfs

Question 2

hdfs

Answer 2

file system to manage hard drive. on top of file system on hard drive

Question 3

command interface

Answer 3

use to communicate hdfs and hdd

Question 4

communicate to server from hdrive

Answer 4

winscp

Question 5

file system deal with

Answer 5

large files. write once, read many times, high throughput

Question 6

data size?

Answer 6

block size. hdfs divided into blocks. 64mb by default, 128mb in practice.

Question 7

can many files be on same block?

Answer 7

YES!

Question 8

check status of file system block

Answer 8

% hadoop fsck –files -blocks

Question 9

Namenode

Answer 9

Manage filesystem namespace, keep track of blocks, block locations, namespace image

Question 10

cluster

Answer 10

name node, datanode

Question 11

single point of failure

Answer 11

persistand metadata files

Question 12

system has 2 namenodes

Answer 12

active and standby

Question 13

datanode known as

Answer 13

workhorse of the file sytem. store and retreive blocks, report to namenode.

Question 14

HDFS high avaiability

Answer 14

use pair of namenodes in active-standby configuration.

standby has latest log entreis and up to date block mapping in memory

Question 15

how do you set replication for data node

Answer 15

set dfs.replication=3

Question 16

Psudo-distribted configuration

Answer 16

fs. dautlname=hdfs://localhost/

dfs. replication=1

Question 17

where is default filesystem

Answer 17

on master computer, namenode

Question 18

where is local filesystem?

Answer 18

on the server

Question 19

command to copy from hard drive to hdfs

Answer 19

hadoop fs -copyFromLocal input/docs/quangle.txt

hdfs://localhost/user/tom/quangle.txt

Question 20

for checksum

Answer 20

use md5 to check file integrity to compare
md5sum used hash function producing a 256 bit hash value, gives a checksum to verify data integrity.
256bit =32bytes=32characters.

Question 21

split of data

Answer 21

you want to split the data so it fits on 1 block..

so use block size for split size

Question 22

hdfs

Answer 22

just one implemenation of hadoop filesystem, s3 another.

Question 23

2 waits to catch exception

Answer 23

try or finally , 2 ways to catch the exception

Question 24

finally

Answer 24

regardless of exception or not. thats how diff from catch

it’s a strong method.

Question 25

how to tell if hdfs command or not

Answer 25

you will see hadoop fs not hadoop URLCAT .etc. hadoop URLCAT is java program

Question 26

FileSystemCat

Answer 26

public java hadoop program to handle file stream | bbbbbbbbbbbbbbb

Question 27

complete java program

Answer 27

public class and main method

Question 28

glob characters

Answer 28

regular expressions

Question 29

datanode error - client

Answer 29

ads and packets in ack queue to data queue removeds the failed datanode from the pileline. namenode - arranges under replicated blocks for further replicas. failed datanode- deleted the parital block when the node recovers later on.

Question 30

ack

Answer 30

sent when data is received by datanode, not when written.

Question 31

way to write in paraallel to speed up process of copying data.

Answer 31

distcp is used for copying large amounts of data to and from hadoop filesystems in parallel.

Question 32

RPC

Answer 32

remote procedure call. communication with the nodes.

Question 33

HAR Files

Answer 33

are file archiving facility that packes files into HDFS blocks more efficiently.

Question 34

L= | r=

Answer 34

long, -r Recursive means show all the entries in the subtree

Question 35

-p

Answer 35

P option preserves file attributes(timestamp, ownership, permission, etc)