Chapters 3&4 Knowledge Testers

Question 1

Do you know how storage and memory technologies (HDD, SDD and RAM) compare in terms of capacity and throughput?

Answer 1

Capacity (least to most): RAM, SSD,HDD
Throughput: (slow to fast): HDD,SSD,RAM

Question 2

Do you know the difference between data and metadata?

Answer 2

metadata: small relational table with the following attributes
- name of the file, access rights, owner, group of owner, last modification time, creation time, size

Data: content of the files

Question 3

Order of magnitude that can be achieved in terms of number of objects, and object size?

Answer 3

S3 - 100 buckets, object is 5TB, chunk is 5GB

Question 4

Can you name a few big players for cloud-based object storage (vendors, consumers)?

Answer 4

Amazon S3, Microsoft Azure

Question 5

Can you describe the features of S3 and Azure Blob Storage on a high level? Do you know what a bucket and object are? What block blob storage, append blob storage and page blob storage are and how they work?

Answer 5

S3: buckets that contain objects (up to 5TB, fit on single disk)
- no hierarchies for objects

Azure: public documentation, IDed by account, container, blobs.
- organized in storage stamps (10-20 racks) 30PB
- exposes more details to users
- block: store data
- append: for logging
- page: for storing and accessing the memory of virtual machines

Question 6

Describe what the most important SLA (Service Level Agreement) parameters mean (e.g., latency, availability, durability) as well as their typical range?

Answer 6

latency: how fast your data will reach you. response times (eg. 99.9% of requests will be served in 30 seconds) S3 has no latency guarantees
availability: How often your data will be available for you (99.99% of year, 52 minutes with no availability)
durability: how much data will be lost (less than 1 in 1 billion or 99.999999999% not lost (nine 9s after decimal))

Question 7

Do you know what each letter stands for in CAP?

Answer 7

Consistency: (Atomic) at any point in time the same request to any server returns the same results (all nodes see same data)
Availability: system available at all times for requests
Partition tolerance: System continues to function even if network linking its machines is occasionally partitioned

Question 8

Can you explain why, for large amounts of data, CAP becomes rel- evant over ACID?

Answer 8

ACID: for data all stored on one machine for relational databases
CAP: For big data where many machines are used and where replicas of nodes must be made and updated

Question 9

Can you explain what a REST API is, what resources and methods are?

Answer 9

data stores ecpose their functionality through APIs. REST = REpresentational State Transfer. “HTTP” done right”.
Resources: anything: document, pdf, person. refered to with a URI (uniform Resource Identifier)
Methods: normal methods/functions that do stuff. Get, put, delete, post.

Question 10

Can you describe a typical use case for object storage?

Answer 10

shopping carts for large online stores

Question 11

Can you explain the difference between block storage and object storage?

Answer 11

object -> flat, key-value pairs
HUGE (billions/trillions) amounts of big objects (5TB)
block -> hierarchies
->a lot (Millions) of HUGE files (>5TB)

Question 12

Can you explain the difference between the (logical) key-value model and a file system?

Answer 12

key-value: flat no hierarchies
file system: hierarchies

Question 13

Do you know the order of magnitude of a block size for a local filesystem and for a distributed file system? Can you explain the rationale behind them with respect to latency and throughput?

Answer 13

Local: 4kB
DFS block size: 64 or 128 MB
large enough that time is not lost in latency waiting for a block to arrive
small enough for a large file to be conveniently spread over many machines (parallel access) to improve throughput. Also small enough that if there is an error and they must be sent again it can be

Question 14

Can you contrast centralized architectures to decentralized (peer-to- peer) architectures?

Answer 14

Decentralized: all machines can talk to each other.
Centralized: there is a central machine. all other machines talk to the central machine, one node is special (queen bee).
the others nodes are basic bitches and interchangeable

Question 15

Can you explain the HDFS architecture, what a NameNode is and what a DataNode is, how blocks are replicated?

Answer 15

HDFS: distributed files system: hierarchy of files over multiple machines
NameNode: the special node that all other nodes communicate with. stores namespace, mapping from file to list of its blocks, mapping from block to locations of replicas
DataNode: store data blocks on local disk, communicate with namenode through regular heartbeats. (datanode always initiates contact). datanodes communicate with each other through replication pipelines

Blocks are replicated through replication pipilines between datanodes. blocks are sent in smaller packets in a streaming fashion. The original node sends the first copy of the data to node #2, node #2 then propogates on. Original node does NOT send data to all the nodes.

Question 16

Can you sketch how the various components communicate with each other (client, NameNode, DataNode)?

Answer 16

NameNode: never initiates communication (like women w sex). namenode responds to heartbeats
DataNode: communicate with namenode through regular heartbeats (always initiates). communicate with each other through replication pipelines
Blocks are replicated through replication pipilines between datanodes. blocks are sent in smaller packets in a streaming fashion. The original node sends the first copy of the data to node #2, node #2 then propogates on. Original node does NOT send data to all the nodes.
Client communicates with namenode and datanode

Question 17

Can you point to the single points of failure of HDFS and explain how they can be addressed?

Answer 17

If the namenode fucks off were screwed. Keep track of edit log, merge into a bigger log periodically. We use a “phantom NameNode” that keeps the exact same structures in its memory as the real NameNode, and performs checkpoints periodically. It is possible to set it up so the phantom NameNode to be able to instantly take over from the NameNode in case of a crash.

Question 18

Can you explain how the NameNode stores the file system namespace, in memory and on disk? In particular, can you explain how the namespace file and the edit log work together at startup time and how they get modified once the system is up and running?

Answer 18

HDFS does not follow a key-value model: instead, an HDFS cluster organizes its files as a hierarchy, called the file namespace. Files are thus organized in directories, similar to a local file system. This is stored on NameNode. The file namespace containing the directory and file hierarchy as well as the mapping from files to block IDs is backed up to a so-called snapshot. The snapshot and edit log are stored either locally or on a network- attached drive (not HDFS itself). For more resilience, they can also be copied over to more backup locations. If the NameNode crashes, it can be restarted, the snapshot can be loaded back into memory to get the file namespace and the mapping of the files to block IDs. Then the edit log can be replayed in order to apply the latest
changes

Question 19

Can you explain what a Standby NameNode is? (Note: it has many predecessors that only have historical relevance in the development of HDFS: Backup NameNode, Secondary NameNode, Checkpoint NameNode, etc, but this is not important for the course)

Answer 19

We use a Standby NameNode keeps the exact same structures in its memory as the real NameNode, and performs checkpoints periodically. It is possible to set it up so the standby NameNode to be able to instantly take over from the NameNode in case of a crash.

Question 20

where does HDFS shine and why?

Answer 20

PBs of data. Write once, read many. fault tolerance. throughput

Question 21

Do you know that HDFS files are updated by appending atomically and why?

Answer 21

simplifies consistency, aids in batch processing, simplifies logging

Question 22

Do you know how HDFS performs in terms of throughput and latency?

Answer 22

optimized for high throughput

Question 23

What are the main benefits of HDFS?

Answer 23

handles massive files, streaming, scalable, fault tolerance, high availability, works well with scaling out

Question 24

Describe the limitations of traditional (local) file systems?

Answer 24

local file systems must fit on one machine, creating size constraints. Single disks cannot store big datasets.

Question 25

contrast or relate object storage with block storage, a file system, and the key-value model

Answer 25

Object Storage: - Store each object -> data, metadata, unique identifier -good with unstructured data -images, videos, backups, data lakes Block storage: store data in fixed sized blocks - read/write blocks directly - requires system to manage files and directories File System: - organizes data into hierarchy structure of directories - built on top of block storage -every day general purpose storage Key-Value model: - key: identifier, value: data - access data using key, no scheme or hierarchy - caching, session data, quick lookups, Object store can be seen as key-value store, object stores -> large key value -> small and fast

Question 26

What is a data centre is made of (numbers)

Answer 26

1000 - 10,000s of machines - most cannot handle 100,000+ due to electricity issues and cooling - trend less servers more cores - server can be called node has 1-64 cores (increasing) - memory on node between 16GB - 6TB - SSD/HDD 1-20 TB - bandwidth 1-100GB/s - nodes - flat rectangular boxes, piled in a rack - cluster - room of racks -module/node in a rack can be server or pure storage or network switch - height of module standardized 1-4 RU (rack units)

Question 27

Do you know rough, typical numbers (per-node storage capacity, memory, number of cores, etc.)?

Answer 27

- 1000 - 10,000s of machines - 1-64 cores (increasing) - memory on node between 16GB - 6TB - SSD/HDD 1-20 TB - bandwidth 1-100GB/s - height of module standardized 1-4 RU (rack units)

Question 28

What is Object Storage

Answer 28

A solution to RDMS to store a lot of data. Can store data in blocks to improve latency. Information is stored on a cluster of machines

Question 29

benefits of object storage

Answer 29

Typically latency improves and more fault tolerant. Many services provide SLA - service level agreement - agreement to prevent data loss and ensure availability S3 no promises on latency

Question 30

why is scaling out less expensive than scaling up?

Answer 30

scale up is buy new expensive computer, scale out is add cheap new computer to your existing clusters of computers

Question 31

explain what aspects of the design of object storage enable scaling out and why?

Answer 31

There are no hierarchies. Data is split into blocks, can put different blocks on different machines. Data is already broken up into pieces. Works well for heterogeneous machines

Question 32

explain the three different ways, on the physical level, to deal with more data.

Answer 32

Scale up - improve machines - more capacity scale out - buy more machines Write better code - improve code efficiency

Question 33

What is the difference between storage, memory, CPU and network and how the three are paramount in a cluster?

Answer 33

Storage - disk (SSDs) can store data up to a few terabytes - local file system - non-volatile Memory - volatile - "working" memory - temporary CPU - calculations/computations network - a disk can be made accessible through network (LAN or WAN)