Lesson 6: Distributed Storage

Question 1

What is Moore’s Law?

Answer 1

The number of transistors in an integrated circuit doubles about every two years

Question 2

Why was there a switch from faster to parallel execution?

Answer 2

Paradigm shift:
speed of light, atomic boundaries, limited 3D layering

Question 3

What is Hadoop?

Answer 3

• Framework that allows us to do reliable, scalable distributed computing and data storage
• It is a flexible and highly available architecture for large scale computation and data processing on a network of commodity hardware
• Apache top level project
• Open-source

Question 4

What do we need to write for MapReduce?

Answer 4

• Mapper: Application code
• Partitioner: send data to correct Reducer machine
• Sort: group input from different mappers by key
• Reducer: application code

Question 5

How do we store large files with the Hadoop Distributed File System

Answer 5

• We have a file
• HDFS splits it into blocks
• HDFS will keep 3 copies of each block
• HDFS stores these blocks on datanodes
• HDFS distributes the blocks to the data nodes
• The NameNode tracks blocks and data nodes
• Sometimes a data node will die → not a problem
• Namenode tells other data nodes to copy blocks, back to 3X replication

Question 6

What are the components of the Hadoop Architecture? (overview)

Answer 6

• MapReduce Framework: implement MapReduce paradigm
• Cluster: host machines (nodes)
• HDFS federation: provides logical distributes storage
• YARN Infrastructure: assign resources

Question 7

What is the infrastructure of YARN?

Answer 7

 Resource Manager (1/cluster): assign cluster resources to applications
 Node Manager (many/cluster): monitor node
 App Master: app (MapReduce)
 Container: task (map, reduce, …)

Question 8

What are the shortcomings of MapReduce?

Answer 8

• Forces your data processing into MAP and REDUCE → other workflows missing
• Based on “Acyclic Data Flow” from Disk to Disk (HDFS)
  → not efficient for iterative tasks
  *Only for batch processing (interactivity, streaming data)

Question 9

What should be counted when calculating algorithmic complexity?

Answer 9

• page faults
• cache misses
• memory accesses
• disk accesses (swap space)
• …

Question 10

What is Google MapReduce?

Answer 10

A framework for processing LOADS of data
-> framework’s job: fault tolerance, scaling & coordination
-> programmer’s job: write program in MapReduce Form

Question 11

What are the 2 main components of Hadoop?

Answer 11

HDFS - big data storage
MapReduce - big data processing

Question 12

How can you tell that the Hadoop Architecture is inspired by LISP (list processing)?

Answer 12

Functional programming:
* Immutable data
* Pure functions (no side effects): map, reduce

Question 13

What is the difference between a job and a task tracker in the Hadoop Architecture?

Answer 13

Job tracker: in charge of managing the resources of the cluster
-> first point of contact when a client submits a process
-> one per cluster

Task tracker: does the actual process
-> mostly connected to one or more specific data nodes

Question 14

What are the 3 functions in Google MapReduce? (2 primary, one optional)

Answer 14

• Map
• Reduce
• Shuffle

Question 15

How does the Map function work, of Google’s MapReduce?

Answer 15

Map each <key, value> data pair of input list onto 0, 1, or more pairs of type <key2, value2> of output list
-> Map to 0 elements in the output = filtering
-> Map to +1 elements in the output = distribution

Question 16

How does the Reduce function work, of Google’s MapReduce?

Answer 16

[summarizing]
Combine the <key, value> pairs of the input list to an aggregate output value

Question 17

What does the Shuffle function do, in Google’s MapReduce?

Answer 17

[consolidating relevant record]
* Helps the pipeline
* It will help channel the partial result to the right or most appropriate reduce node

Question 18

What is YARN short for?

Answer 18

Yet Another Resource Negotiator

Question 19

Explain the Hadoop eco-system.

Answer 19

Hadoop provides very good functions on its own but the main power of Hadoop comes out when we start combining it with different other technologies.
(Ex.: Pig, Hive, Kafka)

Question 20

What is Apache Spark?

Answer 20

• works on top of Hadoop, HDFS
• has many other workflows
• in-memory caching of data
• in-memory data sharing
• supports data analysis, machine learning, graphs,…
• allows development in multiple languages
• can read/write from a range of data types

Question 21

What are RDD’s?

Answer 21

Resilient Distributed Datasets
-> immutable distributed collection of objects
-> fault tolerant
-> used in every spark comonents

Question 22

How do you create new RDD’s?

Answer 22

By using transformations
-> from storage
-> from other RDDs

Question 23

What are DataFrames?

Answer 23

A way to organize the data in named columns.
Similar to a relational database
-> immutable once constructed
-> enable distributed computations

Question 24

How can you construct data frames?

Answer 24

• read from file(s)
• transforming an existing dataframe
• parallelizing a python collection list
• apply transformations and actions

Question 25

Compare RDDs with DataFrames.

Answer 25

• RDDs provide a low level interface into Spark
• DataFrames have a schema
• DataFrames are cached and optimized by Spark
• DataFrames are built on top of the RDDs and the core Spark API
• DataFrames are highly optimized and are faster

Question 26

How and why would we use directed acyclic graphs?

Answer 26

How:
Nodes are RDDs, arrows are transformations
Why:
* track dependencies
* program resonates with humans and computers
* improvement via sequential access to data & predictive processing

Question 27

What is the difference between a narrow and a wide transformation?

Answer 27

Narrow: everything can be found in the same partition
-> required elements for computation in a single partition live in the single partition of parent RDD (map)

Wide: we need to fetch from multiple partitions
-> Required elements for computation in a single partition may live in many partitions of parent RDD (groupByKey)

Question 28

When should you not use Spark?

Answer 28

• for many simple use cases Apache MapREduce and Hive might be a more appropriate choice
• spark was not designed as a multi-user environment
• spark users are required to know that memory they have is sufficient for a dataset
• adding more users adds complications, since the users will have to coordinate memory usage to run code