Spark

Question 1

Two common classes of analytics apps

Answer 1

• Iterative algorithms (machine learning, graph)
• Interactive data mining

Question 2

Enhance programmability

Answer 2

• Integrate into Scala programming language
• Allow interactive use from Scala interpreter

Question 3

What part of the machine does Hadoop access?

Answer 3

Main memory (hard drive)

Question 4

What memory does Spark use?

Answer 4

Uses RAM for faster processing

Question 5

What is Scala?

Answer 5

Based on Java

Question 6

What did MapReduce simplify?

Answer 6

“big data” analysis on large, unreliable clusters

Question 7

What did users want more of as MapReduce became more popular?

Answer 7

-More complex, multi stage applications
-More interactive ad-hoc queries

Question 8

How to share data in MapReduce across jobs?

Answer 8

stable storage, which is slow

Question 9

What do complex apps and interactive queries both need that MapReduce lacks?

Answer 9

efficient primitives for data sharing

Question 10

Resilient Distributed Datasets (RDD)

Answer 10

• Restricted form of distributed share memory
• Efficient fault recovery using lineage (DAG)

Question 11

Immutable

Answer 11

Once you put the data on the RAM, it cannot be changed in that RDD

Question 12

How to modify RDD?

Answer 12

Can only be built through coarse-grained deterministic transformations (map, filter, join…) in a new RDD

Question 13

Directed Acyclic Graph (DAG)

Answer 13

an arrangement of edges and vertices
- vertices: indicate RDDs
- edges: operations applied on the RDD

Question 14

RDD Recovery

Answer 14

Using the lineage graph, the data can be restored

Question 15

SparkContext

Answer 15

represents the connection to a Spark cluster (main driver)
- to create RDDs, accumulators and broadcast variables on that cluster

Question 16

Cluster Manager

Answer 16

provides resources to all worker nodes as per need and operates all nodes accordingly

Question 17

RDD Types

Answer 17

parallelized collections

Question 18

What are the 3 types of operations programmers can perform on the RDD?

Answer 18

Transformations, Actions, Persistence

Question 19

Transformations

Answer 19

create a new dataset form an existing one
- lazy in nature and are only executed when some action is performed

Question 20

Actions

Answer 20

returns to the driver program a value or exports data to a storage system after performing a computation

Question 21

Persistence

Answer 21

For caching datasets in-memory for future operations and option to store on disk or RAM or mixed (storage level)

Question 22

Example transformations functions

Answer 22

Map(func)
Filter(func)
Distinct()

Question 23

Example actions functions

Answer 23

Count()
Reduce(func)
Collect()
Take()

Question 24

Example persistence functions

Answer 24

Persist()
Cache()

Question 25

Map(func)

Answer 25

return a new distributed dataset formed by passing each element of the source through a function func

Question 26

flatMap(func)

Answer 26

first applies map function and then flattens the result

Question 27

Which framework is better for larger data?

Answer 27

Hadoop

Question 28

Log mining

Answer 28

load error messages from a log into memory, then interactively search for various patterns

Question 29

reduce(func)

Answer 29

aggregate the elements of the dataset using a function func

Question 30

What three options does Spark provide for persist RDDs?

Answer 30

(1) in-memory storage as deserialized Java Objs - fastest, JVM can access RDD natively (2) in-memory storage as serialized data - space limited, choose another efficient representation, lower performance cost (3) on-disk storage - RDD too large to keep in memory, and costly to recompute

Question 31

Fault recovery

Answer 31

RDDs maintain lineage information that can be used to reconstruct lost partitions

Question 32

What are the benefits of the RDD Model?

Answer 32

- Consistency is easy due to immutability - Inexpensive fault tolerance (log lineage rather than replicating/checkpointing data) - Locality-aware scheduling of tasks on partitions - Despite being restricted, model seems applicable to a broad variety of applications