Spark

Question 1

What is Spark?

Answer 1

An opensource cluster computing framework

Question 2

What does Spark do?

Answer 2

• It automates distribution of data and computations on a cluster of computers
• Provides a fault tolerant abstraction to distributed datasets
• Based on Functional Programming!!
• Provides list-like RDDs and table-like Datasets

Question 3

What do RDDs do?

Answer 3

They make a distributed list over a cluster of machines like a Scala collection.

Question 4

What are some attributes of RDDs?

Answer 4

• Immutable
• Mostly reside in memory
• Are distributed transparently
• Works like Scala’s List

Question 5

What are the types of operations on RDDs?

Answer 5

• Transformations: Applying a function that returns a new RDD; they are lazy
• Action: Request a computation as a result; these are eager

Question 6

What does RDD stand for?

Answer 6

Resilient Distributed Dataset

Question 7

What are the common transformations?

Answer 7

map, flatMap, filter

Question 8

What are the common actions?

Answer 8

collect, take, reduce, fold, aggregate

Question 9

What is the main aspect of Pair RDDs?

Answer 9

They can be iterated and indexed, and take the form RDD[(K, V)]. Operations such as joins are defined in Pair RDDs.

Question 10

What are the 3 Spark partitioning schemes?

Answer 10

• Default partitioning: Split into equally sized partitions without knowing underlying data properties
• Range partitioning: (Pair) Takes into account the order of keys to split the dataset, requires the keys to be naturally ordered
• Hash partitioning: (Pair) Calculates a hash over each item and produces the modulo to determine this partition

Question 11

What is the difference between narrow and wide partition dependencies?

Answer 11

• Narrow dependencies: Each partition of the source RDD is used by at most one partition of the target RDD
• Wide dependencies: Multiple partitions in the target RDD depend on a single partition in the source RDD

Question 12

How does persistence work in the Spark framework (i.e. Java/serialized/FS)

Answer 12

• Java: each item in the RDD is an allocated object
• Serialized data: Special memory-efficient format. CPU intensive, but faster to send across network
• On FS: IN the case the RDD is too big, it can be mapped to the FS, usually HDFS

Question 13

What are the two main dataset abstractions used by SparkSQL

Answer 13

• Datasets: A collection of strongly typed objects
• Dataframes: Essentially a Dataset[Row], where a Row is an array

Question 14

How can Dataframes and Datasets be created?

Answer 14

• From RDDs containing tuples: rdd.toDF(“name”, “id”, “address”)
• From RDDs with known complex types e.g. RDD[Person]: rdd.toDF()
• From RDDs with manual schema definition
• By reading semi-structued data files: spark.read.json(“json.json”)

Question 15

How can you access a column in a DataFrame?

Answer 15

df(“column_name”)

Question 16

What are some common operations on DataFrames?

Answer 16

• Projection: df.select(“project_name”)
• Selection: df.filter
• Joins: left.join(right, condition, how = full_outer)
• Aggregations can only be done after being grouped