Week6 - Apache Spark

Question 1

What is RDD stand for?

Answer 1

Resilient Distributed dataset

Question 2

Is RDD read only?

Answer 2

Yes

Question 3

RDDs can only be created through? (2)

Answer 3

1) Data in stable Storage

2) other RDDs

Question 4

RDD is a restricted Distributed shared____ what

Answer 4

Memory System ( Cached dataset shared memory)

Question 5

RDD Contains dataset?

Answer 5

Atomic pieces of the dataset

Question 6

RDD Contains dependencies on?

Answer 6

Parent RDDs

for fault tolerance

Question 7

How does a RDD compute the dataset

Answer 7

It is based on its parents (for fault tolerance)

metadata about its partitioning scheme and data placement

Question 8

RDD read only and

Answer 8

Partitioned collections of records

Question 9

Two important features of RDD and Apache Spark

Answer 9

1) Fault Tolerance

2) Lazy Evaluation

Question 10

Describe RDD Fault Tolerance

Answer 10

It is achieved through lineage retrieval

Question 11

Describe RDD Lazy Evaluation

Answer 11

A RDD will not be created until a reduce-like job or persist job is created ( create meaningful output)

Question 12

What two classes of operations can you do on RDDs

Answer 12

1) Transformations

2) actions

Question 13

RDD Transformations

Answer 13

Build RDDs through operations on other RDDs

1) map, filter, join
2) lazy operations

Question 14

RDD Actions

Answer 14

1) Count, Collect, save

2) trigger execution

Question 15

hdfs is ?

Answer 15

1) text file (Hadoop file system)
2) Distributed file system
3) contain text, log files, errors

Question 16

How to find errors in htfs files

Answer 16

file.filter(_.contians(“ERROR”))

Question 17

DAG Scheduler

Answer 17

Partition DAG into efficient stages (think narrow and wide dependencies) Pu

Question 18

Narrow Dependencies

Answer 18

Transformation: output needs input from only one partition (very title communications )

1) map
2) union

Question 19

Wide Dependencies

Answer 19

Multiple dependencies… need data from other partition

1) Group by key
2) join with inputs not on the same partitioned

Question 20

DAG wide dependencies early or late in the process

Answer 20

late (less amount of data)

Question 21

Hadoop scheduler

Answer 21

Only 2

1) Map
2) reduce

Question 22

Hadoop where is data stored?

Answer 22

Assumes all data is on disk (intermediate data has to be on disk)

Why fault tolerance

Question 23

Hadoop API

Answer 23

Only has Map and Reduce procedure programming

Question 24

Hadoop storage

Answer 24

Only on HDSF file system (now extended)

Question 25

Hadoop more or less memory than spark

Answer 25

Less, stores data locally

Question 26

Apache Spark has what APIs

Answer 26

1) Java 2) Scala 3) Python

Question 27

4 Main Apache Spark Libraries

Answer 27

1) Spark SQL 2) Spark Streaming 3) MLibe (machine learning) 4) GraphX

Question 28

What is a DataFrame?

Answer 28

Looks like a table (can run SQL operations on it)

Question 29

What Spark Library does Google PageRank and Shortest Path use for

Answer 29

GraphX

Question 30

What Spark Library is used for Streaming

Answer 30

Spark Streaming (process data in real time)

Question 31

Spark Dstream

Answer 31

Abstraction that represents the data streaming source.

Question 32

What does Spark Dstreamdo?

Answer 32

1) Chops data into batches of x seconds 2) Process the batches like RDD 3) returns the processed RDD results in batches

Question 33

Spark Dstream batch sizes

Answer 33

1) Low 1/2 second | 2) Latency about 1 second

Question 34

Apache Kafka

Answer 34

Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Question 35

Dstream - Operations State / Transformations

Answer 35

1) Window (timed window, counting, finding first or last)

Question 36

Apache Hadoop - Apache Hive

Answer 36

SQL like - A data warehouse infrastructure that provides data summarization and ad hoc querying (HiveQL)

Question 37

Apache Hadoop - Apache Pig

Answer 37

A high-level data-flow language and execution framework for parallel computation (less ridged than normal Hadoop)

Question 38

Apache Hadoop - Apache HBase

Answer 38

NoSQL database. (based on BigTable) | A scalable, distributed database that supports structured data storage for large tables.

Question 39

Apache Hadoop - Apache Zookeeper

Answer 39

Basically Ansible - high-performance Coordination Service for distributed architecture

Question 40

Apache Mahout

Answer 40

Machine Learn algorithms - distributed linear algebra framework and mathematically expressive Scala DSL

Question 41

What does Geospark

Answer 41

It takes the RDD layer (generic data processing ) and extends it with spatial data processing operations.

Question 42

Spatial query processing layer

Answer 42

Out-of-the-box implementation for de facto spatial queries that exist out there. (range query) (KNN k-nearest neighbor query) (join queries)

Question 43

What might cause data skew (Geospark)

Answer 43

Creating a grid and inserting data based on the grid coordinates Creates load balancing problem

Question 44

Load balancing problem

Answer 44

grid problem. data Skew. Some boxes emply, some boxes too full

Question 45

For types of grids

Answer 45

1) uniform gird 2) Quad-tree - based on density 3) KDB-tree - no overlap 4) R-Tree - based on clusters (overlap)

Question 46

When do you build a local index (Geospark)

Answer 46

When running a lot, maybe the same computation | hundreds of thousands of times or thousands of times per each partition

Question 47

Spatial Join Query (Geospark)

Answer 47

Count the number of points within an area

Question 48

(Geospark) Is a join or filter more expensive

Answer 48

Join

Question 49

What are space-filling curves

Answer 49

1) Map 2D data into one number 2) lose details Space-filling curves is a kind of geometrical slash mathematical property of the 2D data as you can see that you can partition, you can easily map each 2D cell, which is represented in a space, which is just one number, and the numbers of cells represent how close they are in space.

Question 50

Spatial indexes

Answer 50

1) similar hash tables (uniform grid) index based on grid id 2) Can also use Guad Tree (partition into 4, and partition that into 4..) 3) R-Tree -find rectangles 4) Voronoi Diagram it partitions the space into different cells such that it has a mathematical property, such that all like, if you do any queries within the cells, the points within these cells are the closest to that point k-nearest