L8

Question 1

def BD processing

Answer 1

a set of techniques or programming models to access large-scale data to extract useful information for supporting and providing decisions
-> in BD happens before storing

Question 2

2 Types of data processing

Answer 2

1. centralized data processing
2. distributed data processing: distributed across different physical locations

Question 3

Batch processing

Answer 3

• computer processes a nr of tasks that have been collected in a group, often simultanously, in non-stop, sequential order
  Pros:
• good when response time is not important
• suitable for large data volume
• fast, inexpensive and accurate
• offline
• query-driven as static and more about historical fact finding
  eg) mothly payroll system, credit card billing system

Question 4

Real-time processing

Answer 4

• streams of data
• processing is done as data is input: filtering, aggregating and preparing data
• often optimized for analytics and visualization and directly ingested in tools for it -> data-driven
  eg) bank ATMs, control systems, social media

Question 4

Parallel computing

Answer 4

• splitting up larger tasks into multiple subtasks and execute at the same time
• reduces execution time
• multiple processing within single machine

Question 5

Distributed computing

Answer 5

• splits up larger tasks into subtasks and executes in separate machines networked together as a cluster

Question 5

Hadoop Ecosystem

Answer 5

system comprised of several components that cover several aspects of data ingestion, processing, analysis exploration and storage

Question 5

Sqoop

Answer 5

• interface application for transferring structured data between relational databases and Hadoop -> data ingestion
• can import and export

Question 6

Flume

Answer 6

• collects large amount of semi and unstructured streaming data from multiple sources

Question 7

Kafka

Answer 7

• streaming platform that handles constant influx of data and processes it incrementally and sequentially
• used to build real-time streaming pipelines
• combines messaging, storage and stream processing -> storage and analysis of both historical and real-time data

Question 8

Storm

Answer 8

• real-time big data processing system
• handles influx of data and easily process unbounded stream of data

Question 9

Storm vs MapRedue?

Question 10

Hadoop MapReduce

Answer 10

• software for programming firn-specific model that can process data in-parallel on clusters of commodity hardware
• fault-tolerant and reliable
• divide and conquer -> only for batch workloads
• disk-based processing
  2 phases:
• Map: splitting and mapping
• Reduce: shuffling and reducing

Question 11

Spark

Answer 11

• for large-scale data processing
• good for batch and stream data
• in-memory processing -> fast
• supports large-scale data science project s, SQL analytics and ML
• many languages: python, SQL, Java, R, etc

Question 12

MapReduce vs Spark

Answer 12

• spark is 100x faster due to in-memory proc
• MR only batch, while spark both real-time and batch
• spark good for ML
• MR low cost, spark high cost
• spark easy to combine databases
• MR linear processing -> slow

Question 13

Pig and Hive

Answer 13

data-analysis platforms on top of MapReduce

Question 14

YARN, Oozie, Zookeeper

Answer 14

1. YARN (Yet Another Resource Negotiatior): takes over resource management and job scheduling from MapReduce
2. Apache Oozie: manages workflow in Hadoop environment at desired order
3. Apache ZooKeeper: maintaining open-source server to enable reliable distributed coordination

Question 15

Explain Hadoop Ecosystem

Answer 15

?

Question 16

BD with AWS

Answer 16

AWS has ecosystem with analytical solutions specifically for growing amount of data

Question 17

Application: Clickstream analysis through AWS

Answer 17

1. clickstream data sent to Kinesis Stream
2. stored exposed for processing
3. custom application programmed on Kinesis makes real-time recommendations
4. output to user who sees personalized content suggestions

Question 18

Application: Data Warehousing through AWS

Answer 18

1. data is uploaded to S3
2. EMR is used to transform and clean data
3. is loaded back into S3
4. loaded into Redshift where it is parallelized for fast analytics
5. analysed and visualized with Quicksight

Question 19

Smart Applications through AWS

Answer 19

1. Amazon Kinesis receives data
   (2. AWS lambda is used to write code to coordinate data flow)
2. Amazon Machine Learning model is for real-time predictions
3. Amazon SNS is used to notify customer support agents

Question 20

Amazon S3

Answer 20

Amazon Simple Storage Service
- object storage service offering industry-leading scalability, data avilability, security and performance

Question 21

Amazon EMR

Answer 21

highly distributed computing framework for processing and storing big data
- EMR apache hadoop allows Hive, Pig, Spark to run on top

Question 22

Amazon Kinesis

Answer 22

data-streaming platform, makes it easy to download and analyze streaming data eg) site clicks, IoT data, data lake

Question 23

AWS Lambda

Answer 23

serverless, event-driven computing service to run code for any type of application or backend service without provisioning or managing servers

Question 24

Amazon ML

Answer 24

makes it easy for anyone to use predictive analytics, ML functions and GenAI

Question 25

Amazon DynamoDB

Answer 25

fully managed NoSQL database service providing fast and predictable performance with scalability

Question 26

Amazon Redshift

Answer 26

fast, large and fully managed cloud data warehouse storage service to analyze structured or semi-structured data efficiently at lower price