Cloud Storage

Question 1

3 Main Categories (steps) to the AWS Data Pipeline

Answer 1

1. Ingest (Gateway)
2. Transform and store (S3)
3. Serve and consume (EMR)

Question 2

Google’s version of AWS 3 step pipeline

Answer 2

1. Ingest
2. Analyze
3. Serve

(similar, but more generic used in professor example)

Question 3

Supporting services that supplement Ingestion, Analytics and Serve (2)

Answer 3

1. Storage
2. Security

Computing, networking

Question 4

DIKW pyramid

Answer 4

Data -> Information -> Knowledge -> Wisdom

Question 5

3 tiers of data structure

Answer 5

Structured
Semi-structured
unstructured

Question 6

3 levels of data abstraction

Answer 6

Block level (EC2)
file level (S3)
database mode (Amazon RDS)

Question 7

data access (models)

Answer 7

NoSQL and the 4 types
relational database (Amazon RDS)

Question 8

AWS S3

Answer 8

Simple storage service, place where you can store all types of data decoupled from processing, enabling a multi-user setup so different users can bring their own data while maintaining isolation and access control.

Object - file and metdata
bucket - logical containers for objects
(can configure access to buckets, geographical region for bucket)

Question 9

AWS S3 storage classes are like bank/investment accounts, why?

Answer 9

Different classes of access frequency, from S3 Standard (frequent) to S3 Deep Glacier (1 or 2 times a year!)

Its like how a bank offers better interest for accounts that are not accessed!

Question 10

Google Cloud Storage Classes (4)

Answer 10

1. Standard (frequent)
2. Nearline (Monthly)
3. Coldline (Yearly)
4. Archive (least frequent)

Question 11

Why is AWS better?

Answer 11

Offers intelligent-tiering class that automatically shifts data based on access patterns

Question 12

AWS Lifecycle Configuration

Answer 12

Set of rules that define actions that S3 will apply to a group of objects

Action types:

Transition - moving from one storage class to another (glacier - deep glacier)

Expiration - when to delete S3 objects

Question 13

Data Pipelines

Answer 13

Automated workflows that move and process data from one system to another, cleaning, transforming and enriching the data along the way

Question 14

Landing Area (LA) Data Lake

Answer 14

Where raw data is served from ingestion

Question 15

Staging Area (SA) Data Lake

Answer 15

Place where Raw data goes after basic quality transformations, ensuring that it conforms to existing schemas

Question 16

Archive Area (A)

Answer 16

Stores the original raw data for future reference, debugging, or reprocessing

Question 17

Production Area (PA) Data Lake

Answer 17

Apply business logic to data from Staging Area (SA)

Aggregating: Summarizing sales by store, region or product
Business specific calculations (profit margin)

Question 18

Pass-through job (Optional) Data Lake

Answer 18

Copy of data from Staging Area (SA) is passed directly to Cloud Data Warehouse without business logic.

For comparison and debugging

Question 19

Failed Area (FA) Data Lake

Answer 19

Captures data that encounters issues such as bugs in pipeline code or cloud resources failures to deal with errors

Question 20

4 folders used to organize data in logical structure (hint: file directory)

Answer 20

Namespace - group pipelines together
Pipeline Name - reflecting purpose, for example pipeline that takes data from the LA applies processing steps and saves to SA could be one pipeline
Data Source name - assigned by ingestion layer
BatchID - unique identifier for any batch of data saved into LA

Question 21

Namespace

Answer 21

Groups multiple pipelines or areas logically, such as “landing,” “staging,” “archive,”

Question 22

Pipeline Name

Answer 22

Each pipeline gets a name reflecting its purpose (sales_oracle_ingest)

Question 23

Data Source Name

Answer 23

The specific data source, such as customer data

Question 24

Batch ID

Answer 24

Unique identifier that is assigned to each batch of ingested data into the landing area

Question 25

Layers of cloud storage from ingestion (added) to archive and its corresponding AWS or Google Cloud tool

Answer 25

1. Ingestion Layer (load raw data from various sources into LA) - AWS Glue, Kinesis (stream) - Google Cloud DataFlow 2. Landing Area (Store raw, unprocessed data) - Amazon S3 - Google Cloud Storage 3. Staging Area (light transformations, quality check)

Question 26

What comes out of the Production Area (PA) and where does it go

Answer 26

Data Products, Cloud Data Warehoes

Question 27

Data Platform vs Data Lake

Answer 27

Lake - store, process, analyze raw data Platform - that + ingestion, querying, security, governance

Question 28

Data Lakehouse

Answer 28

Low-cost storage in open, native format, accessible by a variety of systems (lake) combined with optimized queries on structured data (DWH)

Question 29

Why is Lakehouse good for cloud environments?

Answer 29

Cloud storage often involves separation between storage and computing, renting instances only when needed for specific applications connected to the data storage and Lakehouse ACID qualities allow multiple users or applications to access data concurrently without inconsistencies

Question 30

Data Independence

Answer 30

You can modify the schema at one level of database system without altering the schema at the next higher level (physical, logical, external levels)

Question 31

3 cons to Data Warehouse and Lake

Answer 31

reliability is difficult and costly (both) data staleness (warehouse) limited support for machine learning (both)

Question 32

Is there a real need for many unstructured and integrated dataset?

Answer 32

Governments and organizations are increasingly making structured data more available

Question 33

Extra layer in DataLakehouse to prevent two-tiered ETL

Answer 33

Transactional Metadata layer

Question 34

Delta Lake

Answer 34

Enhances data lake with features like ACID transactions, schema enforcement and improved query performance, being one of the technologies that turn a lake into a lakehouse

Question 35

Special feature of Delta Lake

Answer 35

Uses a transaction log (JSON log records) to help optimize performance

Question 36

Caching (Lakehouse Optimization)

Answer 36

When using transactional metadata layer such as Delta Lake, lakehouse can cache files from cloud to faster storage devices such as SSDs an RAM on nodes for processing

Question 37

Auxiliary Data (Lakehoes Optimization)

Answer 37

Store min-max statistics for each data file (in same Parquet file used for transaction log) which enables skipping optimizations

Question 38

Data Layout (Lakehouse Optimizations)

Answer 38

Order records to make them easier to read together?