Analytics

Question 1

What is AWS Glue?

Answer 1

A serverless ETL service for discovering, preparing, and combining data for analytics, machine learning, and application development.

Question 2

What are the key components of AWS Glue?

Answer 2

Data Catalog, Crawler, ETL Engine, Studio, Data Quality, DataBrew, Workflows

Question 3

What is the purpose of the Glue Data Catalog?

Answer 3

Stores table definitions and schemas for data located in various sources like S3, RDS, and DynamoDB.

Question 4

What does a Glue Crawler do?

Answer 4

Scans data stores to infer schemas and create table definitions in the Data Catalog.

Question 5

What programming languages are supported for Glue ETL jobs?

Answer 5

Scala and Python

Question 6

What is a DynamicFrame in AWS Glue?

Answer 6

A collection of DynamicRecords with a schema, similar to a Spark DataFrame but with additional ETL capabilities.

Question 7

What are some common transformations in Glue ETL?

Answer 7

DropFields, DropNullFields, Filter, Join, Map, FindMatches ML, format conversions

Question 8

How can you modify the Data Catalog using Glue ETL scripts?

Answer 8

By using options like enableUpdateCatalog, partitionKeys, and updateBehavior to add partitions, update schemas, or create new tables.

Question 9

What are AWS Glue Development Endpoints?

Answer 9

Notebook-based environments for developing and testing Glue ETL scripts.

Question 10

What are Glue Job Bookmarks used for?

Answer 10

Persisting state from a job run to prevent reprocessing of old data and ensure incremental processing.

Question 11

How are AWS Glue costs calculated?

Answer 11

Primarily billed by the second for crawler and ETL jobs, with additional charges for Data Catalog storage beyond the free tier and for development endpoints.

Question 12

What is AWS Glue Studio?

Answer 12

A visual interface for creating and managing ETL workflows without writing code.

Question 13

What is the purpose of AWS Glue Data Quality?

Answer 13

To define and enforce data quality rules within Glue jobs using DQDL.

Question 14

What is AWS Glue DataBrew?

Answer 14

A visual data preparation tool with over 250 pre-built transformations for cleaning and normalizing data.

Question 15

What is AWS Lake Formation?

Answer 15

A service that simplifies the setup and management of secure data lakes.

Question 16

What are Governed Tables in AWS Lake Formation?

Answer 16

A new type of S3 table that supports ACID transactions and fine-grained access control.

Question 17

What is Amazon Athena?

Answer 17

A serverless interactive query service that allows you to analyze data in S3 using standard SQL.

Question 18

What data formats are supported by Athena?

Answer 18

CSV, TSV, JSON, ORC, Parquet, Avro, and various compression formats.

Question 19

How is Amazon Athena priced?

Answer 19

Pay-as-you-go, based on the amount of data scanned by queries.

Question 20

What are Athena Workgroups?

Answer 20

A way to organize users, teams, and workloads in Athena, allowing for cost tracking and query access control.

Question 21

How can you optimize Athena query performance?

Answer 21

Use columnar data formats, partition data effectively, and minimize the number of small files.

Question 22

What is the CREATE TABLE AS SELECT (CTAS) statement used for in Athena?

Answer 22

Creating a new table from the results of a query, including data format conversion.

Question 23

What is Apache Spark?

Answer 23

A distributed processing framework for big data processing, known for its in-memory caching and optimized query execution.

Question 24

What are the core components of Apache Spark?

Answer 24

Spark Core, Spark SQL, Spark Streaming, MLLib, GraphX

Question 25

What is Spark Structured Streaming?

Answer 25

A scalable and fault-tolerant stream processing engine built on the Spark SQL engine.

Question 26

What is Amazon EMR?

Answer 26

A managed Hadoop framework on EC2 that allows you to run big data frameworks like Spark, Hive, Presto, and Flink.

Question 27

What are the different node types in an EMR cluster?

Answer 27

Master node, Core node, Task node

Question 28

What is EMRFS?

Answer 28

A file system that allows you to access data in S3 as if it were HDFS.

Question 29

What is EMR Managed Scaling?

Answer 29

A feature that automatically scales EMR clusters based on workload demands.

Question 30

What is EMR Serverless?

Answer 30

A way to run Spark, Hive, and Presto applications on EMR without managing clusters.

Question 31

What is Amazon Kinesis Data Streams?

Answer 31

A service for real-time data streaming with high throughput and low latency.

Question 32

What are the two capacity modes for Kinesis Data Streams?

Answer 32

Provisioned mode and On-demand mode

Question 33

What is the Kinesis Producer Library (KPL)?

Answer 33

A library for building high-performance producers that send data to Kinesis Data Streams.

Question 34

What is Kinesis Enhanced Fan-Out?

Answer 34

A feature that allows multiple consumers to each get dedicated throughput from a Kinesis stream.

Question 35

What is AWS Kinesis Data Firehose?

Answer 35

A fully managed service for loading streaming data into data stores like S3, Redshift, and OpenSearch.

Question 36

What is Amazon Kinesis Data Analytics?

Answer 36

A service for analyzing streaming data in real-time using SQL or Apache Flink.

Question 37

What are Reference Tables in Kinesis Data Analytics?

Answer 37

A way to join streaming data with static data stored in S3 for enrichment and lookups.

Question 38

What is the Managed Service for Apache Flink?

Answer 38

A serverless service that allows you to run Python and Scala Flink applications for stream processing.

Question 39

What is the `RANDOM_CUT_FOREST` function used for in Kinesis Data Analytics?

Answer 39

Anomaly detection on numeric data within a streaming data set.