Data Engineering - Batch Processing for ML

Question 1

Define Batch Processing

Answer 1

Processing usually performed to a specific schedule. Data is often waiting for the next batch to be processed

Question 2

Name a service that is commonly used for batch processing in ML for ETL

Answer 2

Glue

Question 3

What type of service is Glue?

Answer 3

An Extract, Transform Load (ETL) service

Question 4

Name the steps of AWS glue

Answer 4

Gleu crawler followed by data placed in Glue database and tables

Question 5

Name some built-in data classifiers Glue offers?

Answer 5

Parquet, JSON, BSON, XML, CSV, PostgreSQL, MySQL

Question 6

What if the data is not in a format that Glue has built-in classifier for?

Answer 6

You can build a custom classifier using a GROK pattern, XML tag, JSON or CSV

Question 7

Describe AWS Database migration service for data ingestion

Answer 7

Design to transfer data between databases

Question 8

Which source database can be ingested by AWS Database migration tool?

Answer 8

RDS, EC2 instance and on premises

Question 9

Why is the AWS Database migration service so reliable?

Answer 9

It transfers by transactions so if any transfer fails it can roll back any records in transit. You can be confident all data has been transferred.

Question 10

When can AWS Database migration service be used?

Answer 10

• Once off migration
• Configured to move data on schedule
• Continous data replication where data is transferred from the siurce as soon as its made.