Designing data pipelines

Question 1

What is a graph in the concept of DAGs?

Answer 1

A graph is a set of nodes linked by edges.

Question 2

three common types of data pipelines are:

Answer 2

1. Data warehousing pipelines
2. Stream processing pipelines
3. Machine learning pipelines

Question 3

when using GCP Cloud Dataproc for transformations

Answer 3

with cloud dataproc, transformations canbe written in spark or hadoop supported language.

Question 4

when using GCP Cloud Dataflow for transformations

Answer 4

when using cloud dataflow you write transformations using the Apache Beam model, which provides a unified batch and stream processing model

Question 5

Apache Beam has explicit support for pipeline constructs including:

Answer 5

1. Pipelines
2. PCollection
3. PTransform

Question 6

What is a streaming window and what are its types?

Answer 6

A window is a set of consecutive data points in a stream. Windows have a fixed width and a way of advancing. Windows that advance by a number of data points less than the width of the window are called sliding windows; windows that advance by the length of the window are tumbling windows.

Question 7

When are sliding windows used?

Answer 7

Sliding windows are used when you want to show how an aggregate - such as the average of the last three values - change over time.

Question 8

When would you use tumbling windows?

Answer 8

Tumbling windows are used when you want to aggregate data over a fixed period of time, for example, for the last minute.

Question 9

GCP has several services that are commonly used components of pipelines, including:

Answer 9

1. Cloud pub/sub
2. cloud dataflow
3. cloud dataproc
4. cloud composer

Question 10

what are messaging queues?

Answer 10

Messaging queues are used in distributed systems to decouple services in a pipeline. This allows one service to produce more output than the consuming service can process without adversely affecting the consuming service.