Stream Data Processing

Question 1

Why can’t event stream data not be stored as big data?

Answer 1

It would result in one small file per event –> too many files

Question 2

Event hub

Answer 2

A buffer that buffers the events and stores them in batches.

Question 3

The different kinds of stream processing

Answer 3

1 - strema data integration

2 - stream analytics

Question 4

Stream data integration

Answer 4

Focuses on ingestion and processing of the data sources targeting ETL

Question 5

Stream analytics

Answer 5

Targets analytics use cases. Calculates aggregates and detects patterns.

Question 6

Native streaming

Answer 6

Events are processed as they arrive -> lowest latency but high fault tolerance

Question 7

Window

Answer 7

A certain amount of data to perform computations on

Question 8

Three types of windows

Answer 8

1 - fixed/tumbling windows
2 - sliding/hopping windows
3 - session windows

Question 9

Fixed/tumbling windows

Answer 9

Stops if the window is full, based on the count of items or the time

Question 10

Sliding/hopping windows

Answer 10

Stops based on window + sliding interval length

Question 11

Session windows

Answer 11

Sequences of temporarily related events terminated by a gap of inactivity

Question 12

Which two kinds of queries are there?

Answer 12

1 - ad-hoc queries

2 - standing queries

Question 13

Standing queries

Answer 13

Queries that are stored and permanently executed

Question 14

Ad-hoc queries

Answer 14

One time questions

Question 15

Main differences between batch processing and stream processing?

Answer 15

1 - The input is not controlled by the system

2 - The input timing/rate is often unknown

Question 16

Multi-query optimization

Answer 16

Using subqueries to not recompute things in different queries

Question 17

Ways to handle overload?

Answer 17

• Back pressure
• Load shedding
• Distributed stream processing

Question 18

Back pressure

Answer 18

Slows down the sources to avoid data loss (for example by blocking queries)

Question 19

Load shedding variants

Answer 19

1 - random sampling-based shedding
2 - relevance-based shedding
3 - summary-based shedding

Question 20

Random-sampling-based load shedding

Answer 20

Taking a random sample and provide an output based on this (approximation)

Question 21

Relevance-based load shedding

Answer 21

Use an algorithm that understands which distances are relevant while the ones that are not relevant are removed

Question 22

Summary-based load shedding

Answer 22

Given a queue, the summary of the queue is provided, which is used as input for the next step

Question 23

distributed stream processing

Answer 23

Distribute the stream based on data flow (distributes the query) or on key range (distrbutes the data stream itself)

Question 24

Three options for delivery guarantees

Answer 24

1 - at most once –> can cause data loss on failures
2 - at least once –> might create incorrect states because data is processed multiple times
3 - exactly once