Kinesis

Question 1

What is Kinesis?

Answer 1

A managed alternative to Apache Kafka to Easily collect, process, and analyze video and data streams in real time

Question 2

What is mainly great for Kinesis?

Answer 2

for “Real-time” big data, for streaming processing frameworks (Spark, NiFi, etc…)

Question 3

How is data replicated in Kinesis?

Answer 3

automatically to 3 AZs

Question 4

What are the Kinesis services?

Answer 4

Kinesis Streams
Kinesis Analytics
Kinesis Data Firehose

Question 5

What is Kinesis Streams?

Answer 5

It is Kinesis itself, low latency streaming ingests at scale

Question 6

What is Kinesis Analytics?

Answer 6

managed service to perform real-time analytics on streams using SQL

Question 7

What is Kinesis Data Firehose?

Answer 7

fully managed service to load streams into S3, Redshift, ElasticSearch, Splunk

Question 8

What are common streams consumed by Kinesis Streams?

Answer 8

ClickStreams
IoT devices
Metrics and logs

Question 9

Are Kinesis Streams divided?

Answer 9

in ordered Shards / partitions

Question 10

What is the data retention period by default in Kinesis Streams?

Answer 10

1 day, up to 7

Question 11

What ability has Kinesis that SQS does not?

Answer 11

to reprocess / replay data

Question 12

How many consumers can have a Kinesis Stream?

Answer 12

multiple

Question 13

How does Kinesis scale out?

Answer 13

adding new shards, it does not auto scale

Question 14

What can’t you do to data inserted in Kinesis?

Answer 14

delete it, it is inmutable

Question 15

What is the writing speed of a Kinesis Stream Shard?

Answer 15

1 MB/s or 1000 messages at write PER SHARD

Question 16

What is the reading speed of a Kinesis Stream Shard?

Answer 16

2 MB/s at read PER SHARD

Question 17

How is billing in Kinesis Streams?

Answer 17

per shard provisioned

Question 18

What can happens to the number of shards over time?

Answer 18

can evolve, reshard or merge

Question 19

How are records ordered in Kinesis Streams?

Answer 19

per shard

Question 20

What contains a record sent from a producer to Kinesis?

Answer 20

A message key and the data itself

Question 21

What is useful for the record message key in Kinesis Streams?

Answer 21

The same key goes to the same partition (helps with ordering for a specific key)

Question 22

What gets a message sent to a Kinesis shard?

Answer 22

a sequence number

Question 23

What is hot partition in Kinesis?

Answer 23

A partition with a key that is not well distributed

Question 24

What can you do to reduce costs and increase throughput in Kinesis?

Answer 24

Use batching

Question 25

What is ProvisionedThroughputExceeded in Kinesis?

Answer 25

o Happens when sending more data (exceeding MB/s or TPS for any shard) o Make sure you don’t have a hot shard (such as your partition key is bad and too much data goes to that partition)

Question 26

What can you use to produce messages to Kinesis?

Answer 26

CLI, SDK, producer libraries from various frameworks

Question 27

What can you use to consume messages to Kinesis?

Answer 27

CLI, SDK, (KCL) Kinesis Client Library (in many languages)

Question 28

What security can you use in Kinesis?

Answer 28

Control access / authorization using IAM policies

Question 29

Can you encrypt data moving to and at rest in Kinesis?

Answer 29

Yes

Question 30

Can you access Kinesis offline?

Answer 30

Yes, using VPC endpoints

Question 31

Is Kinesis Data Firehose real time?

Answer 31

No, it is near real time, 60 seconds latency

Question 32

What is the format supported by Kinesis Data Firehose?

Answer 32

many

Question 33

What features support Kinesis Data Firehose?

Answer 33

supports conversions, transformations, compression

Question 34

How are you billed in Kinesis Data Firehose?

Answer 34

for the amount of data going through and conversions

Question 35

Who sends data to Kinesis Data Firehose?

Answer 35

Not just Kinesis Streams: | SDK, KPL (Kinesis Producer Library), Kinesis Agent, CloudWatch

Question 36

How can Kinesis Data Firehose transform data?

Answer 36

Using Lambda

Question 37

Where does Kinesis Data Firehose send the data?

Answer 37

To S3, Redshift, ElasticSearch, Splunk

Question 38

Where does Kinesis Data Firehose store the data?

Answer 38

no data storage, just ingest

Question 39

What can you create from Kinesis Analytics real-time queries?

Answer 39

new streams

Question 40

If I have 3 shards, and 2 contains a record and I receive a new record, where will it go?

Answer 40

It can go to any shard, supposing it has a new partition id

Question 41

Do you need to set the number of shards in kinesis?

Answer 41

Yes, Up to 200 shards

Question 42

How does work sending notifications in Kinesis?

Answer 42

Kinesis does not have such feature

Question 43

What you need to send messages to Kinesis?

Answer 43

PutRecord API + Partition key that gets hashed

Question 44

What are possible solutions to ProvisionedThroughputExceeded Exception in Kinesis

Answer 44

o Retries with backoff o Increase shards (scaling) o Ensure your partition key is a good one

Question 45

What is used by the Kinesis Client Library to complement its work?

Answer 45

KCL uses DynamoDB to track other workers and share the work amongst shards

Question 46

What is Kinesis Client Library?

Answer 46

Kinesis Client Library (KCL) is Java library that helps read record from a Kinesis Streams with distributed applications sharing the read workload

Question 47

What is the rule for shards in KCL?

Answer 47

each shard is to be read by only one KCL instance o Means 4 shards = max 4 KCL instances o Means 6 shards = max 6 KCL instances Many to One relationship

Question 48

What needs KCL to use DynamoDB?

Answer 48

IAM access

Question 49

Where can you run KCL?

Answer 49

can run on EC2, Elastic Beanstalk, on Premise Application

Question 50

How are records read by KCLs?

Answer 50

Records are read in order at the shard level, cross shard order is not guaranteed

Question 51

How is Kinesis security?

Answer 51

* Control access / authorization using IAM policies * Encryption in flight using HTTPS endpoints * Encryption at rest using KMS * Possibility to encrypt / decrypt data client side (harder) * VPC Endpoints available for Kinesis to access within VPC