Kinesis Consumers

Question 1

What are the 7 ways to Consume data from a Kinesis Stream?

Answer 1

• Kinesis SDK
• Kinesis Client Library (KCL)
• Kinesis Connector Library
• 3rd party libraries
• Kinesis Firehose
• AWS Lambda
• Kinesis Consumer Enhanced Fan Out

Question 2

What are some examples of 3rd party libraries to consume data from a stream?

Answer 2

• Spark
• Log4J
• Appenders
• Flume
• Kafka
• Connect

Question 3

What is the data read shard limit?

Answer 3

2MB

Question 4

How much data does the SDK GetRecords return?

Answer 4

Up to 10MB, which exceeds to 2MB/s limit, so you need to wait another 5 seconds for the next call

Question 5

How many GetRecords API calls can a shard make per second?

Answer 5

5 GetRecords API calls

Question 6

What is the GetRecords API latency?

Answer 6

200ms

Question 7

Can you increase throughput by adding more consumers to read from the same shard?

Answer 7

No, if more consumers read from the same shard, they share the 2MB/s limit and the 5 API calls. These limits are per shard per second

Question 8

How would you get around the issue of multiple consumers sharing the read limits of shards?

Answer 8

Use Fan out

Question 9

What are 6 features of the Kinesis Client Library?

Answer 9

• Exists for multiple languages such as Java, Node, Python
• Read records from the stream that were produced with the KPL (decode)
• Share multiple shards with multiple consumers in one group, and shard discovery
• Checkpointing system to resume progress
• Uses DynamoDB for Checkpointing
• Record Processors will process the data

Question 10

What should you do if your KCL is not reading fast enough even if your stream has enough throughput?

Answer 10

The DynamoDB may not have enough WCU/RCU to efficiently Checkpoint

Question 11

What are 4 features of using Lambda to read from a stream?

Answer 11

• Lambda has a library to de-aggregate records from KPL
• Lambda can be used to run lightweight ETL
• Lambda can be used to trigger
• Lambda has a configurable batch size

Question 12

What is the data limit between Kinesis Consumers and Kinesis Fan Out

Answer 12

Kinesis Fan Out pushes data to consumers to it gets 2MB per second per consumer vs 2MB per second per shard

Question 13

What are 3 reasons you would choose Classic Consumers?

Answer 13

• Low number of consuming applications, less than 5
• You can tolerate latency of about 200ms
• Minimize cost

Question 14

What are 4 reasons you would choose Enhanced Fan Out Consumers?

Answer 14

• Multiple consumers for the same stream
• Low latency of about 70ms
• Higher costs
• Default limit of 5 consumers using enhanced fan out per data stream

Question 15

How would you divide a hot shard?

Answer 15

Use shard splitting

Question 16

What is shard splitting?

Answer 16

It is a process of splitting an existing shard into 2 shards. The data in the existing shard remains until it is expired, then the 2 new shards take over

Question 17

What is merging shards?

Answer 17

It is to take shards away

Question 18

What happens when we merge shards?

Answer 18

Old shards are closed and deleted when data expires

Question 19

Is auto scaling possible in Kinesis?

Answer 19

Its possible but not easy. You can use the “UpdateShardCount” API but there are manual steps

Question 20

What are 3 main limitations of Kinesis scaling?

Answer 20

• Resharding cannot be done in parallel, so plan capacity in advance
• You can only perform 1 resharding operation at a time and it takes a few seconds
• For 1000 shards, it takes 30k seconds to double to 2000 shards, thats 8.3 hours

Question 21

What are 5 options for Kinesis Security?

Answer 21

• Control access using IAM policies
• Encryption in flight using HTTPS endpoints
• Encryption at rest using KMS
• Manually implemented Client Side encryption
• VPC endpoints to access within a VPC