Kinesis Producers Flashcards Preview

AWS Data Analytics Cert > Kinesis Producers > Flashcards

Flashcards in Kinesis Producers Deck (13)
Loading flashcards...
1
Q

What are the 4 options to send data into a Kinesis Stream?

A
  • Kinesis SDK
  • Kinesis Producer Library (KPL)
  • Kinesis Agent
  • 3rd party libraries
2
Q

What is the Kinesis Producer Library (KPL)?

A

It is a more advanced way to send data to the stream

3
Q

What is the Kinesis Agent?

A

It is an agent that runs on a Linux server

4
Q

What are some examples of 3rd party libraries to connect to Kinesis?

A
  • Spark
  • Log4J
  • Appenders
  • Flume
  • Kafka Connect
  • NiFi
5
Q

When would you decide to use Kinesis SDK?

A

A case with low throughput, higher latency, simple API using AWS Lambda etc

6
Q

What would a “ProvisionedThroughputExceeded” exception indicate?

A

It indicates you have exceeded you data sending limits

7
Q

What are the 3 possible solutions to resolve “ProvisionedThroughputExceeded”?

A
  • Retry with backoff
  • Increase shards (scaling)
  • Ensure your partition key is good
8
Q

What are the 8 features of the Kinesis Producer Library (KPL)?

A
  • Easy to use and highly configurable with C++/Java
  • Used for building high performance, long-running producers
  • Automated and configurable retry mechanism
  • Synchronous or Asynchronous API
  • Submits records to Cloudwatch for monitoring
  • 2 Batch features enabled by default
  • Compression must be enabled by the user
  • KPL records must be decoded by KCL
9
Q

What are the 2 Kinesis Producer Library (KCL) Batch features enabled by default?

A
  • Collect records and write to multiple shards in the same PutRecords API call
  • Aggregate records
10
Q

What are the 2 things that happen when Kinesis Producer Library (KPL) Aggregates records?

A
  • Store multiple records in 1 record, allowing you to go over the 1000 records per limit
  • Increase payload size and improve throughput (max 1MB/s)
11
Q

What is an example of Kinesis Producer Library (KPL) Batching records?

A

The limit in Kinesis is 1MB. But if KPL finds 3 records that have a combined total less than that, it will combine them into 1 message.

12
Q

How does KPL know how long to wait before sending batched records?

A

You can set this with the “RecordMaxBufferedTime”, which defaults to 100ms

13
Q

What are the 4 features of Kinesis Agent?

A
  • Write from multiple directories and write to multiple streams
  • Routing feature based on directory/log file
  • Pre-processing data before sending to streams (csv to json etc)
  • Handles log rotation and retry