Section 19: AWS Integration & Messaging: SQS, SNS & Kinesis: Kinesis

Question 1

What does AWS Kinesis do?

Answer 1

Collect, process, and analyze real-time video and data streams

Question 2

What kind of real-time data is Kinesis well suited to ingest?

Answer 2

logs, metrics, website clickstreams, IoT telemetry data

Question 3

There are four types of Kinesis, what are they? (names only)

Answer 3

• Kinesis Data Streams
• Kinesis Data Firehose
• Kinesis Data Analytics
• Kinesis Video Streams

Question 4

Which Kinesis type is best to: capture, process, and store data streams?

Your options are: Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, Kinesis Video Streams.

Answer 4

Kinesis Data Streams

Question 5

Which Kinesis type is best to: load data streams into AWS data stores?

general Kinesis pic

Your options are: Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, Kinesis Video Streams.

Answer 5

Kinesis Data Firehose

Question 6

Which Kinesis option is best suited to: analyze data streams with SQL or Apache Flink?

Your options are: Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, Kinesis Video Streams.

Answer 6

Kinesis Data Analytics

Question 7

Which Kinesis option is best to: capture, process, and store video streams?

general kinesis pic.

Your options are: Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, Kinesis Video Streams.

Answer 7

Kinesis Video Streams.

Question 8

What is a shard, in terms of data?

Answer 8

A part of a dataset, when that dataset has been partitioned.

Question 9

Can applications, clients, sdk, kinesis produer library (kpl), and kinesis agents all be Kinesis Data Stream producers?

Answer 9

Yes. maybe even at the same time idk.

Question 10

Is a Kinesis Data Stream stream made up of shards?

Answer 10

yes. Again, as it pertains to data, a shard is a part of a dataset.

Question 11

Can you scale up the number of shards in a Kinesis Data Stream… stream?

Answer 11

Yes

Question 12

1. What is the max size of a record that can be sent to a Kinesis Data Stream?
2. What is the MB/sec throughput at which a record can be sent to a Kinesis Data Stream shard?
3. What is the alternative number of messages per sec throughput at which a records can be sent to a Kinesis data stream shard?

Answer 12

1. 1 MB.
2. 1 MB/sec
3. 1000 msg/sec

Question 13

When sending a record from a producer to a Kinesis stream-shard, what three things does the record consist?

Answer 13

1. sequence number (unique per partition-key within shard)
2. A partition key (A partition key is used to group data by shard within a stream.) (myst specify while put records into stream)
3. data blob (up to 1 MB)

Question 14

Can all of the following be consumers of Kinesis Data Stream records?
* lambda
* kinesis data firehose
* kinesis data analytics
* custom consumer (aws sdk) - Classic or Enhanced Fan Out
* Kinesis Client Library (KCL) - Library to simplify reading from a data stream
* apps with an ec2 symbol was also one of the slides, but not another

Answer 14

Yes

Question 15

What are the three things that are in each record being sent from a Kinesis Data Stream stream-shard to a Kinesis Data Stream consumer?

Answer 15

• Partition key
• Sequence number
• data blob

Question 16

What are the two reates at which records can be sent from a Kinesis Data Stream stream-shard to a Kinesis Data Stream consumer?

Answer 16

1. 2 MB/sec (shared) per shard - all consumers
   OR
2. 2 MB/sec (enhanced) per shard per consumer

I’m thinking you can only pick one per kinesis data stream? or maybe only one can happen at a time?

Question 17

About Kinesis Data Streams:
1. what is the retention
2. do you have the ability to reprocess (replay) data
3. can data be deleted once it’s been inserted into Kinesis?

Answer 17

1. between 1 day and 365 (inclusive (including 1 and 365))
2. yes
3. Nope. data inserted into Kinesis is immutable.

Question 18

About Kinesis Data Streams:
4. Does data that shares the same partition go into the same shard (this is really confusing, given that I assumed a shard was just a partition).

Answer 18

Yes, they call it ordering.

Question 19

1. What are the producers available to have as a Kinesis Data Stream

Answer 19

1. AWS SDK;
2. Kinesis Producer Library (KPL);
3. Kinesis Agent

Question 20

2. What are the consumers available to a Kinesis Data Stream?

Answer 20

1. you can write your own Kinesis Client Library (KCL) or by using AWS SDK
2. Alternatively, you can use a managed consumer like aws lambda, kinesis data firehose, or kinesis data analytics.

Question 21

Kinesis Data Streams have two capacity modes, what are they?

Answer 21

1. Provisioned
2. on-demand mode

Question 22

You need the following things, do you use Kinesis Data Stream Provisioned capacity mode, or Kinesis Data Stream On-Demand capacity mode?
* you need to choose the number of shards provisioned, and scale manually or using API
* you need each shard to have up to 1MB/s in (or 1000 records per second)
* you need each shard to get 2MB/s out (for a classic or enhanced fan-out consumer)
* you need to pay per shard provisioned per hour

Answer 22

Capacity mode: Provisioned

Question 23

You need the following things, do you use Kinesis Data Stream Provisioned capacity mode, or Kinesis Data Stream On-Demand capacity mode?
* you don’t want to provision or manage the capcity
* you’re perfectly happy with 4MB/s in or 4000 records per second (this is the default capacity provisioned for this capacity mode)
* you’re very happy for your stream to scale automatically based on observed throughout peak during the last 30 days
* you’re happy to pay per stream per hour & data in/out per GB

Answer 23

Capacity mode: On-demand mode

Question 24

Let’s talk about Kinesis Data Stream Security:
1. how do you control access/authorization?
2. can you do encryption in flight? using what?
3. can you do encryption at rest? using what?

Answer 24

1. control access/authorization using IAM policies
2. encryption in flight using https endpoints
3. encryption at rest using kms

Question 25

Let's talk about Kinesis Data Stream Security: 1. can you implement encryption/decryption of data on the client side? 2. are vpc endpoints available for kinesis to access within VPC? 3. how can you monitor api calls?

Answer 25

1. you can implement encyrption/decryption of data on the client side, but it is harder 2. vpc endpionts are available for Kinesis to access within VPC 3. you can monitor api calls using CloudTrail

Question 26

What do Kinesis producers do?

Answer 26

put data records into data streams

Question 27

What are the Kinesis data stream producer options? (possible these kinesis producers are not specific to data streams, but applicable to the other kinesis subservices (firehose, video stream, analystics)

Answer 27

* aws sdk (simple producer) * kinesis produver library (kpl): C++, Java, batch, compressoin, retries * kinesis agent (monitor log files)

Question 28

Kinesis producer write throughput?

Answer 28

1MB/sec or 1000 records/sec per shard

Question 29

Kinesis producer APIs?

Answer 29

* PutRecord API * PutRecords API - this one will help reduce costs and increase throughput

Question 30

# T/F A hash function is used to map record partition keys (from the producer) to the shard handling that particular set of partition keys.

Answer 30

True

Question 31

What is the SubscribeToShard API

Answer 31

The SubscribeToShard API is a high-performance streaming API that pushes data from shards to consumers over a persistent connection without a request cycle from the client. The SubscribeToShard API uses the HTTP/2 protocol to deliver data to registered consumers whenever new data arrives on the shard, typically within 70 milliseconds, offering approximately 65% faster delivery compared to the GetRecords API. The consumers will enjoy fast delivery even when multiple registered consumers are reading from the same shard.

Question 32

Q: What is enhanced fan-out? Q: When should I use enhanced fan-out?

Answer 32

* Enhanced fan-out is an optional feature for Kinesis Data Streams consumers that provides logical 2 MB/second throughput pipes between consumers and shards. This allows you to scale the number of consumers reading from a data stream in parallel, while maintaining high performance. * You should use enhanced fan-out if you have, or expect to have, multiple consumers retrieving data from a stream in parallel, or if you have at least one consumer that requires the use of the SubscribeToShard API to provide sub-200 millisecond data delivery speeds between producers and consumers.

Question 33

1. Why might you be seeing "ProvisionedThroughputExceeded"? 2. What can you do about it option 1. 3. What can you do about it option 2. 4. What can you do about it option 3.

Answer 33

1. You might be seeing it if your from-producers-traffic grew more than double the previous peak within a 15 minute duration. 2. add more shards, then retry the throttled requests. 3. retries with exponential backoff 4. use highly distributed partition key

Question 34

What is high distrubution, as it relates to keys?

Answer 34

Key distribution * An indexed storage system will use keys to determine where it should store, or look for, associated data. One strategy to optimize access to data is to spread that data out over a number of storage locations. As a result, because each location should have less data, it will be faster to find something within a given collection. Consider having one million records: searching for a given record will, in the worst case, require the system to look at one million different elements to find what was asked for. On the other hand, if those one million records were divided into one-thousand groups, being able to limit your search to one group would reduce that worst-case to only one thousand elements. * The relative probability that a key will direct a search to a given location is known as the "key distribution". An even distribution means that, for any given key, the probability of being directed to any location is as likely as another and finding data will therefore be more efficient.

Question 35

What does "retries with exponential backoff" mean?

Answer 35

Retries with exponential backoff is a technique that retries an operation, with an exponentially increasing wait time, up to a maximum retry count has been reached (the exponential backoff). https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/implement-retries-exponential-backoff (everything else is from aws and steph's course)

Question 36

What are some Kinesis Data Stream Consumers?

Answer 36

* lambda * kinesis data firehose * kinesis data analytics * custom consumer (aws sdk) - Classic or Enhanced Fan Out * Kinesis Client Library (KCL) - Library to simplify reading from a data stream * apps with an ec2 symbol was also one of the slides, but not another

Question 37

What do Kinesis Data Stream Consumers do?

Answer 37

get data records from data streams and process them

Question 38

There are two Kinesis Custom Consumer options, what are they? (name only)

Answer 38

Shared (Classic) Fan-out and Enhanced Fan-out consumer

Question 39

What is the main difference between Kinesis Custom Consumers: Shared (Classic) Fan-Out Consumer and Enhanced Fan-out Consumer?

Answer 39

* With Shared Classic Fan-Out Consumer you get 2 MB/sec per shard across all consumers. That means if four consumers are getting records from Shard 1, then each consumer might be taking up 0.5 MB/sec throughput (it was not mentioned whether throughput was distrubuted easily, i just made up those numbers for the purpose of illustration). * With enhanced fan out consumers you get 2MB/sec per consumer per shard. That means if four consumers are trying to get a record for Shard 1, each consumer might be taking up 2MB/sec throughput, and together the four consumers might be using 8 MB/sec throughput.

Question 40

What is the API consumers use to obtain data records from Kinesis Data Stream Shards?

Answer 40

For Shared (Classis) Fan-out COnsumer the api is GetRecords(), but for Enhanced Fan-out Consumer it's SubscribeToShard().

Question 41

Does this describe Kinesis Shared Classic Fan-Out Consumers or enhanced fan-out Consumers? * allows only for (or perhaps best for) a low number of consuming applications * max 5 GetRecords API calls/sec * latency ~200 ms * Minimizes costs * consumers poll data from Kinesis using GetRecords API call * Returns up to 10 MB (then throttle for 5 seconds) or up to 10000 records

Answer 41

Shared (Classic) Fan-Out Consumer

Question 42

Is Kinesis Consumer Shared (Classic) Fan-out Consumer the kind where consumers poll data from Kinesis using GetRecords API, or the kind where Kinesis pushes data to consumers over HTTP/2 using SubscribeToShard API?

Answer 42

It's the kind where consumers poll data from Kinesis using GetRecords API.

Question 43

Is Kinesis Consumer Enhanced Fan-out Consumer the kind where consumers poll data from Kinesis using GetRecords API, or the kind where Kinesis pushes data to consumers over HTTP/2 using SubscribeToShard API?

Answer 43

The kind where kinesis pushes data to consumers over HTTP/2 using SubscribeToShard API.

Question 44

# T/F does this describe enhanced fan out consumer? * multiple consuming applications for the same stream * 2MB/sec per consumer per shard * Latency ~70ms * Higher costs ($$$) * Kinesis pushes data to consumers over HTTP/2 SubscribeToShard API * Soft limit of 5 consumer applications (KCL) per data stream (default)

Answer 44

T

Question 45

Kinesis Consumers - AWS Lambda 1. Does it support classic and enhanced fan-out consumers? 2. What happens if you're using kinesis data streams with aws lambda and an error occurs?

Answer 45

1. yes 2. lambda will retry until it succeeds or data expires.

Question 46

How many batches per shard can kinesis data streams + aws lambda handle simultaneously?

Answer 46

10

Question 47

When using Kinesis Consumers with AWS Lambda: 1. are records read in batches? 2. can you configure batch size and batch window?

Answer 47

1. Yes 2. Yes

Question 48

You want to use Kinesis Data Streams cuz you have a ton of streaming data. What's a popular aws service for processing records, and where is a popular place to save records after that?

Answer 48

1. AWS Lambda 2. DynamoDB

Question 49

What is the Kinesis Client Library (KCL)?

Answer 49

A java library that helps read records from a Kinesis Data Stream with distributed applications sharing the read workload.

Question 50

Is a java library basically just a python package?

Answer 50

no it sounds like both java and python have packages and libraries. It seems like libraries tend to have maybe a ton of packages in them? or a bunch of smaller pieces of code anyway.

Question 51

If you're using Kinesis Client Libraries each shard is to be read by only one KCL instance (where a KCL instance might be running on ec2 or elastic beanstalk or on some on-premise solution). Does this mean each shard needs it's own KCL instance?

Answer 51

No. It means that two shards could be being read from the same KCL instance, but that each shard will only get read by one instance. Just look at the picture.

Question 52

What can KCL run on?

Answer 52

ec2, elastic beanstalk, on-premise

Question 53

when using KCL, are records read in order at the shard level?

Answer 53

Yes.

Question 54

When using KCL, is progress checkpointed into DynamoDB? Do you need a kind of access for that?

Answer 54

1. yes 2. yes, IAM access

Question 55

1. What type(s) of kinesis data stream consumer does KCL 1.x support? 2. KCL 2.x?

Answer 55

1. shared consumer 2. supports shared and enhanced fan out consumer

Question 56

How do you increase the stream capacity? (what is stream capacity?)

Answer 56

1. well i think stream capacity is the number of shards in your stream. 2. you can increase it by shard splitting (a kinesis operation). When you shard split, you turn one "hot shard" into two shards. In split shards, each shard gets 1MB/s data in (incoming?) per shard). You can't split a shard into more than two shards in a single operation. There is no automatic scaling - this happens manually.

Question 57

You maybe don't have a ton of use for your shards at the moment. How can you save costs?

Answer 57

* You can decrease stream capacity by merging shards (shard merging is another kinesis operation). * you can merge two shards with low traffic (shards with low traffic are called cold shards, btw). You can't merge more than two shards in a single operation. * old shards are closed and will be deleted once the data is expired.

Question 58

Popular producers (also called sources, though that understanding should be verified, not feeling particularly careful atm) for Kinesis Data Firehose?

Answer 58

* Apps * clients * sdks * kinesis agents * AWS IoT * Amazon CloudWatch (Logs and Events) * Kinesis Data Streams

Question 59

Size of records that can go through Kinesis Data Firehose ()

Question 60

What is a destination in Kinesis Data Firehose?

Answer 60

A destination is the data store where your data will be delivered. Kinesis Data Firehose currently supports Amazon S3, Amazon Redshift, Amazon OpenSearch Service, Splunk, Datadog, NewRelic, Dynatrace, Sumo Logic, LogicMonitor, MongoDB, and HTTP End Point as destinations.

Question 61

Q: What is Streaming ETL?

Answer 61

Streaming ETL is the processing and movement of real-time data from one place to another. ETL is short for the database functions extract, transform, and load. Extract refers to collecting data from some source. Transform refers to any processes performed on that data. Load refers to sending the processed data to a destination, such as a warehouse, a datalake, or an analytical tool.

Question 62

Q: What is a delivery stream in Kinesis Data Firehose?

Answer 62

A delivery stream is the underlying entity of Kinesis Data Firehose. You use Firehose by creating a delivery stream and then sending data to it. You can create an Kinesis Data Firehose delivery stream through the Firehose Console or the CreateDeliveryStream operation. For more information, see Creating a Delivery Stream.

Question 63

Q: What is a record in Kinesis Data Firehose?

Answer 63

* A record is the data of interest your data producer sends to a delivery stream. The maximum size of a record (before Base64-encoding) is 1024 KB if your data source is Direct PUT or Kinesis Data Streams. The maximum size of a record (before Base64-encoding) is 10 MB if your data source is Amazon MSK. * Okay, this is possibly noteworthy. Steph says that the max size of a record is 1MB. the aws docs says the max size of record (within parameters mentioned above) is 1024 KB. It turns out that 1MB is 1000 KB. So I'm thinking maybe he rounded here, and am wondering where else he might have rounded. So if you see that on the exam go with either. Maybe 1024 first. https://aws.amazon.com/kinesis/data-firehose/faqs/

Question 64

How does (or maybe it's just one way of doing this) Kinesis Data Firehose get data to whatever destinations the data is going?

Answer 64

batch writes.

Question 65

What aws service is popularly used (or perhaps the only option?) by Kinesis Data Firehose for data transformation?

Answer 65

AWS Lambda/lambda functions.

Question 66

When you're thinking about Kinesis Data Firehose, what is important to remember about costs?

Answer 66

You pay for data going through FIrehose.

Question 67

Does Kinesis Data Firehose work in real time?

Answer 67

No, it works in near real time. It has 60 second latency minimum for non full batches, or a minimum of 1MB of data at a time (you have the option)

Question 68

Can Kinesis Data Firehose send failed or all data to a backup S3 backup?

Answer 68

Yes

Question 69

What's your guess for when you'd use Kinesis Data Streams vs Kinesis Data Firehose?

Answer 69

My guess is you'd use Kinesis Data Streams when you're like, "no, i need real time (~200ms) processing/movement/whatever and I don't care if I have to write custom code for my producers/consumers to get that real time stuff. and yes I need data storage for 1 to 365 days and replay capability". And then you'd use Kinesis Data Firehose when you're like "actually i don't care if processing/storage happens in about ~200 ms, I'm perfectly happy with that processing/tranforming/storage stuff taking about 60 seconds and I do like that firehose is aws fully managed and yes I will gladly pay the extra cost for all that. And also I don't need data storage or replay capability".

Question 70

# T/F does this look accurate? 1. Kinesis Data FIrehose: * streaming service for ingest at scale * write custom code for producers consumers * real time (~200ms) * manage your own scaling using shard splitting and merging * data storage for 1 to 365 days * supports replay capability AAAAAAAAAAAA 2. Kinesis Data Streams * Load streaming data into S3/Redshift/OpenSearch/3rd party/custom HTTP * fully managed * near real-time (buffer time of about 60 seconds) * automatic scaling * no data storage * doesn't support replay capability

Answer 70

Nope. 1 is actually Kinesis Data Streams, and 2 is actually Kinesis Data Firehose.

Question 71

Okay, Kinesis Data Analyticsfor SQL applications. 1. What do they call producers (hint, it's not producers). 2. what do they call consumers (hint, it's not 'consumers')

Answer 71

1. sources 2. sinks (eye roll. like they couldn't just use the same term)

Question 72

Okay, so what are the two options for Kinesis Data Analytics for SQL Application sources?

Answer 72

1. Kinesis Data Streams 2. inesis Data Firehose

Question 73

Aside from sources (Kinesis Data Streams or Kinesis Data Firehose), what are the other inputs needed for Kinesis Data Analytics for SQL Applications?

Answer 73

1. SQL Statements 2. Reference Data from S3 (used to enrich streaming data - note that i'm not 100% sure you *have* to add the reference data from S3 and a hands on would probably clear that up)

Question 74

What are the sink options for Kinesis Data Analytics for SQL Applications (just names)?

Answer 74

Okay, unfortunately aws is confusing about this. 1. Per steph's slides, it looks like Kinesis Data Analytics for SQL Applications can connect directly to sink options Kinesis Data Streams (which can connect directly to aws lambda or applications) and Kinesis Data Firehose (which can connect to s3 or redshift-copy-through-s3 or other firehose destinations). 2. *However* the aws doc https://docs.aws.amazon.com/kinesisanalytics/latest/dev/what-is.html is not clearly written. In one place it says what steph says above. But in the next paragraph it says that "Kinesis Data Analytics supports Amazon Kinesis Data Firehose (Amazon S3, Amazon Redshift, Amazon OpenSearch Service, and Splunk), AWS Lambda, and Amazon Kinesis Data Streams as destinations.". It's possible that the secondary paragraph, the one in quotations above, is referring to Kinesis Data Analytics *as a whole* and the thing steph and aws agree on is specific to Kinesis Data Analytics for SQL Applications? But then again, that quote is from a page *titled* Kinesis Data Analytics for SQL Applications. Good luck figuring that out.

Question 75

1. With Amazon Kinesis Data Analytics for SQL Applications elevator pitch. 2. Name a few use cases for Kinesis Data Analytics for SQL Applications.

Answer 75

1. A: With Amazon Kinesis Data Analytics for SQL Applications, you can process and analyze streaming data using standard SQL (first line from an aws doc). B: real time analytics on Kinesis Data Streams and Firehose using SQL (steph slide) 2. The service enables you to quickly author and run powerful SQL code against streaming sources to perform time series analytics, feed real-time dashboards, and create real-time metrics.

Question 76

So lets say the two output/sink options for Kinesis Data Analytics for SQL Applications are Kinesis Data Streams and Kinesis Data Firehose. When would you use the former, and when would you use the latter?

Answer 76

1. you'd use Kinesis Data Stream as an ouput when you want to create streams out of hte real-time analytics queries 2. you'd use Kinesis Data Firehose as an output when you want to send analytics query results to destinations

Question 77

1. Is Kinsis Data Analytics for SQL Applications fully manged? 2. does it scale automatically or manually 3. do you pay for anything?

Answer 77

1. fully managed, no servers to provision 2. automatic scaling 3. pay for actual consumption rate

Question 78

Kinesis Data Analytics for Apache Flink, it's new name is?

Answer 78

Amazon Managed Service for Apache Flink

Question 79

Amazon Managed Service for Apache Flink, it's old name was?

Answer 79

Kinesis Data Analytics for Apache Flink

Question 80

What is apache flink?

Answer 80

Stateful Computations over Data Streams Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. source: https://flink.apache.org/, since this quote didn't come from steph or aws.

Question 81

What Does Kinesis Data Analytics for Apache Flink (Amazon Managed Service for Apache Flink) do?

Answer 81

It uses Flink (Java, Scala or SQL) to process and analyze streaming data.

Question 82

# What aw 1. what aws services seem like popular inputs (a name like sources, consumers whatever isn't actually given for inputs this time) for Kinesis Data Analtyics for Apache Flink (Amazon Managed Service for Apache Flink) 2. Is Firehose an option?

Answer 82

* Kinesis Data Streams * Amazon MSK (whatever that is) * another slide says that Flink does not read from Firehose, but that you can use Kinesis Analytics for SQL instead.

Question 83

# T/F The Kinesis Data Analytics for Apache Flink (Amazon Managed Service for Apache Flink) flowchart doesn't really look like there's anything going on other than an input going into Kinesis Data Analytics for Apache Flink (Amazon Managed Service for Apache Flink).

Answer 83

True.

Question 84

Is Kinesis Data Analytics for Apache Flink (Amazon Managed Service for Apache Flink) aws managed?

Answer 84

Yes. Apache Flink applications run on a managed aws cluster (hope i'm getting this wording correct)

Question 85

What does aws management include for Kinesis Data Analytics for Apache Flink (Amazon Managed Service for Apache Flink)

Answer 85

* provisioning compute resources, parallel computation, automatic scaling * application backups (implemented as checkpoints and snapshots) * use any apache flink programming features

Question 86

Is there application backup when using Kinesis Data Analytics for Apache Flink (Amazon Managed Service for Apache Flink)?

Answer 86

Yes. application backups are implemented as checkpoints and snapshots.

Question 87

Can you use any apache filnk programming feature when using Kinesis Data Analytics for Apache Flink (Amazon Managed Service for Apache Flink)?

Answer 87

Yes

Question 88

# Ordering data into (getting data in an orderly manner into) Kinesis Say you have 100 trucks (truck_1, ..., truck_100) on the road sending their GPS positions regularly into AWS. You want to consume the data in order for each truck, so that you can track their movement accurately. How should you send that data into Kinesis?

Answer 88

* Send using "Partition Key" value of the "truck_id. * The same key will always go to the same shard.

Question 89

# Ordering data into (getting data in an orderly manner into) SQS For SQS standard, is there ordering? How?

Answer 89

For SQS standard, there is no ordering.

Question 90

For SQS FIFO is there ordering? How does it work if you have one consumer?

Answer 90

For SQS FIFO, if you don't use a Group ID, messages are consumed in the order they are sent, with only one consumer (i think they mean that only one consumer is used with this method).

Question 91

For SQS FIFO is there ordering? How does it work if you have more than one consumer?

Answer 91

If you want to scale out the number of consumers and you want the messages to be grouped when they are related to each other, then you use a Group ID (similar to Partition Key in Kinesis).

Question 92

# Kinesis vs SQS ordering. Assume you have 100 trucks, 5 kinesis shards, 1 sqs fifo. For Kinesis Data Streams: 1. how many trucks per shard do you have on average? 2. will trucks have their data ordered within each shard? 3. what is the max amount of consumers you can have in parallel? 4. what is the max MB/s you can receive?

Answer 92

1. 20 2. yes 3. 5 4. 5 MB/s

Question 93

# Kinesis vs SQS ordering. Assume you have 100 trucks, 5 kinesis shards, 1 sqs fifo. For SQS FIFO: 1. how many SQS queues do you have? 2. how many Group IDs 3. up to how many consumers can you have? 4. up to how many messages per second can you have?

Answer 93

1. one 2. 100 3. 100 (due to the 100 group id) 4. 3000 if using batching

Question 94

Which one is Kinesis, which one is SQS, and which one is SNS?

Answer 94

1. SQS 2. SNS 3. Kinesis