SQS, SNS, Kinesis Flashcards
(63 cards)
1
Q
- Oldest offering (over 10 years old)
- Fully managed service, used to decouple applications
- Attributes:
- Unlimited throughput, unlimited number of messages in queue
- Default retention of messages: 4 days, maximum of 14 days
- Low latency (<10 ms on publish and receive)
- Limitation of 256KB per message sent
- Can have duplicate messages (at least once delivery, occasionally)
- Can have out of order messages (best effort ordering)
A
Amazon SQS – Standard Queue
2
Q
Produced to SQS using the ______?
A
SDK (SendMessage API)
3
Q
The message is ___________ in SQS until a consumer deletes it
A
persisted
4
Q
SQS – Producing Messages - Message retention
A
default 4 days, up to 14 days
5
Q
What is the SQS standard
A
unlimited throughput
6
Q
SQS Consumers run on ____?
A
EC2 instances
On-premise servers
AWS Lambda
7
Q
How many messages can a SQS Consumer receive at a time?
A
10
8
Q
SQS Consumer - Delete the messages using the __________ API
A
DeleteMessage
9
Q
- Consumers receive and process messages in parallel
- At least once delivery
- Best-effort message ordering
- Consumers delete messages after processing them
- We can scale consumers horizontally to improve throughput of processing
A
SQS – Multiple EC2 Instances Consumers
10
Q
Amazon SQS - Encryption
A
- In-flight encryption using HTTPS API
- At-rest encryption using KMS keys
- Client-side encryption if the client wants to perform encryption/decryption i
11
Q
Amazon SQS - Access Controls
A
IAM policies to regulate access to the SQS API
12
Q
2 uses for SQS Access Policies?
A
- Useful for cross-account access to SQS queues
- Useful for allowing other services (SNS, S3…) to write to an SQS queue
13
Q
After a message is polled by a consumer, it becomes _______ to other consumers
A
invisible
14
Q
By default, the “message visibility timeout” is __________?
A
30 seconds
15
Q
If a message is not processed within the visibility timeout, it will be processed ___________?
A
TWICE
16
Q
A consumer could call the ____________ API to get more time
A
ChangeMessageVisibility
17
Q
What happens if a visibility timeout is high (hours)?
A
consumer crashes, re-processing will take time
18
Q
What happens if visibility timeout is too low (seconds)?
A
we may get duplicates
19
Q
If someone wants to decrease the latency and increase efficiency and decrease there API calls to a SQS Queue .. what should they do?
A
Long Polling
20
Q
- Limited throughput: 300 msg/s without batching, 3000 msg/s with
- Exactly-once send capability (by removing duplicates)
- Messages are processed in order by the consumer
A
Amazon SQS – FIFO Queue
21
Q
What if you want to send one message to many receivers?
A
Amazon SNS
22
Q
- The “event producer” only sends message to one SNS topic
- As many “event receivers” (subscriptions) as we want to listen to the SNS topic notifications
- Each subscriber to the topic will get all the messages (note: new feature to filter messages)
- Up to 12,500,000 subscriptions per topic
- 100,000 topics limit
A
Amazon SNS
23
Q
What are the 2 types of publishing for AWS SNS?
A
Topic Publish (using the SDK)
Direct Publish (for mobile apps SDK)
24
Q
Amazon SNS –
* Create a topic
* Create a subscription (or many)
* Publish to the topic
A
Topic Publish (using the SDK)
25
Amazon SNS –
* Create a platform application
* Create a platform endpoint
* Publish to the platform endpoint
* Works with Google GCM, Apple APNS, Amazon ADM...
Direct Publish (for mobile apps SDK)
26
What is SNS Encryption?
* In-flight encryption using HTTPS API
* At-rest encryption using KMS keys
* Client-side encryption if the client wants to perform encryption/decryption itself
27
2 uses for the SNS Access Policies
* Useful for cross-account access to SNS topics
* Useful for allowing other services ( S3...) to write to an SNS topic
28
* Push once in SNS, receive in all SQS queues that are subscribers
* Fully decoupled, no data loss
* SQS allows for: data persistence, delayed processing and retries of work
* Ability to add more SQS subscribers over time
* Make sure your SQS queue access policy allows for SNS to write
* Cross-Region Delivery: works with SQS Queues in other regions
SNS + SQS: Fan Out
29
For the same combination of: event type (e.g. object create) and prefix (e.g. images/) you can only have one S3 Event rule.
What do you use if you want to send the same S3 event to many SQS queues?
Fan Out Method
30
With SNS - FIFO Topics, what type of SQS queues can you have as subcribers?
SQS FIFO
31
Does Amazon SNS – FIFOTopic have limited throughput?
YES! Just like SQS FIFO
32
* JSON policy used to filter messages sent to SNS topic’s subscriptions
* If a subscription doesn’t have a filter policy, it receives every message
SNS – Message Filtering
33
* Makes it easy to collect, process, and analyze streaming data in real-time
* Ingest real-time data such as: Application logs, Metrics, Website clickstreams, IoT telemetry data...
Kinesis
34
capture, process, and store data streams
Kinesis Data Streams
35
load data streams into AWS data stores
Kinesis Data Firehose
36
analyze data streams with SQL or Apache Flink
Kinesis Data Analytics
37
capture, process, and store video streams
Kinesis Video Streams
38
What are the 4 Kinesis Products?
Kinesis Data Streams
Kinesis Data Firehose
Kinesis Data Analytics
Kinesis Video Streams
39
Kinesis Data Streams - Retention
between 1 day to 365 days
40
Does Kinesis Data Streams have the ability to reprocess (replay) data??
YES
41
Once data is inserted in Kinesis, it ________?
can’t be deleted (immutability)
42
Does data that share the same partition go to the same shard?
YES
43
What is it called when data shares the same partition and goes to the same shard?
Ordering
44
What are 3 examples of Kinesis Data Streams Producers?
AWS SDK, Kinesis Producer Library (KPL), Kinesis Agent
45
What are the 2 different types and they're examples of Kinesis Data Streams Consumers?
Write your own: Kinesis Client Library (KCL), AWS SDK
Managed: AWS Lambda, Kinesis Data Firehose, Kinesis Data Analytics,
46
Kinesis Data Streams – Capacity Modes (2)
Provisioned mode
On-demand mode
47
Kinesis Data Streams – Capacity Modes:
* No need to provision or manage the capacity
* Default capacity provisioned (4 MB/s in or 4000 records per second)
* Scales automatically based on observed throughput peak during the last 30 days
* Pay per stream per hour & data in/out per GB
On-demand mode
47
Kinesis Data Streams – Capacity Modes:
* You choose the number of shards provisioned, scale manually or using API
* Each shard gets 1MB/s in (or 1000 records per second)
* Each shard gets 2MB/s out (classic or enhanced fan-out consumer)
* You pay per shard provisioned per hour
Provisioned mode
48
* Control access / authorization using IAM policies
* Encryption in flight using HTTPS endpoints
* Encryption at rest using KMS
* You can implement encryption/decryption of data on client side (harder)
* VPC Endpoints available for Kinesis to access within VPC
* Monitor API calls using CloudTrail
Kinesis Data Streams Security
49
Fully Managed Service, no administration, automatic scaling, serverless
Kinesis Data Firehose
50
What are 3 consumers (and examples) for Kinesis Data Firehose
* AWS: Redshift / Amazon S3 / OpenSearch
* 3rd party partner: Splunk / MongoDB / DataDog / NewRelic / ...
* Custom: send to any HTTP endpoint
51
How do you get charged for using Kinesis Data Firehose
Pay for data going through Firehose
52
What is Kinesis Data Firehose latency?
* Near Real Time
* 60 seconds latency minimum for non full batches
* Or minimum 1MB of data at a time
53
Does Kinesis Data Firehose support many data formats, conversions, transformations, compression
YES
54
Does Kinesis Data Firehose support custom data transformations using AWS Lambda
YES
55
Where can Kinesis Data Firehose send failed or all data?
a backup S3 bucket
56
Kinesis Data Streams vs Firehose
Kinesis Data Streams:
* Streaming ser vice for ingest at scale
* Write custom code (producer / consumer)
* Real-time (~200 ms)
* Manage scaling (shard splitting / merging)
* Data storage for 1 to 365 days
* Supports replay capability
Kinesis Data Firehose:
* Load streaming data into S3 / Redshift / OpenSearch / 3rd party / custom HTTP
* Fully managed
* Near real-time (buffer time min. 60 sec) * Automatic scaling
* No data storage
* Doesn’t support replay capability
57
How is data sent into Kinesis?
using a Partition Key
58
Does the same key always go to the same shard?
YES
59
What is similar to Partition Key in SQS?
Group ID
60
* Consumer “pull data”
* Data is deleted after being consumed
* Can have as many workers (consumers) as we want
* No need to provision throughput
* Ordering guarantees only on FIFO queues
* Individual message delay capability
SQS
61
* Push data to many subscribers
* Up to 12,500,000 subscribers
* Data is not persisted (lost if not delivered)
* Pub/Sub
* Up to 100,000 topics
* No need to provision throughput
* Integrates with SQS for fan- out architecture pattern
* FIFO capability for SQS FIFO
SNS
62
