Section 12: AWS Integration & Messaging: SQS, SNS & Kinesis Flashcards
1
Q
What are the two patterns of application communication?
A
Synchronous communications (application to application) Asynchronous / Event based communications (application to queue to application)
2
Q
Why can synchronous communications between applications be problematic?
A
If there are sudden spikes of traffic, the destination probably wonโt be able to handle everyhting at once (Ex: An app usually encode 10 videos/hour but now suddenly receives 1000 videos in a short amount time)
3
Q
What are the three integration and messaging fully managed services by AWS?
A
SQS
SNS
Kinesis
4
Q
What is the model of SQS?
A
Queue model
5
Q
What is the model of SNS?
A
Pup/Sub model
6
Q
What is the model of Kinesis
A
Real-time streaming model
7
Q
What are the two types of queue available in SQS?
A
Standard Queue
FIFO Queue
8
Q
What name defines the entities sending messages to an SQS queue?
A
Producers
9
Q
What name defines the entities consuming messages of an SQS queue?
A
Consumers
10
Q
How are messages consumed from an SQS queue?
A
They are polled by the consumers
11
Q
How old is the AWS SQS standard queue?
A
Over 10 years old
12
Q
What do you have to do in order to scale your standard AWS queue?
A
Nothing, it scales automatically to 10,000s messages per second
13
Q
What is the default retention rate of messages in a standard SQS queue?
A
4 days
14
Q
What is the maximum retention configurable for messages in an SQS queue?
A
14 days
15
Q
What is the limit of how many messages can be in a standard SQS queue?
A
No limit
16
Q
What is the latency for SQS queues
A
< 10ms
17
Q
Can the number of consumers of an SQS queue scale? If so, on which axis?
A
Yes, it can scale horizontally
18
Q
Can a standard SQS queue have duplicate messages?
A
It can occasionally
19
Q
Are messages in order in a standard SQS queue?
A
Not necessarily (best effort ordering is built into the service)
20
Q
How can you get messages 100% in order in SQS?
A
Use a FIFO Queue rather than a standard SQS queue
21
Q
What is the maximum size of messages in an SQS queue?
A
256KB
22
Q
How can you add a delay between the moment a message is sent and the moment consumers see the message in an SQS queue?
A
By adding a delivery delay at the queue level
23
Q
How can you override the default delivery delay for a certain message sent in an SQS standard queue?
A
By overriding the default DelaySeconds parameter when sending a message
24
Q
How can you override the default delivery delay for a certain message sent in an SQS FIFO queue?
A
You canโt, DelaySeconds is only available in standard queues (It make sense, other FIFO would not be respected)
25
What consists of a message (sent to an SQS queue) ?
Body (String, up to 256KB)
| Attributes (Metadata)
26
What does the producer gets back when sending a message to an SQS queue?
Message ID
| MD5 hash of the body
27
How many messages at a time can a consumer receive when polling an SQS queue?
Up to 10
28
What is called the time period within which a message is hidden from the consumers because it has been consumed?
The visibility timeout
29
What happens during the visibility timeout?
The message received by the consumer is still in the SQS queue but is considered "in flight" therefore it can't be received by other consumers
30
What is the responsibility of the consumer when he successfully finishes to process a message?
He needs to delete the message from the queue using the message ID and the receipt handle
31
What is the default time period of the visibility timeout in an SQS queue?
30 seconds
32
What is the maximum time period of the visibility timeout in an SQS queue?
12 hours
33
What can happen if the time period of the visibility timout of an SQS queue is too high?
If the consumer fails to process the message, there will be a long delay before trying to process the message again
34
What can happen if the time period of the visibility timout of an SQS queue is too low?
If the consumer needs time to process the message, another consumer will receive the message and the message will be processed more than once
35
What can you do if you set the visibility timeout of your SQS queue too low and a consumer needs more time to process a message?
Use the ChangeMessageVisibility API to increase the length of the visibility timeout of the message being processed at the moment
36
What is the API that a consumer of an SQS queue needs to call when successfully finishing to process a message?
The DeleteMessage API
37
(SQS queue) Where should messages that fail to get processed multiple times in a row be transferred to?
To a DLQ
38
What does DLQ stand for?
Dead Letter Queue
39
What is the redrive policy?
The redrive policy specifies the source queue, the dead-letter queue, and the conditions under which Amazon SQS moves messages from the former to the latter if the consumer of the source queue fails to process a message a specified number of times.
40
What strategy for our consumers allows us to save costs when using SQS queues
Long polling
41
What is the long polling?
A consumer requests message from queue and "wait" for message if there are none at the moment
42
What is the maximum wait time when doing long polling to as SQS queue?
20 seconds
43
What is the preferred wait time when doing long polling to as SQS queue?
20 seconds
44
At what level can long polling be enable?
At the queue level or at the API level
45
What is the name of the parameter which allows us to set the time for long polling?
WaitTimeSeconds
46
What happens to messages which don't get deleted within the visibility timeout period?
They become visible again in the SQS queue, up to the defined treshold (redrive policy), when they will therefore be transferred to the DLR
47
What is the particular naming rule for FIFO queues in SQS
They must end with .fifo
48
What is the maximum number of messages per second (with / without batching) for FIFO queues?
3000 messages/sec with batching
| 300 messages/sec without batching
49
Can there be duplicates in a FIFO queue?
No
50
Can there be "per message delay" in SQS FIFO queues?
No, only per queue delay
51
What are the two features exclusive to FIFO queues?
Deduplication
| Sequencing
52
How can you get deduplication in FIFO queues?
By providing a MessageDeduplicationId with your message
53
Do you have to provide your own MessageDeduplicationId when using deduplication in a FIFO queue or is there a better way to do it?
You can use content based deduplication (the MessageDeduplicationId is generated as the SHA-256 of the message body (not the attributes))
54
What is the deduplication interval in FIFO queues?
A 5-minute period where the queue will track for duplicate messages
55
How to get sequencing in FIFO queue?
By specifying the same MessageGroupId to messages which you absolutely want to get processed in order
56
Are messages with different MessageGroupId sure to get processed in order in a FIFO queue?
No, different consumers can poll and receive messages with different MessageGroupId. Only messages with the same MessageGroupId are sure to get processed one after the other
57
Let's say you have user which executes some actions like "Adds X to cart", "Purchase X", "Cancels X", what could be the MessageGroupId for the messages (that will get added to a FIFO queue) related to that user?
user_id
58
What can you use if you need to send messages that are larger than 256KB in a SQS queue?
Use the SQS Extended Client for Java or a custom solution for other environments
59
What does the SQS Extended Client do?
Producer sends large message to S3
Producer sends small metadata message to SQS queue
Consumer polls/receive the small message
Consumer retrieves the large message from S3
60
What encryption do you get with SQS?
In flight using the HTTPS endpoint
| SSE can be enabled using KMS
61
What does SSE stand for?
Server Side Encryption
62
What is encrypted when using SSE in SQS?
Only the messages body, not the metadata
63
What might be wrong if we can't make our applications work with SQS
We probably have a problem with our IAM policies attached to the Roles of our applications
64
How to get finer grained control over IP when working with SQS?
Using SQS queue access policy
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-overview-of-managing-access.html
65
Which of these SQS API don't have a batch API
SendMessage
ReceiveMessage
DeleteMessage
ChangeMessageVisibility
ReceiveMessage
Because you call already receive up to 10 messages at a time
66
What does PurgeQueue does (SQS)?
Delete all the messages in a queue
67
What are the three most common use cases for SQS?
Decouple applications
(for example to handle payments asynchronously)
Buffer writes to a database
(for example a voting application)
Handle large loads of messages coming in
(for example an email sender)
68
Can SQS be integrated with Auto Scaling? If so, how?
Yes, through CloudWatch
69
What can you use if you want to send a message to multiple receivers in a single call?
SNS
70
Where does an event producer sends messages to with SNS?
To one SNS topic
71
What is the maximum amount of receivers per topic in SNS?
10,000,000
72
What is the maximum number of topics in an AWS account?
100,000
73
What can SNS subscribers be?
```
SQS
HTTP/HTTPS
Lambda
Emails
SMS messages
Mobile Notifications
```
74
```
How does these services use SNS?
CloudWatch
ASG
Amazon S3
CloudFormation
```
CloudWatch: Alarms
ASG: Notifications from alarms to trigger auto scalling
Amazon S3: On bucket events
CloudFormation: State changes, failed to build, etc.
75
What are the steps to publish in a topic in SNS?
Create a topic
Create a subscription
Publish to the topic
76
What are the steps to direct publish in SNS (mobile apps sdk)?
Create a platform application
Create a platform endpoint
Publish to the platform endpoint
77
What is the SNS + SQS: Fan out strategy?
Push once in SNS, receive in many SQS
78
What is AWS Kinesis?
Kinesis is a managed alternative to Apache Kafka
79
What is Kinesis great for?
Application logs, metrics, IoT, clickstreams
"Real-time" big data
Streaming processing frameworks (Spark, NiFi, etc.)
80
Is there data replication with Kinesis?
Yes, data is automatically replicated to 3 AZ.
81
What are the three compoenents of Kinesis?
Kinesis Streams
Kinesis Analytics
Kinesis Firehose
82
What is Kinesis stream?
Low latency streaming ingest at scale
83
What does Kinesis Analytics do?
Performs real-time analytics
84
What language does Kinesis Analytics leverages?
SQL
85
What does Kinesis Firehose do?
Loads streams into S3, Redshift, ElasticSearch, etc.
86
Of what are composed Kinesis Streams?
Shards/Partitions
87
What is the default data retention period in Kinesis Streams?
1 day
88
What is the max data retention period in Kinesis Streams?
7 days
89
Do you have the ability to reprocess / replay data in Streams?
Yes
90
Can many applications consume the same stream?
Yes
91
Once data is inserted in Kinesis, can it be deletted?
No
92
What is the write capacity of a shard?
1 MB/s or 1000 messages/sec
93
What is the read capacity of a shard?
2 MB/s
94
How are Kinesis Streams billed?
They are billed per hour per shard provisioned
95
How many shards can a stream have?
As many as you want
96
Can the number of shards of a stream evolve over time?
Yes (reshard/merge)
97
When records enters a stream, do they stay in order?
Records in the same shard are in order
98
How can you make sure that records that need to stay in order get in the same shard/partition?
By providing the same PartitionKey
99
How can you avoid the "hot partition" ?
By providing highly distributed Partition Keys
100
What do messages received by Kinesis stream get?
A sequence number
101
What API can you use to send messages to Kinesis
PutRecord (without batching)
| PutRecords (with batching)
102
Can you send messages to a Kinesis Stream from the Console?
No, you have to use the CLI, SKSs, or producer libraries from various frameworks
103
What error will we get if we go over the limit from our Kinesis Stream? (exceeding MB/s or TPS for any shard)
ProvisionedThroughputExceeded
104
What might be the cause of a ProvisionedThroughputExceeded error?
A "hot partition/shard"
105
How can you solve ProvisionedThroughputExceeded ?
Retries with backoff
Increase shards (scaling)
Ensure your partion keys are distributed enough not to get a "hot partition"
106
What can you use to consume a Kinesis Stream?
CLI
SDK
or the Kinesis Client Library
107
What does KCL stand for?
Kinesis Client Library
108
What does KCL uses to checkpoint offsets?
DynamoDB
109
What does KCL uses to track other workers and share the work amongst shards?
DynamoDB
110
How many shards can a KCL read?
Many
111
You can't have more ___ than ___
Words to place:
KCL
shards
You can't have more KCL than shards
112
What can KCL run on?
EC2, EB, even on premise applications
113
How is Kinesis secure?
IAM policies (control access / authorization)
Encryption in flight (HTTPS)
Encryption at rest (KMS)
114
Can you encrypt/decrypt client side when using Kinesis?
Yes but it's harder
115
Are VPC endpoints available for Kinesis?
Yes!
116
What does AWS Kinesis Data Analytics do?
Perform real-time analytics on Kinesis Streams using SQL
117
What do you have to do to make Kinesis Data Analytics scale?
Nothing, it's a fully managed, you pay for what you use
118
What can you load data into when using AWS Kinesis Firehose?
Redshift, S3, ElasticSearch, Splunk
119
What do you have to do to make AWS Kinesis Firehose scale?
Nothing, it's a fully managed service, you pay for the amount of data going through Firehose
120
What is AWS Kinesis meant for?
For real-time big data, analytics and ETL
121
What does ETL stand for?
"Extract, Transform, Load"
