Integration and Messaging

Question 1

Describe the Synchronous and Asyncronous application communication patterns

Answer 1

Synchronous - Application to Application
Asynchronous - Application to queue to Application (Event based)

Question 2

Which model does each of the following services use:
1. SQS
2. SNS
3. Kinesis

Answer 2

SQS - Queue
SNS - Pub/sub
Kinesis - Real-time streaming

Question 3

What shoulkd be the first service that comes to your mind when you think of decoupling applications?

Answer 3

SQS

Question 4

What are two properties of the delivery of messages within SQS to consider?

Answer 4

Can have duplicate and out of order messages

Question 5

Suppose you have configured an SQS queue and have EC2 instances downstream consuming messages. How can you ensure you have enough EC2 instances to handle the throughput of mesasges?

Answer 5

Use an ASG with scaling based on CloudWatch Metric - Queue Length

Question 6

If you have an EC2 instance in one account that needs to access an SQS queue in the other, what must you create?

Answer 6

A cross account policy attached to the SQS queue

Question 7

If you want S3 to be able to publish event notifications to SQS queue, what must you do?

Answer 7

Create a cross account policy and attach it to theSQS queue

Question 8

What is Message Visibility Timeout and what is its default value?

Answer 8

The duration (by default 30 seconds) of which a message becomes invisible to other consumers once it is polled by a consumer

Question 9

What is a Dead Letter Queue and how can messages be sent there?

Answer 9

It is where messages that have exceeded the MaximumRecieves threashhold are sent

Question 10

What is the relation between the type of the DLQ and SQS queue?

Answer 10

They must be the same

Question 11

How are you able to trigger a DLQ message to be returned to the main queue?

Answer 11

By “redriving to source”

Question 12

What is Long Polling in relation to SQS?

Answer 12

Allow the consumer to “wait” until a message is recieved into the queue after polling. This reduces the number of API calls and decreases latency.

Question 13

Given the limitation of 256kb of data in a SQS message, how would you send large messages?

Answer 13

Use the SQS extended client (library) which uses an S3 bucket alongside metadata messages in the queue to store large data

Question 14

What do the following API related methods and paramters do in SQS
1. CreateQueue(MessageRetentionPeriod), DeleteQueue
2. PurgeQueue
3. SendMessage(DelaySeconds), RecieveMessage,DeleteMessage
4. MaxNumberOfMessages
5. RecieveMessageWaitTimeSeconds
6. ChangeMessageVisibility

Answer 14

1. Create or delete a queue
2. Delete all message in a queue
3. Send (with a delay), recieve, or delete a message
4. Limit how many messages will be polled at once by consumer
5. Long polling
6. Change message timeout

Question 15

Suppose you configure your SQS queue to FIFO, what is the impact on throughput and duplicate message?

Answer 15

Reduced throughput however messages will be in order using MessageGroup ID and deduplicated using DeduplicationID

Question 16

What are the two methods to determine if a message is a duplicate in SQS FIFO?

Answer 16

1. Content based - Uses SHA256 hash of message body
2. Deduplication ID - Manually assign an ID

Question 17

How can you achieve parallel processing of an ordered subset of messages?

Answer 17

Specify common Message Group ID and define a different consumer per ID

Question 18

What service should you consider if you want to send a single message to multiple consumers? What pattern does this service use?

Answer 18

Consider SNS which uses a Pub/Sub pattern

Question 19

Describe how a message in SNS works?

Answer 19

1. A message is published to an SNS topic
2. Consumers that are “subscribed” to that SNS topic get the message

Question 20

How can you achieve the Fan Out pattern using SNS and SQS? What is one use case?

Answer 20

1. Push a message from a producer to an SNS topic
2. Message is recieved by the SQS queues that are subscribed to the topic

Usecase: For a single event type and prefix in s3 you can only have one S3 event rule. But what if you want to send the same s3 event to multiple sqs queues? Use fanout

Question 21

How would you achieve fanout with ordering and deduplication for SNS?

Answer 21

Create a SNS FIFO topic and have multiple subscrbed SQS FIFO queues

Question 22

What is an easy way to filter through messages in SNS?

Answer 22

Create a filter policy defined in a JSON document

Question 23

If you hear real time data, what AWS service should you think of?

Answer 23

Kinesis Data Streams

Question 24

If you hear near-real time data, what AWS service should you think of?

Answer 24

AWS Data Firehose

Question 25

How can you guarentee ordering in Kinesis Data Stream?

Answer 25

Send data with the same Partition ID

Question 26

Describe the following Capacity Modes for Kinesis Data Streams 1. Provisioned Mode 2. On-demand Mode

Answer 26

1. Provisioned Mode - Pick how many shards you want and scale manually - Pay per shard 2. On-demand Mode - Scales automatically - Pay per stream and data in/out

Question 27

Why is Amazon Data Firehose considered "Near Real-Time"

Answer 27

It has a buffering capability based on size / time in which all the messages are sent at once to the destinations

Question 28

Compare Kinesis Data Streams to Amazon Data Firehose in terms of real time vs near real-time

Answer 28

Kinesis Data Streams - Real time Amazon Data Firehose - Near real-time

Question 29

Compare Kinesis Data Streams to Amazon Data Firehose in terms of data storage and replay capabiltiy

Answer 29

Kinesis Data Streams - Data storage up to 365 days and replayability Amazon Data Firehose - No data storage so no message replay

Question 30

Where can flink NOT read from?

Answer 30

Amazon Data Firehose

Question 31

Review this important SQS information :)

Answer 31

- Consumers "pull" data - Data is deleted once consumer - Can have many workers (consumers) - No need to provision throughput - Ordering guarentees only FIFO queue - Messages can be delayed on purpose

Question 32

Review this important SNS information :)

Answer 32

- "Push" data to many subscribers - Data is not persisted - Pub/sub - No need to provision throughput - FIFO capabiltiy for SQS FIFO - Integrats with SQS for Fanout

Question 33

Review this important Kinesis information :)

Answer 33

- Standard pattern "pulls" data - Enhanved-fan out "pushes" data - Can replay data - Meant for real time big data - Provisioned or on-demand capacity modes

Question 34

Can Kinesis Data Firehose and Kinesis Data Streams be a subscriber to an SNS topic?

Answer 34

No! Only Data Firehose can be not Data Streams

Question 35

Due to the spike in traffic, SQS consumers may fail multiple times while processing messages within the Visibility Timeou. What should you do to handle failed messages?

Answer 35

Add a Dead Letter Queue with a redrive policy

Question 36

If you have an SQS FIFO queue with10 message groups, whats the maximum number of consumer that can consumer simultaniously?

Answer 36

10

Question 37

You have a hot shard in a Kinesis Data Stream, thus causing many ProvisionedThroughputExceededException exceptions. What operation allows you to increase the number of shards?

Answer 37

Shard Splitting