Which of the following is an example of a poison pill?

A poison pill is inherently malformed data that will fail every retry. The other options represent transient failures or bugs that can be fixed.

Your notification service fails because the Twilio SMS API is down. What type of failure is this?

A down external API is a transient failure. Once Twilio recovers, the message may succeed on retry without code changes.

A consumer function has a bug: `total = price / (1 - discount)`. What happens when discount = 1.0?

Division by zero is a code bug that causes failure. After fixing the code, the messages in the DLQ can be replayed and will process successfully.

In SQS, what does maxReceiveCount = 3 mean?

maxReceiveCount is the number of times a consumer can receive (pick up) a message from the queue. After 3 receives without deletion, it moves to the DLQ.

Which of these is a valid SQS redrive policy configuration approach?

A redrive policy requires both the DLQ ARN and maxReceiveCount. Setting count to 0 or infinity are invalid; practical values are 3-5.

In Kafka, what is the naming convention for a dead-letter topic for an 'orders' topic?

Kafka convention uses suffixes like .DLT or -dlt appended to the original topic name for clarity and consistency.

How does SQS visibility timeout relate to maxReceiveCount?

After visibility timeout expires, a failed message becomes visible again to other consumers, incrementing its receive count. When count reaches max, it goes to DLQ.

What is the first step when you receive a CloudWatch alert that your DLQ has messages?

Investigation comes before any action. You must understand what failed and why before attempting replay or other recovery.

You inspect a DLQ message and see it's a poison pill (invalid JSON). What should you do?

Poison pills cannot be fixed by the consumer. Discard with logging and alert the upstream system that produced malformed data.

Your logs show messages failed because a dependent microservice was down. The service is now healthy. What's the next step?

With the dependency now healthy, replay the messages. They should succeed without code changes since the failure was transient.

Which CloudWatch metric should you alert on for SQS DLQ monitoring?

ApproximateNumberOfMessagesVisible shows how many messages are waiting in the DLQ. High values indicate problems requiring investigation.

For a critical e-commerce payment DLQ, what should your alert threshold be?

Payment failures are critical and must be addressed immediately. Alert when any message arrives in the DLQ.

A Kafka DLT has consumer lag of 50 messages. What does this indicate?

Lag on the DLT means messages arrived but haven't been consumed yet. This indicates an operational issue requiring investigation and manual remediation.

You should alert on ApproximateAgeOfOldestMessage to catch which scenario?

Age of oldest message alerts you when a message has been stuck in DLQ for an extended time, indicating lack of response to an incident.

What is the purpose of a DLQ alert runbook?

A runbook is a documented procedure guide enabling consistent, repeatable incident response by on-call team members.

Which step should NOT be in a DLQ alert runbook?

Bulk replay without investigation is dangerous. You must understand the failure first, especially for payment or critical data systems.

In a notification system, why is a DLQ critical?

Without a DLQ, failed notification messages (e.g., email delivery failures) would disappear and users would never be notified.

Your notification service sends to Twilio (SMS), SendGrid (email), and FCM (push). A single message should trigger which DLQ behavior?

A single notification event should be one message in the queue. If it fails, it goes to one DLQ, then is investigated once and fixed for all channels.

Why is idempotency critical in an arcade payment system?

Idempotency ensures that if a payment message is retried or replayed, the customer is only charged once, preventing catastrophic double-charges.

How does an idempotency store prevent double-charging in payment replay?

Before charging, the system checks the idempotency store. If paymentId exists and is marked completed, it skips the charge but returns success.

Why should payment DLQ messages NEVER be automatically replayed?

Financial transactions are too critical. Each DLQ message must be manually reviewed and approved before replay to prevent regulatory issues and fraud.

What is a FIFO queue useful for in payment processing?

FIFO queues preserve order (critical for ledger consistency) and support deduplication to help prevent double-processing of the same paymentId.

You receive a message with required field 'customerId' as an empty string instead of a UUID. This is:

Empty string for a UUID field is invalid data that will always fail validation. The upstream system sent malformed data (poison pill).

In Kafka, if you use Spring Kafka with DeadLetterPublishingRecoverer, what happens after maxRetries are exceeded?

DeadLetterPublishingRecoverer intercepts failed messages and publishes them to the configured DLT after retries are exhausted.

You fix a consumer bug and redeploy. Now you must:

Deploying a fix doesn't automatically process DLQ messages. You must explicitly replay them using SQS move-task or similar mechanism.

Which combination of metrics should you monitor for comprehensive DLQ alerting?

Together these metrics catch both acute incidents (sudden message spike) and chronic issues (old messages not investigated).

What information should always be recorded when responding to a DLQ alert?

Complete incident logging enables trend analysis, post-mortems, and prevents repeated mistakes. All details matter for operational excellence.

If your email provider (SendGrid) is down when a notification fails, what should happen?

With a DLQ, the system safely captures the message. Once SendGrid is back, replay will succeed with no human intervention needed.

A customer retries a payment button 3 times in quick succession. Without idempotency, what happens?

Without idempotency, 3 separate payment messages go through the system, resulting in 3 charges. Idempotency + FIFO deduplication prevent this.

What's the risk of setting maxReceiveCount too high (e.g., 100)?

High maxReceiveCount wastes compute retrying messages that will always fail. Typical values 3-5 are better for detecting issues quickly.

Dead-Letter Queues — System Design Concept

📚Dead-Letter QueuesPro

10 sections

~28 min total

30 quick quizzes

3 SD challenges linked

0 of 10 done·~27 min left

Concepts›Dead-Letter Queues›What is Dead-Letter Queues?

0 / 10

10 sections~28 min

What is Dead-Letter Queues?

Safety nets for failed messages in distributed systems using queues

Read

~3 min

⋯

Why Messages Fail: Poison Pills, Bugs, Unavailable Dependencies

Messages fail due to malformed data, code bugs, or temporary service unavailability

ReadQuizCode

~3 min

⋯

Configuring SQS Redrive Policy / Kafka Dead-Letter Topics

Set up DLQs using SQS redrive policies or Kafka dead-letter topics with retry limits

ReadQuizCode

~3 min

⋯

The Re-Drive Workflow: Investigate → Fix → Replay

Structured process to diagnose failures, fix root causes, and safely replay messages

ReadQuizCode

~3 min

⋯

Alerting on DLQ Depth

Monitor queue depth metrics and configure CloudWatch alarms for DLQ accumulation

ReadQuizCode

~3 min

⋯

DLQ Alert Runbook

Step-by-step operational procedures for responding to DLQ alerts and resolving failures

ReadQuiz

~3 min

⋯

Relevant System Design Questions

Apply DLQ patterns to real-world notification and payment system designs

Read

~3 min

⋯

Notification System

Design reliable email/SMS/push notifications at scale using DLQs to prevent message loss

ReadQuizCode

~3 min

⋯

Arcade Payment System

Build payment processing with idempotency and DLQs to prevent double-charging and ensure exactly-once semantics

ReadQuiz

~3 min

⋯

Practice test

30 questions

~10 min

🔒

Pro content

Dead-Letter Queues is a Pro concept. Upgrade to access full content, practice questions, and design challenges.

Upgrade to Pro →

Notes

🔍

Loading…