Message Queues
Async processing with Kafka, RabbitMQ, and SQS
Message Queues
Message queues enable asynchronous communication between services. Producers send messages to a queue, and consumers process them independently. This decouples services, improves reliability, and enables load leveling.
Message Queue Architecture
When to Use Message Queues
- Async Processing: Email sending, image processing, report generation
- Load Leveling: Buffer traffic spikes, process at constant rate
- Decoupling: Services communicate without knowing each other
- Fan-out: Same message processed by multiple consumers
- Retry Logic: Failed messages re-queued automatically
- Event Sourcing: Store events as immutable sequence
Comparison
text
āāāāāāāāāāāāāāāāāā¬āāāāāāāāāāāāāāāāāāā¬āāāāāāāāāāāāāāāāāāā¬āāāāāāāāāāāāāāāāāāā
ā Feature ā Kafka ā RabbitMQ ā SQS ā
āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāā¤
ā Model ā Log/Stream ā Queue/Exchange ā Queue ā
ā Throughput ā Millions/sec ā Thousands/sec ā Thousands/sec ā
ā Ordering ā Per-partition ā Per-queue ā Best-effort ā
ā Retention ā Configurable ā Until consumed ā 14 days max ā
ā Use Case ā Event streaming ā Task queues ā Serverless async ā
ā Replay ā Yes ā No ā No ā
ā Protocol ā TCP binary ā AMQP ā HTTP/HTTPS ā
āāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāā