How to buildOctober 11, 202614 min read
Software architectureStep 7 of 9

Event-driven architecture and message queues

In short

Event-driven architecture connects parts of a system with messages: a service records a fact (“order placed”) and publishes an event, and interested services react on their own without blocking the original request. Brokers deliver at least once, so a reliable design rests on two techniques: a transactional outbox, which writes the event to the database in the same transaction as the data, and an idempotent consumer, which recognises a repeat by message id. For external side effects, “exactly once” is only achievable this way: repeats are allowed but do not change the result.

Deep diveLevel: AdvancedEvent-Driven & Message Queues

After this you will be able to

  • tell a command from an event and choose between a RabbitMQ queue and a Kafka log for a given task
  • implement a transactional outbox in PostgreSQL 17 with a relay that waits for broker confirms
  • make a consumer idempotent: the message id and the effect are committed in one transaction, so a redelivery creates no duplicate
  • configure delayed retries, a delivery limit and a dead letter queue for a RabbitMQ 4.3 quorum queue

Verified with

Node.js 24 LTS (24.18)RabbitMQ 4.3.6 (rabbitmq:4.3-management-alpine)PostgreSQL 17 (17.11, postgres:17-alpine)amqplib 2.2.0node-postgres (pg) 8.23TypeScript 7.0

Checked against docs: October 11, 2026

Contents
  1. 01A synchronous call, a command and an event
  2. 02Job queue and event log: RabbitMQ and Kafka
  3. 03Delivery guarantees, and the truth about exactly-once
  4. 04Transactional outbox: the event is neither lost nor ahead of the transaction
  5. 05The idempotent consumer
  6. 06Ordering, retries and dead lettering
  7. 07Sagas, and when events are not needed
  8. 08Exercises

A synchronous call, a command and an event

After an order is placed, the orders service has to award loyalty points, send a notification and pass the order to the warehouse. The simplest option is to call three APIs in a row inside the handler. But then placing an order depends on all three services being up, the response times add up, and if the third call fails the first two have already happened. The event-driven approach reverses the dependency: the orders service saves the order and announces “order placed”, while loyalty, notifications and the warehouse subscribe to that message and process it at their own pace.

Command

What it expresses
A request to perform an action; one addressee, may be refused
Example
ReserveStock, “award points”
Who decides what to do
The sender

Event

What it expresses
A fact that has happened, in the past tense; any number of subscribers, or none
Example
order.placed, “order placed”
Who decides what to do
Each subscriber

Message

What it expresses
The envelope carried by the broker: body, headers, id
Example
JSON with messageId and type
Who decides what to do
—

In “What do you mean by Event-Driven?” Martin Fowler points out a trap: if the sender expects the receiver to act, it is not an event but a command in disguise. A good sign of an event is that the source does not care who reacts or how. The same article separates event notification (the message carries only the fact and an id, and the subscriber asks the source for details) from event-carried state transfer (the event carries all the data the subscriber needs, so it keeps its own copy and does not depend on the source). Event sourcing and CQRS are separate patterns that need not be asynchronous; they are out of scope here.

  1. 01Orders service
  2. 02outbox in PostgreSQL
  3. 03Relay
  4. 04RabbitMQ
  5. 05Loyalty service
The design from the hands-on part: the event goes through the outbox table instead of being sent straight from the handler

Job queue and event log: RabbitMQ and Kafka

Message brokers follow two models. In a queue, a message is removed once acknowledged, and several consumers of one queue share the stream (competing consumers). Fan-out to everyone interested (pub/sub) is built in RabbitMQ with an exchange, a routing point: each subscriber has its own queue bound to the exchange, and each queue gets a copy. In a log (Kafka), an event is appended to the end of a partition and is not deleted after reading: each consumer group keeps its own position (offset) and can rewind it.

RabbitMQ: queues

  • A message lives until a consumer acknowledges it, then it is removed
  • Flexible routing: direct, topic, fanout and headers exchanges
  • Retries, delivery limits and dead lettering are handled by the broker
  • FIFO within a queue, but redelivery can change the order
  • Fits work distribution and service integration

Kafka: event log

  • Events are not deleted after reading; retention is configured per topic
  • The read position is an offset that can be moved back to re-read history
  • Order is guaranteed within a partition; the same key means the same partition
  • Parallelism is bounded by the number of partitions
  • Fits event streams, analytics and rebuilding state by replay

A background job queue inside one application is a separate case: BullMQ on Redis, from the lesson on background jobs, enqueues a job and runs it with a worker from the same project. A broker such as RabbitMQ is needed when messages cross the boundaries of services, teams or languages and a message has several independent receivers. Kafka pays off when events must be kept and re-read: connecting a new service to the history, rebuilding a read model, investigating an incident.

Delivery guarantees, and the truth about exactly-once

The Kafka documentation defines three delivery semantics, and they apply to any broker. Choosing between them means choosing when to acknowledge processing: before the work is done or after.

At-most-once

What it guarantees
A message may be lost but is never repeated
How it is achieved
Acknowledge before processing, automatic acks
The cost
Losses on any consumer failure

At-least-once

What it guarantees
A message is never lost but may arrive again
How it is achieved
Broker confirms on publish and manual acks after processing
The cost
Duplicates: the consumer must detect them

Exactly-once

What it guarantees
Each message is processed once
How it is achieved
Inside Kafka: idempotent producer and transactions; elsewhere only with the receiver’s cooperation
The cost
Complexity and a limited scope

The RabbitMQ reliability guide states it plainly: acknowledgements give at-least-once delivery, and without them messages can be lost. A publisher confirm means the broker has taken responsibility for the message: for a persistent message in a durable queue it has been written to disk; for a quorum queue a majority of replicas have accepted it. A consumer acknowledgement (ack) means the receiver has finished its work. If the connection drops before the ack, the broker requeues the message and delivers it again with the redelivered flag.

The redelivered flag is only a hint. RabbitMQ guarantees that a message without the flag has not been delivered before, but not the reverse: a duplicate can arrive without the flag, for example when a publisher resends after a dropped connection. So protect yourself by message id, not by the flag.

Transactional outbox: the event is neither lost nor ahead of the transaction

The dual write problem: a handler has to save an order to the database and publish an event to the broker. Save first and publish second, and the process can crash in between: the order exists, the event does not. Publish first, and subscribers get an event about an order whose transaction may then roll back. A distributed transaction (2PC) across the database and the broker is usually unavailable or undesirable.

The transactional outbox pattern (microservices.io) solves this by writing the event to an outbox table in the same database and the same transaction as the business data. A separate message relay reads unpublished rows and sends them to the broker. The event appears if and only if the transaction commits. The relay can poll the table (a polling publisher, as below) or read the database transaction log (transaction log tailing, CDC); the latter puts no query load on the database but needs separate infrastructure.

The example below has an orders service and a loyalty service. For clarity their tables live in one database in separate orders and loyalty schemas; in a real system each service has its own database. Ports 55432 and 55672 are non-standard so they do not clash with a local PostgreSQL or RabbitMQ.

bash
docker run -d --rm --name kb-pg-events -e POSTGRES_PASSWORD=test \
  -p 127.0.0.1:55432:5432 postgres:17-alpine
docker run -d --rm --name kb-rabbit-events \
  -p 127.0.0.1:55672:5672 -p 127.0.0.1:55673:15672 rabbitmq:4.3-management-alpine

mkdir outbox-demo && cd outbox-demo
npm init -y && npm pkg set type=module
npm install pg@8.23.1 amqplib@2.2.0
npm install -D typescript@7.0.2 @types/pg @types/node@24
PostgreSQL 17 and RabbitMQ 4.3 in containers (bound to 127.0.0.1, --rm removes them on stop), the project and its dependencies
tsconfig.json
{
  "compilerOptions": {
    "target": "es2024",
    "module": "nodenext",
    "strict": true,
    "noEmit": true,
    "allowImportingTsExtensions": true,
    "erasableSyntaxOnly": true,
    "verbatimModuleSyntax": true,
    "skipLibCheck": true
  },
  "include": ["src"]
}
For type checking only (npx tsc -p .): Node.js 24 runs .ts files directly by stripping the type annotations
sql/schema.sql
-- Orders service: business table + outbox in the same database
CREATE SCHEMA orders;

CREATE TABLE orders.orders (
  id          bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  customer_id bigint  NOT NULL,
  total_kzt   integer NOT NULL CHECK (total_kzt > 0),
  created_at  timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE orders.outbox (
  id             bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  event_id       uuid   NOT NULL UNIQUE DEFAULT gen_random_uuid(),
  aggregate_type text   NOT NULL,
  aggregate_id   text   NOT NULL,
  event_type     text   NOT NULL,
  payload        jsonb  NOT NULL,
  created_at     timestamptz NOT NULL DEFAULT now(),
  published_at   timestamptz
);
-- the relay only looks for unpublished rows
CREATE INDEX outbox_unpublished ON orders.outbox (id) WHERE published_at IS NULL;

-- Loyalty service: its own tables (in production — its own database)
CREATE SCHEMA loyalty;

-- a ledger: every row is a points accrual, nothing stops a duplicate row
CREATE TABLE loyalty.ledger (
  id          bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  customer_id bigint  NOT NULL,
  order_id    bigint  NOT NULL,
  points      integer NOT NULL
);

CREATE TABLE loyalty.processed_messages (
  consumer     text NOT NULL,
  message_id   uuid NOT NULL,
  processed_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer, message_id)
);
Schema: orders and the outbox belong to the orders service; the points ledger and processed messages belong to the loyalty service
bash
docker exec -i kb-pg-events psql -U postgres < sql/schema.sql
Applying the schema
src/db.ts
import { Pool } from 'pg';

export const pool = new Pool({
  connectionString: process.env.DATABASE_URL ?? 'postgres://postgres:test@localhost:55432/postgres',
});
PostgreSQL connection pool; the password test is for the local container only
src/place-order.ts
import { pool } from './db.ts';

export type OrderPlaced = { orderId: number; customerId: number; totalKzt: number };

// The order and its event are written in ONE transaction:
// either both exist or neither does.
export async function placeOrder(customerId: number, totalKzt: number): Promise<number> {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const { rows } = await client.query<{ id: string }>(
      'INSERT INTO orders.orders (customer_id, total_kzt) VALUES ($1, $2) RETURNING id',
      [customerId, totalKzt],
    );
    const orderId = Number(rows[0].id);
    const event: OrderPlaced = { orderId, customerId, totalKzt };
    await client.query(
      `INSERT INTO orders.outbox (aggregate_type, aggregate_id, event_type, payload)
       VALUES ('order', $1, 'order.placed', $2)`,
      [String(orderId), event],
    );
    await client.query('COMMIT');
    return orderId;
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

// node src/place-order.ts <customerId> <totalKzt>
if (import.meta.main) {
  const [customerId, totalKzt] = process.argv.slice(2).map(Number);
  const orderId = await placeOrder(customerId, totalKzt);
  console.log(`[orders] order ${orderId} saved, event written to outbox`);
  await pool.end();
}
The order and its event in one transaction: no broker in the handler and no gap between two writes

The RabbitMQ topology: a topic exchange orders, the loyalty service’s queue loyalty.order-placed and a dead letter queue. A quorum queue is replicated and always durable; its arguments are covered in the section on retries.

src/rabbit.ts
import { connect, type Channel } from 'amqplib';

export const EXCHANGE = 'orders'; // events of the orders service
export const DLX = 'orders.dlx'; // where rejected messages go
export const QUEUE = 'loyalty.order-placed'; // the loyalty service's queue
export const DLQ = 'loyalty.order-placed.dlq';

export function connectRabbit() {
  return connect(process.env.AMQP_URL ?? 'amqp://guest:guest@localhost:55672');
}

// Declarations are idempotent: every process may call this on start
export async function assertTopology(ch: Channel) {
  await ch.assertExchange(EXCHANGE, 'topic', { durable: true });
  await ch.assertExchange(DLX, 'fanout', { durable: true });
  await ch.assertQueue(DLQ, { durable: true, arguments: { 'x-queue-type': 'quorum' } });
  await ch.bindQueue(DLQ, DLX, '');
  await ch.assertQueue(QUEUE, {
    durable: true,
    arguments: {
      'x-queue-type': 'quorum', // replicated durable queue
      'x-dead-letter-exchange': DLX,
      'x-delivery-limit': 5, // more than 5 failed deliveries -> DLQ
      'x-delayed-retry-type': 'failed', // RabbitMQ 4.3+: pause before a retry
      'x-delayed-retry-min': 1000, // 1 s, 2 s, 3 s... (linear back-off)
      'x-delayed-retry-max': 10000,
    },
  });
  await ch.bindQueue(QUEUE, EXCHANGE, 'order.placed');
}
Declaring exchanges and queues is idempotent, so both the relay and the consumer call it
src/relay.ts
import { setTimeout as sleep } from 'node:timers/promises';
import { pool } from './db.ts';
import { assertTopology, connectRabbit, EXCHANGE } from './rabbit.ts';

type OutboxRow = { id: string; event_id: string; event_type: string; payload: unknown; created_at: Date };

const BATCH = 100;
const once = process.argv.includes('--once');

const conn = await connectRabbit();
const ch = await conn.createConfirmChannel(); // the broker confirms every publish
await assertTopology(ch);

async function publishBatch(): Promise<number> {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    // SKIP LOCKED: several relay instances take different rows instead of waiting
    const { rows } = await client.query<OutboxRow>(
      `SELECT id, event_id, event_type, payload, created_at
         FROM orders.outbox
        WHERE published_at IS NULL
        ORDER BY id
        LIMIT $1
        FOR UPDATE SKIP LOCKED`,
      [BATCH],
    );
    if (rows.length === 0) {
      await client.query('COMMIT');
      return 0;
    }
    for (const row of rows) {
      ch.publish(EXCHANGE, row.event_type, Buffer.from(JSON.stringify(row.payload)), {
        messageId: row.event_id, // consumers deduplicate by this id
        type: row.event_type,
        contentType: 'application/json',
        persistent: true,
        timestamp: Math.floor(row.created_at.getTime() / 1000),
      });
    }
    await ch.waitForConfirms(); // rejects if the broker did not take a message
    console.log(`[relay] published ${rows.length}: ${rows.map((r) => r.event_id.slice(0, 8)).join(', ')}`);

    if (process.env.CRASH_AFTER_PUBLISH) {
      console.log('[relay] simulated crash before marking rows as published');
      process.exit(1); // the transaction is rolled back, the rows stay unpublished
    }

    await client.query('UPDATE orders.outbox SET published_at = now() WHERE id = ANY($1::bigint[])', [
      rows.map((r) => r.id),
    ]);
    await client.query('COMMIT');
    return rows.length;
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

do {
  const sent = await publishBatch();
  if (!once && sent === 0) await sleep(1000); // polling interval when idle
} while (!once);

await ch.close();
await conn.close();
await pool.end();
The relay locks a batch of rows, publishes, waits for broker confirms and only then marks the rows
  • FOR UPDATE SKIP LOCKED: the PostgreSQL documentation describes this mode as suitable for queue-like tables with several consumers. Locked rows are skipped, so two relay instances never take the same event at once.
  • createConfirmChannel and waitForConfirms: the promise rejects if the broker did not accept at least one message; the transaction then rolls back and the rows are sent again.
  • messageId is the event_id from the outbox. A resend of the same row carries the same id, which is how the consumer spots the duplicate.
  • The transaction stays open while the relay waits for confirms. This is a deliberate trade-off: the rows are locked, but with a slow broker the transaction is long. The alternative is to “lease” rows with an expiry timestamp and publish outside the transaction.

The idempotent consumer

The idempotent consumer pattern (microservices.io): the consumer records the id of a processed message in a table, in the same transaction as its own changes. The primary key (consumer, message_id) will not accept the id twice, and INSERT … ON CONFLICT DO NOTHING reports 0 inserted rows, which is the “already processed” signal. The key phrase is “in the same transaction”: record the id separately, and a crash between the two writes reopens the door to a duplicate or a loss.

src/consumer.ts
import type { ConsumeMessage } from 'amqplib';
import { pool } from './db.ts';
import type { OrderPlaced } from './place-order.ts';
import { assertTopology, connectRabbit, QUEUE } from './rabbit.ts';

const CONSUMER = 'loyalty.points';

const conn = await connectRabbit();
const ch = await conn.createChannel();
await assertTopology(ch);
await ch.prefetch(10); // at most 10 unacknowledged messages per consumer

function parse(msg: ConsumeMessage): OrderPlaced | null {
  try {
    const e = JSON.parse(msg.content.toString());
    return Number.isInteger(e.orderId) && Number.isInteger(e.customerId) && e.totalKzt > 0 ? e : null;
  } catch {
    return null;
  }
}

async function handle(msg: ConsumeMessage) {
  const id: string | undefined = msg.properties.messageId;
  const tag = `${id?.slice(0, 8)}${msg.fields.redelivered ? ' (redelivered)' : ''}`;
  const event = parse(msg);
  if (!id || !event) {
    console.log(`[loyalty] ${tag} malformed -> DLQ`);
    ch.reject(msg, false); // requeue = false: a retry cannot fix bad data -> DLQ
    return;
  }

  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    // Idempotency: the message id and the effect are committed together
    const seen = await client.query(
      `INSERT INTO loyalty.processed_messages (consumer, message_id)
       VALUES ($1, $2) ON CONFLICT DO NOTHING`,
      [CONSUMER, id],
    );
    if (seen.rowCount === 0) {
      await client.query('ROLLBACK');
      console.log(`[loyalty] ${tag} duplicate, skipped`);
      ch.ack(msg);
      return;
    }
    const points = Math.floor(event.totalKzt / 100); // 1 point per 100 KZT
    await client.query('INSERT INTO loyalty.ledger (customer_id, order_id, points) VALUES ($1, $2, $3)', [
      event.customerId,
      event.orderId,
      points,
    ]);
    await client.query('COMMIT');
    console.log(`[loyalty] ${tag} order ${event.orderId}: +${points} points`);

    if (process.env.CRASH_BEFORE_ACK) {
      console.log('[loyalty] simulated crash after commit, before ack');
      process.exit(1); // the broker will redeliver the unacknowledged message
    }
    ch.ack(msg); // ack only after the effect is committed
  } catch (err) {
    await client.query('ROLLBACK').catch(() => {});
    console.error(`[loyalty] ${new Date().toISOString().slice(11, 19)} ${tag} failed: ${(err as Error).message}`);
    ch.reject(msg, true); // counts as a failed delivery: delayed retry, then DLQ
  } finally {
    client.release();
  }
}

console.log(`[loyalty] waiting for messages in ${QUEUE}`);
await ch.consume(QUEUE, (msg) => {
  if (!msg) return; // null means the broker cancelled the consumer
  handle(msg).catch((err) => {
    // e.g. the database is unreachable: return the message to the queue
    console.error(`[loyalty] ${(err as Error).message}`);
    ch.reject(msg, true);
  });
});
The loyalty consumer: manual acks, deduplication by messageId, ack only after COMMIT

Let’s test two sources of repeats. The first: the relay crashes after publishing but before marking the outbox. Next to two orders we add an event with broken data to see the dead letter path as well. The consumer runs in another terminal.

terminal
$ node src/place-order.ts 101 12000
[orders] order 1 saved, event written to outbox
$ node src/place-order.ts 102 4500
[orders] order 2 saved, event written to outbox
$ docker exec kb-pg-events psql -U postgres -c "INSERT INTO orders.outbox (aggregate_type, aggregate_id, event_type, payload) VALUES ('order', '999', 'order.placed', '{\"orderId\": \"oops\"}')"
INSERT 0 1
$ CRASH_AFTER_PUBLISH=1 node src/relay.ts --once
[relay] published 3: a17a9f69, f883ea62, e1c7b7f3
[relay] simulated crash before marking rows as published
$ node src/relay.ts --once
[relay] published 3: a17a9f69, f883ea62, e1c7b7f3

# node src/consumer.ts (another terminal)
[loyalty] waiting for messages in loyalty.order-placed
[loyalty] e1c7b7f3 malformed -> DLQ
[loyalty] f883ea62 order 2: +45 points
[loyalty] a17a9f69 order 1: +120 points
[loyalty] e1c7b7f3 malformed -> DLQ
[loyalty] a17a9f69 duplicate, skipped
[loyalty] f883ea62 duplicate, skipped
Real output: three events published twice, points awarded once each, repeats skipped by id

The second source: the consumer crashes after COMMIT but before ack. The broker got no acknowledgement, so once the connection dropped it requeued the message and delivered it again, this time with the redelivered flag.

terminal
$ node src/place-order.ts 103 8000
[orders] order 3 saved, event written to outbox
$ node src/relay.ts --once
[relay] published 1: 09d38984

# CRASH_BEFORE_ACK=1 node src/consumer.ts
[loyalty] waiting for messages in loyalty.order-placed
[loyalty] 09d38984 order 3: +80 points
[loyalty] simulated crash after commit, before ack

# node src/consumer.ts (restarted)
[loyalty] waiting for messages in loyalty.order-placed
[loyalty] 09d38984 (redelivered) duplicate, skipped

$ docker exec kb-pg-events psql -U postgres -c "SELECT customer_id, order_id, points FROM loyalty.ledger ORDER BY order_id"
 customer_id | order_id | points 
-------------+----------+--------
         101 |        1 |    120
         102 |        2 |     45
         103 |        3 |     80
(3 rows)
Real output: the redelivery after the consumer crash did not create a second ledger row

Without the processed_messages table, loyalty.ledger would hold six rows instead of three: it has no unique constraint on the order, just like a real accrual ledger where a customer can have several entries. If the effect does have a natural key (one accrual per order), a unique constraint on it is a second line of defence. For external calls (a payment, an email), the message id is passed on as an idempotency key if the external API supports one.

Ordering, retries and dead lettering

Look at the first output again: the event for order 1 was published first, yet the points for order 2 were awarded first. The queue delivered messages in order, but a consumer with prefetch(10) processes them concurrently, and the database queries finish in arbitrary order. The RabbitMQ documentation lists other sources of reordering too: messages from different channels interleave, and redelivery with several consumers changes the order. In Kafka, order is guaranteed only within a partition.

  1. Design without global order

    Most subscribers don’t need it: points for orders 1 and 2 can be awarded in any order.

  2. Order per entity

    Put the entity’s version number in the event; the subscriber stores the last applied version and ignores older ones. In Kafka the message key has the same effect: events of one order land in one partition.

  3. Strict queue order

    In RabbitMQ: a single active consumer with prefetch(1), or a stream. This cuts parallelism, so use it selectively.

Now failures. Retrying cannot fix broken data, so the consumer rejected the event with "orderId": "oops" via reject(msg, false), and the broker moved it to the orders.dlx exchange and on to the dead letter queue (DLQ). A transient error (the database is down, a table is locked by a migration) is worth retrying, but not forever.

The second addition in 4.3 is delayed retry: a returned message is held back before the next delivery. The delay is linear: min(delayed-retry-min × delivery_count, delayed-retry-max). The failed type applies it only to messages whose counter was incremented. To test it, we “break” the ledger table by renaming it, as a failed migration might.

terminal
$ docker exec kb-pg-events psql -U postgres -c "ALTER TABLE loyalty.ledger RENAME TO ledger_broken"
ALTER TABLE
$ node src/place-order.ts 104 3000
[orders] order 4 saved, event written to outbox
$ node src/relay.ts --once
[relay] published 1: 670fd021

# node src/consumer.ts
[loyalty] waiting for messages in loyalty.order-placed
[loyalty] 02:28:57 670fd021 failed: relation "loyalty.ledger" does not exist
[loyalty] 02:28:58 670fd021 (redelivered) failed: relation "loyalty.ledger" does not exist
[loyalty] 02:29:00 670fd021 (redelivered) failed: relation "loyalty.ledger" does not exist
[loyalty] 02:29:03 670fd021 (redelivered) failed: relation "loyalty.ledger" does not exist
[loyalty] 02:29:07 670fd021 (redelivered) failed: relation "loyalty.ledger" does not exist
[loyalty] 02:29:12 670fd021 (redelivered) failed: relation "loyalty.ledger" does not exist

$ docker exec kb-rabbit-events rabbitmqctl -q list_queues name messages
name	messages
loyalty.order-placed	0
loyalty.order-placed.dlq	3
Real output: delays of 1, 2, 3, 4 and 5 seconds; after the sixth failure the message went to the DLQ
bash
curl -s -u guest:guest -H 'content-type: application/json' \
  -X POST http://localhost:55673/api/queues/%2F/loyalty.order-placed.dlq/get \
  -d '{"count":5,"ackmode":"ack_requeue_true","encoding":"auto"}'
Peek at the DLQ through the management plugin’s HTTP API without removing the messages
terminal
e1c7b7f3 {"orderId":"oops"} | reason: rejected | x-delivery-count: 1
e1c7b7f3 {"orderId":"oops"} | reason: rejected | x-delivery-count: 1
670fd021 {"orderId":4,"totalKzt":3000,"customerId":104} | reason: delivery_limit | x-delivery-count: 6
Condensed output: fields from the JSON response — id, body, the reason from the x-death header and the delivery count

The DLQ holds three messages: the broken event twice (the crashed relay published it twice, so both copies share one id) with the reason rejected, and the order 4 event with the reason delivery_limit. Once the cause is fixed, DLQ messages are re-sent to the working queue; the consumer is idempotent, so the replay is safe.

Sagas, and when events are not needed

When a business operation spans several services (place an order, reserve stock, charge the payment), one transaction cannot cover it. A saga (microservices.io) is a sequence of local transactions: each step changes its own service’s data and publishes an event or command for the next one. If a step fails, compensating transactions undo the earlier steps: release the reservation, refund the money. Coordination is either choreography (services react to each other’s events) or orchestration (a central orchestrator tells participants what to do).

  • Developers write the compensations: there is no automatic rollback across services.
  • There is no isolation: other operations see the intermediate state (stock reserved, payment pending). Countermeasures such as “awaiting payment” statuses and semantic locks cover this.
  • Every saga step is itself a message consumer, so it needs an outbox and idempotency too.

Event-driven architecture is a tool for loose coupling and independent scaling, not a mandatory maturity level. It is unnecessary when:

  • Everything runs in one application and one database: a plain transaction is simpler and more reliable than any saga
  • The user needs the result now: “is this item in stock” is a synchronous query, not an event
  • There is exactly one known subscriber: it may be a command, and a background job will do
  • The team is not ready to operate a broker, queue monitoring, DLQs and tracing
  • There are no delivery requirements under failure, and an external API call with retries does the job
The outbox turns loss into repetition; the idempotent consumer turns repetition into nothing. Together they give a reliability that no broker promises on its own.

Exercises

Next, read the lesson on webhooks: the same at-least-once delivery, only over HTTP between companies, with the same idempotent receiver. For which service boundaries deserve events in the first place, see the comparison of a modular monolith and microservices.

How this was verified3 · passedchecked 11 October 2026

“Run” means the code was executed in a scratch project or container and the output in the article comes from that run. “Partial” means it was checked against stubs instead of external services. “Not run” means it needs a device, an account or a key; the steps were checked against the official documentation. The verification project files and run logs are kept in the InterLAB archive.

Tell us about your task — we'll review your request and get back to you

Got a project?
We'll scope and estimate it — for free.

Tell us about the product or the problem. After reviewing the task we'll come back with an architecture, a timeline and a price for the first version.

  • Scope and risks estimated before we start
  • NDA — yours or ours
  • Work starts within a week

Please do not send passwords, access keys or source code through this form — we will agree a safe way to share access separately.

By clicking the button you agree to the processing of your personal data.