Idempotent Consumer Patterns for Event Streams

Part of: Idempotency Fundamentals & API Guarantees

Every mainstream message broker — Kafka, RabbitMQ, SQS, SNS, Google Pub/Sub — ships with at-least-once delivery as its default, and most cannot be configured to guarantee anything stronger end-to-end. A consumer that acknowledges a message after processing it but before the acknowledgment reaches the broker will see that same message again after a crash, a rebalance, or a visibility-timeout expiry. This is not a bug in the broker; it is the fundamental trade-off behind exactly-once vs. at-least-once delivery — a broker cannot know whether a missing acknowledgment means “the consumer never got the message” or “the consumer processed it and died before confirming,” so it must redeliver. The idempotent consumer (or idempotent receiver) pattern pushes the burden of exactly-once effects onto the consumer, which is the only place in the pipeline with enough context to define what “the same message” means for its own business logic.

This page covers the pattern from first principles: how to derive a deduplication key when the producer gives you nothing to work with, the inbox pattern for coupling that key to a durable check, how deduplication interacts with (and is distinct from) message ordering, and the four implementation variants you’ll actually deploy in production.

---

Problem Framing

A naive consumer treats every delivery as a new event:

on_message(msg):
    apply_to_ledger(msg.amount)
    ack(msg)

If the broker redelivers msg — because the consumer crashed after apply_to_ledger but before ack, or because a Kafka consumer group rebalance reassigned the partition mid-processing — apply_to_ledger runs twice. In a payments ledger this is a double-post; in an inventory system it’s a double-decrement; in a notification service it’s a duplicate email. The failure is not exotic: rebalances happen on every deploy, every scale-out event, and every consumer restart, and each one creates a window where in-flight messages may be redelivered to a different consumer instance than the one that was already processing them.

The idempotent consumer pattern closes this gap by making the “has this logical event already been applied?” check a first-class, durable part of the processing path — not an afterthought bolted onto retry handling.

---

Guarantee Model

A correctly implemented idempotent consumer provides effectively-once processing on top of at-least-once delivery: the broker may deliver a message any number of times, but the business effect is applied exactly once, and every redelivery after the first receives the same terminal outcome (acknowledge and skip) rather than a partial or duplicated effect.

This guarantee holds only under these conditions:

• The dedup key is stable across every redelivery of the same logical event. If the key changes between deliveries (a common failure with client-generated UUIDs regenerated on each publish retry), the consumer cannot recognize the duplicate and the guarantee silently degrades to none.
• The dedup check and the business write happen in the same atomic unit. A check-then-write race (read inbox, find it absent, then write) between two concurrently processing redeliveries reintroduces the exact race the pattern exists to close.
• The dedup window covers the maximum possible redelivery delay. A key that expires before a very late redelivery (e.g. after an hours-long consumer outage) lets the duplicate back in as if it were new.

Deduplication guarantees exactly-once effects for a single logical event. It does not guarantee ordering between distinct events — that is a separate concern, addressed later in this page.

---

Core Pattern: The Idempotent Receiver with an Inbox

The canonical implementation pairs two ideas: an idempotent receiver — a handler that checks a durable store before executing business logic — and an inbox table, which is the receiver’s own transactionally-consistent record of every message id it has already committed. The inbox lives in the same datastore and the same transaction as the business write, so the “have I seen this?” check and the “apply the effect” write either both commit or both roll back — there is no window in which one succeeds without the other.

Deriving the Dedup Key When the Producer Sends None

Not every producer supplies a usable identifier. Three fallbacks cover nearly every real system:

1. Broker-native message id. Kafka exposes (topic, partition, offset) as a naturally unique triple for that topic’s log; SQS assigns a MessageId; RabbitMQ has no built-in equivalent unless you enable publisher confirms with a client-supplied message_id header. Prefer this when available — it costs nothing to compute.
2. Business key. Compose a key from fields that are semantically stable across retries: order_id + event_type + version for a domain event, or account_id + statement_period for a batch job trigger. This is the right choice when the same logical event might be re-published by the producer itself with a different transport-level id (a common failure mode when producers retry a failed publish and regenerate a UUID each attempt).
3. Content hash. When neither of the above is reliable, compute SHA-256 over the canonicalized payload — sorted keys, fixed number formatting, volatile fields (timestamps, trace ids, request ids) stripped before hashing. Two deliveries of the same logical event, even with different transport metadata, hash to the same key.

Never derive the dedup key from a field the producer regenerates per attempt (a fresh UUID minted at publish time is not a message identity — it’s proof the producer thinks it might be sending something new).

Ordering vs. Deduplication

These are separate axes and conflating them causes real bugs:

• Ordering guarantees that events affecting the same entity are applied in the order they were produced. It is enforced by routing all events for a given key to the same partition (Kafka) or the same FIFO message group (SQS FIFO), and processing that partition/group single-threaded.
• Deduplication guarantees that a given event, however many times it is delivered, is applied at most once.

A consumer with perfect deduplication can still misorder two distinct events if it processes partitions concurrently without a partition-affinity guarantee. Conversely, a perfectly ordered consumer can still double-apply a single redelivered event if it has no dedup check. Where both matter — most financial and inventory workloads — attach a monotonically increasing per-key sequence number to each event and enforce two independent checks: reject (or buffer) any event whose sequence number is not the next expected value for that key, and reject any event whose id/hash has already been recorded in the inbox. The first check protects ordering; the second protects idempotency; neither substitutes for the other.

---

Implementation Variants

Variant 1: Message-ID Dedup

The cheapest option when the broker or producer supplies a stable identifier. Store the id in a unique-indexed table and rely on the database’s own constraint to reject the second insert atomically — no separate read-then-write step required.

// Java: Kafka consumer with unique-constraint dedup on (topic, partition, offset)
public void onMessage(ConsumerRecord<String, String> record, Connection conn) throws SQLException {
    String dedupKey = record.topic() + "-" + record.partition() + "-" + record.offset();
    conn.setAutoCommit(false);
    try (PreparedStatement insertInbox = conn.prepareStatement(
            "INSERT INTO inbox (dedup_key, received_at) VALUES (?, now()) " +
            "ON CONFLICT (dedup_key) DO NOTHING")) {
        insertInbox.setString(1, dedupKey);
        int rows = insertInbox.executeUpdate();
        if (rows == 0) {
            conn.rollback();
            return; // already processed — ack and move on
        }
        applyBusinessLogic(record.value(), conn);
        conn.commit();
    } catch (Exception e) {
        conn.rollback();
        throw e;
    }
}

Variant 2: Content-Hash Dedup

Used when the transport-level id is unreliable — for example, an SNS producer that retries a failed HTTP publish and mints a new MessageId on each attempt, even though the payload is identical.

import hashlib
import json

def canonical_hash(payload: dict) -> str:
    # Strip volatile fields before hashing
    stable = {k: v for k, v in payload.items() if k not in ("published_at", "trace_id")}
    canonical = json.dumps(stable, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

def handle_message(payload: dict, db_conn) -> None:
    dedup_key = canonical_hash(payload)
    cur = db_conn.cursor()
    cur.execute(
        "INSERT INTO inbox (dedup_key, received_at) VALUES (%s, now()) "
        "ON CONFLICT (dedup_key) DO NOTHING",
        (dedup_key,),
    )
    if cur.rowcount == 0:
        db_conn.commit()
        return  # duplicate — no-op
    apply_business_logic(payload, db_conn)
    db_conn.commit()

Variant 3: Inbox Table Pattern

The general-purpose variant: a dedicated table, always written in the same transaction as the business effect, with an index that supports both the dedup lookup and periodic cleanup.

CREATE TABLE inbox (
    dedup_key    TEXT PRIMARY KEY,
    consumer_id  TEXT        NOT NULL,
    received_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_inbox_received_at ON inbox (received_at);

-- Atomic claim: fails silently (0 rows) if dedup_key already exists
INSERT INTO inbox (dedup_key, consumer_id)
VALUES ($1, $2)
ON CONFLICT (dedup_key) DO NOTHING;

// Go: business write and inbox insert in one transaction
func HandleEvent(ctx context.Context, db *sql.DB, dedupKey, consumerID string, payload []byte) error {
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return err
    }
    defer tx.Rollback()

    res, err := tx.ExecContext(ctx,
        `INSERT INTO inbox (dedup_key, consumer_id) VALUES ($1, $2)
         ON CONFLICT (dedup_key) DO NOTHING`, dedupKey, consumerID)
    if err != nil {
        return err
    }
    rows, _ := res.RowsAffected()
    if rows == 0 {
        return tx.Commit() // duplicate: commit the no-op, ack normally
    }
    if err := applyBusinessLogic(ctx, tx, payload); err != nil {
        return err
    }
    return tx.Commit()
}

This pairs naturally with a transactional outbox on the downstream side: the inbox row proves the inbound event was consumed exactly once, and the outbox row in the same transaction proves any resulting downstream event was published exactly once — closing both ends of the exactly-once chain within a single ACID boundary.

Variant 4: Broker-Native Dedup

Some brokers offer a narrower, transport-scoped dedup guarantee that reduces (but does not eliminate) the need for an application-level inbox:

# Kafka: idempotent producer configuration (prevents duplicate writes
# from the producer's own retries; does not dedup across restarts
# or across independent producer instances)
enable.idempotence: true
acks: all
max.in.flight.requests.per.connection: 5
retries: 2147483647

SQS FIFO queues offer MessageDeduplicationId with either an explicit id or content-based dedup, but only within a 5-minute deduplication window — anything redelivered after that window is treated as new. Broker-native dedup is a useful first line of defense against producer-side retry duplicates, but it never covers consumer-side failures (crash after processing, before ack) or duplicates that arrive after the broker’s window closes, so an application-level inbox is still required wherever business correctness matters. The full SQS/SNS runbook, including the content-based dedup hash algorithm and standard-queue fallback, is covered in Deduplicating SQS and SNS Message Deliveries.

Variant Comparison

---

Edge Cases and Failure Scenarios

---

Operational Concerns

Dedup Window Sizing

Size the inbox retention window — the period a dedup_key must remain queryable before it can be purged — as:

dedup_window = max_redelivery_delay + P99_processing_latency + safety_margin

For a Kafka consumer group with a session.timeout.ms of 10000 (10 seconds), a worst-case consumer outage of 30 minutes before autoscaling replaces the pod, a P99 processing latency of 500 ms, and a 5-minute safety margin: dedup_window = 30 min + 0.5 min + 5 min ≈ 36 minutes. Round up to a clean operational figure — 48 hours is a common default for systems where a stalled deployment pipeline might delay redelivery far longer than the steady-state case. For SQS standard queues (no broker-native dedup at all), align the application inbox window to at least the queue’s configured maxReceiveCount × visibility timeout.

Storage

Each inbox row is typically 40–80 bytes (a UUID or hash key plus a timestamp and small metadata). At 1 million events/day retained for 48 hours, that’s roughly 2 million rows × 80 bytes ≈ 160 MB — trivial for PostgreSQL, negligible for DynamoDB. Two indexing decisions matter:

• The primary key (dedup_key) must be indexed for O(1) lookups on the hot path — this is automatic if it’s declared PRIMARY KEY or UNIQUE.
• A secondary index on received_at supports the cleanup job without a full table scan:

CREATE INDEX CONCURRENTLY idx_inbox_received_at ON inbox (received_at);

-- Nightly cleanup job
DELETE FROM inbox WHERE received_at < now() - INTERVAL '48 hours';

For very high-throughput streams, partition the inbox table by day or hour so the cleanup job drops whole partitions instead of running row-level deletes.

SRE Alert Thresholds

• inbox_duplicate_hit_rate — baseline varies by producer; alert if it exceeds 5× baseline over 5 minutes (signals a redelivery storm, often from a rebalance loop or a stuck consumer group).
• inbox_store_error_rate — alert if > 1% of dedup checks fail over 60 seconds.
• late_duplicate_detected_total — any non-zero rate means the retention window is undersized relative to actual redelivery latency; investigate before widening blindly.
• consumer_lag_max correlated with rebalance events — a rising lag right after a rebalance, combined with a spike in duplicate hits, indicates the new partition owner is re-processing a large in-flight backlog.

---

Related

• Idempotency Fundamentals & API Guarantees — parent reference covering the full guarantee model, failure boundary map, and production readiness checklist this pattern builds on.
• Deduplicating SQS and SNS Message Deliveries — queue-specific runbook covering FIFO MessageDeduplicationId, standard-queue application dedup, and SNS fan-out.
• Exactly-Once vs. At-Least-Once Delivery — the delivery-semantics foundation that explains why brokers redeliver and what “exactly-once” actually means in practice.
• Using Redis SET NX for Distributed Request Deduplication — the equivalent atomic-claim primitive for HTTP-triggered deduplication, applicable when the inbox lives in Redis instead of a relational store.
• PostgreSQL Unique Constraints vs. Application-Level Checks — why ON CONFLICT DO NOTHING beats a read-then-write dedup check under concurrent redelivery.
• Implementing the Transactional Outbox Pattern — the matching pattern for the publish side, closing the exactly-once chain end to end.
• Message Queue Deduplication Patterns — broker-agnostic comparison of dedup mechanisms across RabbitMQ, Kafka, and cloud-managed queues.