DynamoDB Conditional Writes for Deduplication

Part of: Backend Implementation & Storage Patterns

DynamoDB has no built-in lock server, no SELECT ... FOR UPDATE, and no cross-request transaction that spans an HTTP round trip. What it does have is a single-item write path that is linearizable when guarded by a ConditionExpression. That primitive — a PutItem or UpdateItem that fails atomically when a condition is not met — is sufficient to build exactly-once request deduplication without operating a separate coordination service such as ZooKeeper or a Redis-based distributed lock. This page covers the guarantee model, the PENDING→COMPLETED item state machine, three implementation variants with production code, the edge cases that trip up teams new to conditional writes, and the operational budgeting DynamoDB idempotency tables require at scale.

The core mechanism is deceptively small: attribute_not_exists(pk) on a PutItem call means “insert this item only if no item with this key already exists.” Because DynamoDB serializes all writes to a given partition key through a single leader replica, exactly one concurrent caller receives a success response; every other caller receives a ConditionalCheckFailedException. That single boolean outcome — success or condition failure — is the entire mechanism a deduplication layer needs.

Guarantee Model

DynamoDB conditional writes provide linearizable single-item consistency, not cross-item consistency. The precise contract:

  • For a single partition key (and sort key, if the table has one), at most one PutItem/UpdateItem carrying a given ConditionExpression succeeds when the condition is false for all other concurrent callers.
  • Reads performed with ConsistentRead: true observe the effects of any write that has already returned success — DynamoDB does not expose read-your-writes staleness on strongly consistent reads.
  • The guarantee does not extend across items. Two related items — an idempotency-key item and a business-state item — can be written independently with each individually consistent, while the pair together is not atomic, unless both are enclosed in a single TransactWriteItems call.

Where this breaks down in practice:

  1. Multi-item workflows without a transaction. A service that writes the idempotency-key item first and the business-state item second, as two separate PutItem calls, can crash between the two writes. The key now claims a completed request that never actually mutated business state.
  2. Cross-table or cross-region assumptions. Global Tables replicate asynchronously; a conditional write that succeeds in us-east-1 is not visible with the same linearizability guarantee to a reader hitting eu-west-1 until replication catches up. Conditional writes are only linearizable within the region (and table) they were issued against.
  3. TTL-based cleanup racing with a new claim. The background TTL deletion process and a new PutItem for the same key can interleave in ways that require careful ConditionExpression design — covered in Edge Cases below.

The corrective pattern for all three is the same: model the operations that must be atomic as a single TransactWriteItems call, treat TTL as a cost-control mechanism rather than a correctness mechanism, and pin idempotency-critical reads to the region where the write was issued.

Core Algorithm: The PENDING → COMPLETED State Machine

Every DynamoDB-backed idempotency key moves through a small, explicit state machine. The item is written once at claim time in a PENDING state, and transitioned to COMPLETED (or FAILED) once business logic resolves. The diagram below shows two concurrent PutItem calls racing for the same partition key — only one wins the ConditionExpression.

DynamoDB conditional write race Sequence diagram: Writer A and Writer B both issue PutItem with ConditionExpression attribute_not_exists(pk) for the same key against a DynamoDB table. The table processes A's write first and returns success, registering the item in PENDING state. B's write arrives second, the condition now evaluates false because the item exists, and the table returns ConditionalCheckFailedException. Writer A later transitions the item to COMPLETED via UpdateItem. Writer A DynamoDB Table Writer B PutItem cond: attribute_not_exists(pk) PutItem cond: attribute_not_exists(pk) item absent → write A 200 OK — pk=PENDING ConditionalCheckFailedException GetItem pk (consistent read) status=PENDING → poll or 409 execute business logic UpdateItem cond: status = PENDING 200 OK — pk=COMPLETED expires_at TTL attribute set at claim time governs eventual cleanup

The state machine has exactly three terminal-adjacent states:

  • PENDING — written by the first successful PutItem. Signals a claim is in flight. Carries expires_at set to a short window (for example now + 60 seconds) so a crashed worker does not block the key forever.
  • COMPLETED — written by an UpdateItem guarded with ConditionExpression: status = :pending, so only the caller that actually holds the claim can transition it. Carries the serialized response payload and a longer-lived expires_at.
  • FAILED — written on business-logic failure, with a short expires_at, so the next retry can immediately re-claim the key rather than waiting out a long TTL window.

Duplicate callers that lose the initial PutItem race read the item back with ConsistentRead: true, inspect status, and either return the cached COMPLETED response, poll briefly if still PENDING, or immediately retry if FAILED and its expires_at has passed.

Implementation Variants

Variant 1 — Put-If-Not-Exists with attribute_not_exists(pk)

The canonical first-writer-wins primitive. No prior read is required; the condition is evaluated server-side against the current state of the item.

import boto3, time
from botocore.exceptions import ClientError

table = boto3.resource("dynamodb").Table("idempotency_keys")

def claim(idempotency_key: str, ttl_seconds: int = 60) -> bool:
    now = int(time.time())
    try:
        table.put_item(
            Item={
                "pk": idempotency_key,
                "status": "PENDING",
                "created_at": now,
                "expires_at": now + ttl_seconds,
            },
            ConditionExpression="attribute_not_exists(pk)",
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise

This variant is the right default for single-item idempotency records with no related state that must commit atomically.

Variant 2 — Optimistic Version Increment

Once an item exists, subsequent transitions (PENDING → COMPLETED, or a retried counter increment) use a version attribute rather than presence/absence. Every writer reads the current version, then writes with a condition that the version has not changed since the read — the same optimistic-concurrency pattern used for version-column concurrency control, applied to a DynamoDB item instead of a relational row.

def complete(idempotency_key: str, response_payload: dict, expected_version: int):
    table.update_item(
        Key={"pk": idempotency_key},
        UpdateExpression=(
            "SET #s = :completed, response = :payload, "
            "version = version + :one, expires_at = :exp"
        ),
        ConditionExpression="version = :expected",
        ExpressionAttributeNames={"#s": "status"},
        ExpressionAttributeValues={
            ":completed": "COMPLETED",
            ":payload": response_payload,
            ":one": 1,
            ":expected": expected_version,
            ":exp": int(time.time()) + 86400,
        },
    )

A ConditionalCheckFailedException here means a concurrent writer already advanced the version — the caller must re-read the item and decide whether the request is already COMPLETED (return the cached response) or still PENDING under a different worker (back off and poll).

Variant 3 — TransactWriteItems for Multi-Item Atomicity

When the idempotency claim and the actual business-state mutation live in separate items — or separate tables — a lone conditional PutItem is not enough; the two writes must commit or fail together. TransactWriteItems bundles up to 100 item-level operations, each with its own ConditionExpression, into a single all-or-nothing commit.

def claim_and_reserve_inventory(idempotency_key: str, sku: str, qty: int):
    dynamodb = boto3.client("dynamodb")
    now = int(time.time())
    dynamodb.transact_write_items(
        TransactItems=[
            {
                "Put": {
                    "TableName": "idempotency_keys",
                    "Item": {
                        "pk": {"S": idempotency_key},
                        "status": {"S": "PENDING"},
                        "expires_at": {"N": str(now + 60)},
                    },
                    "ConditionExpression": "attribute_not_exists(pk)",
                }
            },
            {
                "Update": {
                    "TableName": "inventory",
                    "Key": {"sku": {"S": sku}},
                    "UpdateExpression": "SET reserved = reserved + :qty",
                    "ConditionExpression": "available >= :qty",
                    "ExpressionAttributeValues": {":qty": {"N": str(qty)}},
                }
            },
        ]
    )

If the inventory reservation’s condition fails, the idempotency-key Put is rolled back too — no orphaned PENDING claim is left behind for a request that never actually reserved inventory. TransactWriteItems costs 2x the write capacity of the equivalent non-transactional operations and has a hard ceiling of 100 items per call, so it is reserved for operations where cross-item atomicity is a correctness requirement, not applied by default to every write.

Variant Comparison

Variant Primitive Atomicity Scope Extra Cost Best For
Put-if-not-exists attribute_not_exists(pk) Single item None (standard write) Initial key claim, no related items
Optimistic version increment version = :expected Single item None (standard write) State transitions on an existing item
TransactWriteItems Multi-item ConditionExpression set Up to 100 items, all-or-nothing 2x WCU of equivalent standalone writes Key claim + business-state mutation that must commit together

These variants compose: a production deduplication path typically uses Variant 1 for the initial claim, Variant 3 when that claim must be atomic with a business-state write, and Variant 2 for the later PENDING → COMPLETED transition.

Edge Cases and Failure Scenarios

Failure Scenario Remediation Steps Observability Hooks
TTL delete race — DynamoDB’s background TTL sweeper deletes an expired COMPLETED item at the same moment a very late duplicate request arrives, so the duplicate sees no item and reprocesses Set the COMPLETED TTL window to comfortably exceed the client’s maximum retry ceiling (not just the PENDING window); never treat “item absent” as proof a request is new without also checking a durable audit log for recently expired keys dynamo_ttl_delete_before_retry_total counter cross-referenced against the audit log; DynamoDB Streams REMOVE event age vs. last-seen request timestamp
Hot partition key — a single high-volume idempotency key (or a poorly distributed key scheme, e.g. hashing only by tenant_id) concentrates all traffic on one partition, exceeding its throughput allocation Derive the partition key from the full request fingerprint (tenant + payload hash), never from a low-cardinality field alone; enable on-demand capacity mode or provision with adaptive capacity headroom CloudWatch ThrottledRequests filtered by table; ConsumedWriteCapacityUnits per-partition estimate via DescribeTable throughput dashboards
Throttling under burst load — a traffic spike exceeds provisioned WCU/RCU, and ProvisionedThroughputExceededException is returned instead of a clean condition-check result Implement exponential backoff with jitter on ProvisionedThroughputExceededException (distinct from ConditionalCheckFailedException, which must never be retried blindly); switch to on-demand billing mode for spiky idempotency workloads CloudWatch ThrottledRequests and ConsumedWriteCapacityUnits alarms; structured log field dynamo_error_code separating throttling from condition failures
Transaction cancellation without a clear reason — TransactWriteItems fails with TransactionCanceledException, and the CancellationReasons array must be parsed to know which item’s condition failed Always inspect CancellationReasons per-item rather than treating the whole transaction failure as a single generic error; map each reason code (ConditionalCheckFailed, ItemCollectionSizeLimitExceeded, None) to a distinct retry or reject branch transact_write_cancellation_reason label on a counter metric; log the full CancellationReasons array with the request’s idempotency key
Stale reads on a Global Table replica — a request routed to a secondary region reads before replication of a PENDING→COMPLETED transition has propagated, and reprocesses a request already completed elsewhere Pin idempotency-sensitive reads and writes to a single home region per key (regional affinity routing on a hash of the key); never rely on cross-region strong consistency for conditional writes Global Tables replication latency metric (ReplicationLatency); dynamo_cross_region_duplicate_detected_total reconciliation counter

Operational Concerns

WCU/RCU Budgeting

Every conditional PutItem or UpdateItem consumes standard write capacity regardless of whether the condition passes or fails — a failed condition check still costs a write unit. Budget capacity for the worst case, not the steady-state success rate:

required_WCU = peak_requests_per_second × (1 + expected_duplicate_ratio)

For an endpoint seeing 2,000 requests per second at peak with an expected 15% duplicate-retry ratio, budget for 2,300 WCU, not 2,000. TransactWriteItems operations consume 2x the WCU of the equivalent number of standalone item writes, so an idempotency-plus-business-state transaction touching two items costs 4 WCU-equivalents per call, not 2. For unpredictable or bursty traffic, on-demand capacity mode removes the need to pre-provision this budget at the cost of a higher per-request price.

TTL Lag: Plan for Up to 48 Hours

DynamoDB documents TTL deletion as a best-effort background process that typically completes within a day but can lag up to 48 hours under load. Two consequences follow directly:

  1. Never size storage cost projections assuming immediate deletion at the TTL boundary. Table storage and any GSI mirroring the idempotency table must account for a 48-hour buffer of “expired but not yet deleted” items.
  2. Never use item absence as the sole signal that a key has never been seen. Combine the expires_at attribute check in application code with the TTL sweeper — read the item, and if present, check whether expires_at has passed before trusting its status, rather than assuming TTL has already removed it.

Partition-Key Design

The partition key should be the full deterministic request fingerprint — typically the client-supplied idempotency key, or a hash combining tenant ID, endpoint, and normalized payload. Avoid partition keys derived from low-cardinality fields (a single tenant ID, a fixed queue name, a date bucket) — these concentrate all writes for that value onto a single partition and produce the hot-partition throttling scenario above. A well-distributed key scheme spreads write load evenly and lets DynamoDB’s adaptive capacity allocate throughput per-partition without manual sharding.

For workloads that also need range queries over idempotency records (for example, “all keys created for tenant X in the last hour”), add tenant ID as the partition key and the fingerprint as the sort key, then rely on TransactWriteItems rather than a bare attribute_not_exists(pk) condition when the sort key alone does not uniquely identify the claim.