Using Redis SETNX for Distributed Request Deduplication

Part of: Redis & Cache-Based Deduplication

Problem Statement and Prerequisites

This runbook shows you how to build an idempotency middleware layer around Redis SET key value NX EX ttl so that duplicate HTTP requests — arriving within a configurable window — are detected and rejected before they reach your business logic. The technique is covered in depth under Redis & Cache-Based Deduplication; read that page first if you need to understand the guarantee model and where this approach sits relative to database-level uniqueness constraints.

Before starting, you should understand:

• How idempotency keys are generated and what entropy is required — a weak or predictable key negates the entire guard.
• The retry logic and backoff fundamentals your clients use, because your TTL must outlast the maximum retry window.
• The backend storage patterns landscape so you know when to fall back to database unique constraints instead.

The architecture of this middleware looks like the diagram below. The critical property: the SET NX check runs atomically on the Redis primary, so no two concurrent workers can both proceed past the guard for the same key.

---

Step-by-Step Implementation

Step 1 — Design a Deterministic Key Schema

The key must be deterministic: the same logical request must always produce the same key, and different requests must never collide. A safe schema is:

idemp:{HTTP_METHOD}:{ROUTE_TEMPLATE}:{SHA256(X-Idempotency-Key)[0:32]}

Hashing the client-supplied value prevents injection attacks and bounds the key to a fixed 32-character hex prefix (128 bits of collision resistance). Use the route template, not the resolved URL, so /payments/123 and /payments/456 map to /payments/:id and do not share a key namespace.

Step 2 — Issue the Atomic SET NX EX Command

SETNX key value is deprecated; use SET key value NX EX <seconds> in all modern Redis clients. This is a single round-trip atomic operation: if the key does not exist the command sets it, starts the TTL, and returns OK; if the key already exists it returns nil. No separate EXPIRE step is needed, which removes the race window that existed with the old two-command pattern.

Node.js / ioredis

const Redis = require('ioredis');
const crypto = require('crypto');
const redis = new Redis({ enableAutoPipelining: true });

async function acquireIdempotencyKey(method, routeTemplate, rawKey, ttlSeconds = 60) {
  const keyHash = crypto.createHash('sha256').update(rawKey).digest('hex').slice(0, 32);
  const redisKey = `idemp:${method}:${routeTemplate}:${keyHash}`;
  // Returns 'OK' if the key was newly set; returns null if it already existed.
  const result = await redis.set(redisKey, '1', 'EX', ttlSeconds, 'NX');
  return { acquired: result === 'OK', redisKey };
}

Python / redis-py async

import hashlib
import redis.asyncio as aioredis

redis = aioredis.Redis(host="redis-primary", decode_responses=True)

async def acquire_idempotency_key(
    method: str, route_template: str, raw_key: str, ttl: int = 60
) -> bool:
    key_hash = hashlib.sha256(raw_key.encode()).hexdigest()[:32]
    redis_key = f"idemp:{method}:{route_template}:{key_hash}"
    # set returns True if newly created, False if the key already existed.
    acquired = await redis.set(redis_key, "1", nx=True, ex=ttl)
    return bool(acquired)

Go / go-redis

import (
    "context"
    "crypto/sha256"
    "fmt"
    "time"
    "github.com/redis/go-redis/v9"
)

var rdb = redis.NewClient(&redis.Options{Addr: "redis-primary:6379"})

func AcquireIdempotencyKey(ctx context.Context, method, routeTemplate, rawKey string, ttl time.Duration) (bool, string, error) {
    h := sha256.Sum256([]byte(rawKey))
    keyHash := fmt.Sprintf("%x", h[:16]) // 32 hex chars from 16 bytes
    redisKey := fmt.Sprintf("idemp:%s:%s:%s", method, routeTemplate, keyHash)
    // SetNX returns true if the key was newly set, false if it already existed.
    acquired, err := rdb.SetNX(ctx, redisKey, "1", ttl).Result()
    return acquired, redisKey, err
}

Step 3 — Wire the Middleware and Return 409 on Duplicates

In all three runtimes, the pattern is the same: call acquireIdempotencyKey at the start of your request handler. If acquired is false, return 409 Conflict immediately — do not proceed to business logic.

// Express middleware (Node.js)
async function idempotencyMiddleware(req, res, next) {
  const rawKey = req.headers['x-idempotency-key'];
  if (!rawKey) return next(); // non-idempotent endpoints can skip

  const { acquired } = await acquireIdempotencyKey(
    req.method, req.route?.path ?? req.path, rawKey, 60
  );

  if (!acquired) {
    return res.status(409).json({
      error: 'duplicate_request',
      message: 'An identical request is already being processed or was recently completed.',
    });
  }
  next();
}

Step 4 — Calibrate the TTL

The TTL must be at least: max_downstream_latency_p99 + max_client_retry_interval + 10 s buffer. For synchronous payment APIs with a 5 s SLA and clients retrying up to 3 times over 30 s, use ttl = 60 s. For webhook delivery with multi-minute retry windows, use ttl = 300 s. Never use TTLs shorter than 10 s — that window is too narrow to absorb realistic network variance.

For an in-depth discussion of how TTL interacts with idempotency key storage and eviction under memory pressure, see that dedicated page.

Step 5 — Handle Downstream Failure Without Blocking Retries

The most dangerous failure mode: SET NX succeeds (key is created), but your downstream service call fails or times out. The key remains set for the full TTL, blocking legitimate client retries. Prevent this with a status-field pattern using a Redis hash:

async function acquireWithStatus(redisKey, ttlSeconds = 60) {
  const pipeline = redis.pipeline();
  // Atomic: only sets all fields if key does not exist (HSETNX on first field)
  pipeline.hsetnx(redisKey, 'status', 'PENDING');
  pipeline.hsetnx(redisKey, 'created_at', Date.now());
  pipeline.expire(redisKey, ttlSeconds);
  const results = await pipeline.exec();
  const acquired = results[0][1] === 1; // hsetnx returns 1 on first set
  return acquired;
}

async function commitIdempotencyKey(redisKey) {
  await redis.hset(redisKey, 'status', 'COMMITTED');
}

async function releaseIdempotencyKey(redisKey) {
  // On terminal failure, delete the key so the client can retry immediately.
  await redis.del(redisKey);
}

Call commitIdempotencyKey after a successful downstream response. Call releaseIdempotencyKey in your catch block for terminal errors (not timeouts — for timeouts, let the TTL expire to prevent double-processing).

---

Verification and Testing

Simulate a Duplicate Request with curl

KEY="test-$(uuidgen)"

# First request — should succeed with 200
curl -s -o /dev/null -w "%{http_code}" -X POST https://api.example.com/v1/payments \
  -H "X-Idempotency-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"amount": 100}'

# Immediate second request with the same key — must return 409
curl -s -o /dev/null -w "%{http_code}" -X POST https://api.example.com/v1/payments \
  -H "X-Idempotency-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"amount": 100}'

Inspect the Key in Redis

KEY_HASH=$(echo -n "$KEY" | sha256sum | cut -c1-32)
REDIS_KEY="idemp:POST:/v1/payments:$KEY_HASH"

# Confirm the key exists
redis-cli EXISTS "$REDIS_KEY"

# Inspect its TTL (should be close to 60)
redis-cli TTL "$REDIS_KEY"

# Inspect status field if using the hash pattern
redis-cli HGETALL "$REDIS_KEY"

Monitor Live SET NX Traffic

# Stream all SET NX EX commands in real time
redis-cli MONITOR | grep -E "SET.*NX.*EX"

# Retrieve recent slow SET NX operations (threshold: 10 ms = 10000 µs)
redis-cli CONFIG SET slowlog-log-slower-than 10000
redis-cli SLOWLOG GET 20

Integration Test Pattern (Node.js / Jest)

test('duplicate request returns 409', async () => {
  const key = crypto.randomUUID();
  const headers = { 'x-idempotency-key': key, 'content-type': 'application/json' };
  const body = JSON.stringify({ amount: 100 });

  const first = await fetch('/v1/payments', { method: 'POST', headers, body });
  expect(first.status).toBe(200);

  const second = await fetch('/v1/payments', { method: 'POST', headers, body });
  expect(second.status).toBe(409);
});

---

Failure Scenarios and Debugging

---

SRE / Observability Checklist

1. Emit idempotency.setnx.result — a counter with label result=ok|nil|error incremented on every SET NX call. This is your primary signal for duplicate request volume.
2. Emit idempotency.key.ttl_remaining — a histogram of the TTL at commit time (redis-cli TTL after business logic succeeds). Values close to 0 indicate the TTL is too short.
3. Attach redis.setnx.result span attribute — OpenTelemetry span attribute on every idempotency check span; value ok or nil; set span status to ERROR for nil and return 409.
4. Log structured fields — every middleware decision must emit {"idempotency_key_hash": "...", "redis_result": "ok|nil|error", "http_status": 200|409|503, "trace_id": "..."} as a single JSON log line.
5. Alert dedup_bypass_detected — fire when http_status=200 is returned for a request that carried a previously seen X-Idempotency-Key within the same TTL window (detectable via log correlation).
6. Track redis_command_duration_ms{command="set_nx"} — p50/p99 latency for the SET NX EX call; alert if p99 exceeds 5 ms, which may indicate Redis memory pressure or network saturation.

---

Related

• Redis & Cache-Based Deduplication — parent page covering the full guarantee model, memory budgeting, and TTL eviction strategy for cache-layer idempotency.
• Idempotency Key Storage & TTL Management — deep dive into TTL calibration, eviction policies, and storage cost at scale.
• How to Generate Cryptographically Secure Idempotency Keys — key generation strategies that feed the SETNX guard; covers UUIDv4, UUIDv7, and HMAC-deterministic keys.
• Implementing Exponential Backoff Without Overlapping Retries — client-side retry design that must align with your TTL window.
• PostgreSQL Unique Constraints vs Application-Level Checks — when to replace or supplement the Redis guard with a database-level uniqueness constraint.