Chaos Engineering for Idempotency

Part of: Observability & Operations for Idempotent Systems

A dedup layer that has never been tested under fault is a dedup layer with an unknown failure mode. Unit tests confirm that a single duplicate request is deduplicated under ideal conditions; they say nothing about what happens when the dedup store fails over mid-write, a lease-holder’s clock drifts, or ten thousand retried requests arrive in the same 200 millisecond window. Chaos engineering closes that gap by deliberately injecting the faults that break idempotency guarantees in a controlled, observable, reversible way — and proving, with evidence rather than assumption, that the steady-state holds.

This page adapts the standard hypothesis-driven chaos method (define steady state, form a hypothesis, inject a real-world fault, observe, and roll back) to the specific failure surface of deduplication systems: idempotency key storage, distributed locks and leases, and consensus-backed logs. For runnable, copy-pasteable game-day scripts, see the two runbooks linked at the end of this page: chaos-testing duplicate request handling and injecting clock skew to validate TTL safety.

---

Problem Framing

Deduplication logic is deceptively easy to verify in the happy path and deceptively hard to verify under fault, because the faults that break it are precisely the faults that are absent from a developer’s laptop: a Redis primary failing over mid-SETNX, a load balancer retrying a request whose response was lost in flight, a container node whose wall clock has stepped forward 90 seconds after an NTP correction. Each of these faults is rare in isolation and common in aggregate across a large fleet running for months. Waiting for them to occur naturally in production means learning about a broken dedup guarantee from a customer support ticket or a reconciliation report — after the duplicate charge has already settled.

Chaos engineering inverts this: it manufactures the rare fault on a schedule you control, in a blast radius you bound, with rollback automation you have already tested. The goal is never to “break things randomly.” The goal is to falsify a specific, written hypothesis about how your idempotent consumer or webhook handler behaves when one well-understood thing goes wrong.

---

Guarantee Model

The steady-state metric for an idempotency-focused chaos program is narrower than general availability chaos: it is not “the service stayed up,” it is the duplicate-side-effect rate stayed at zero. Formally, for any two logically identical requests (same idempotency key, same payload hash) received within the configured deduplication window, the system commits at most one side effect — one charge, one email, one inventory decrement — regardless of:

• How many times the request was retried (retry_count from 1 to unbounded).
• Whether the dedup store failed over between the first and second attempt.
• Whether the two attempts landed on different application instances.
• Whether the wall clock of any participating node drifted during the window.

This guarantee breaks under two conditions that chaos experiments are specifically designed to surface:

1. The dedup check and the side effect are not atomic. If a key is marked “seen” in a separate transaction from the side effect it guards, a crash between the two leaves the system either double-executing (check failed to persist) or permanently stuck (side effect failed but key persisted). Wrapping transactions for safe retries closes this specific gap.
2. The TTL or lease window is shorter than the true tail latency of the operation it guards. A fault that adds even 500 milliseconds of latency can push a legitimate retry past a TTL boundary that looked safe under nominal conditions, causing the dedup store to treat a legitimate retry as a brand-new request.

Chaos experiments exist to find the gap between the guarantee you believe you have and the guarantee your system actually enforces under fault.

---

Core Method: The Hypothesis-Injection-Observation Loop

Every experiment in this program follows the same four-stage loop. The loop is deliberately small and fast so that a single game day can run many experiments, each testing one fault in isolation.

Hypothesis. Written before any fault is injected, in the form: “When [fault] occurs for [duration] affecting [blast radius], the duplicate-side-effect rate remains at 0 and p99 latency stays under [threshold].” A hypothesis without a numeric threshold is not falsifiable and should be rejected in game-day review.

Inject. Apply the smallest fault that can falsify the hypothesis — one pod, one availability zone, one percent of traffic — never the largest fault available in the tool.

Observe. Watch the steady-state metric on a real-time dashboard, not a batch report. The observation window must be short enough that a human or an automated stop condition can react before damage compounds — typically 30 to 120 seconds for dedup experiments.

Rollback or widen. If the steady-state metric breaches its threshold, abort immediately, restore normal conditions, and write a postmortem before touching the experiment again. If it holds, widen the blast radius (more traffic, longer duration, or a second concurrent fault) on the next run.

---

The Idempotency Fault Taxonomy

Not every fault is worth injecting. These six categories account for nearly all real-world dedup incidents and map directly onto the failure scenarios table below:

1. Duplicate delivery storms — an upstream retries the same request many times in a short window, typically after a timeout that was too aggressive relative to true p99 latency.
2. Dedup-store failover — the backing store (Redis, DynamoDB, Postgres) promotes a replica mid-request, and in-flight writes to the old primary are lost or replayed against the new one.
3. Clock skew — a node’s wall clock jumps or drifts, causing premature TTL expiry or a lease to appear valid after it has already been reassigned.
4. Lock or lease loss — a lease expires mid-operation due to a GC pause or renewal failure, and the holder keeps writing as if it still owned the resource.
5. Network partition — the application can reach the dedup store but not a downstream dependency, or vice versa, producing an asymmetric split where retries and the original request disagree about outcome.
6. Retry storms — a downstream slowdown causes synchronized retries across many clients simultaneously, amplifying load on the exact component (the dedup store) that is supposed to absorb duplicates.

---

Implementation Variants

Manual Fault-Injection Scripts

The lowest-overhead starting point: shell scripts that use tc netem for latency/loss and iptables for partitions, invoked directly against a staging host. No new infrastructure, but blast radius control and rollback are entirely manual — a forgotten cleanup step leaves the fault active indefinitely.

#!/usr/bin/env bash
# Inject 400ms of latency + 5% packet loss on the path to the dedup store
# to simulate degraded Redis connectivity ahead of a fault-tolerance game day.
set -euo pipefail

IFACE="eth0"
DEDUP_STORE_IP="10.0.4.12"

# Add a delay + loss qdisc scoped to traffic destined for the dedup store only
tc qdisc add dev "$IFACE" root handle 1: prio
tc qdisc add dev "$IFACE" parent 1:3 handle 30: netem delay 400ms loss 5%
tc filter add dev "$IFACE" protocol ip parent 1:0 prio 3 u32 \
  match ip dst "$DEDUP_STORE_IP" flowid 1:3

echo "Fault active. Rollback with: tc qdisc del dev $IFACE root"

Toxiproxy

Toxiproxy sits as a programmable TCP proxy between the application and the dedup store, letting you add and remove faults (latency, timeout, bandwidth caps, connection resets) via an HTTP API without touching kernel networking. This is the recommended default for dedup-layer experiments because faults are removed with a single API call — there is no host-level state to forget to clean up.

# Create a proxy in front of the Redis dedup store, then inject a 2s timeout
# toxic that simulates the store becoming unresponsive mid-SETNX.
toxiproxy-cli create redis_dedup --listen 0.0.0.0:26379 --upstream redis:6379
toxiproxy-cli toxic add redis_dedup --type timeout --attribute timeout=2000
# ... observe steady-state metric ...
toxiproxy-cli toxic remove redis_dedup --toxicName timeout_downstream

Litmus / Chaos Mesh

For dedup layers running on Kubernetes, Chaos Mesh and Litmus define faults as CRDs, giving you declarative blast-radius scoping (label selectors, percentage of pods) and built-in scheduling for recurring game days.

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: dedup-store-partition
spec:
  action: partition
  mode: fixed-percent
  value: "25"
  selector:
    labelSelectors:
      app: payment-api
  direction: to
  target:
    selector:
      labelSelectors:
        app: redis-dedup-store
    mode: all
  duration: "60s"
  scheduler:
    cron: "@every 30m"

AWS FIS

AWS Fault Injection Service is the managed option for teams already on AWS, with built-in stop conditions tied directly to CloudWatch alarms — the experiment halts automatically if the alarm fires, without a separate watcher process.

description: "API Gateway to DynamoDB dedup table: inject throttling"
targets:
  DedupTable:
    resourceType: "aws:dynamodb:table"
    resourceTags:
      chaos-target: "idempotency-store"
actions:
  throttleWrites:
    actionId: "aws:dynamodb:global-table-pause-replication"
    parameters:
      duration: "PT2M"
stopConditions:
  - source: "aws:cloudwatch:alarm"
    value: "arn:aws:cloudwatch:us-east-1:111122223333:alarm:duplicate-side-effect-rate-nonzero"

Summary Comparison

---

Edge Cases and Failure Scenarios

---

Operational Concerns

Blast Radius

Every experiment must declare its blast radius before injection: percentage of traffic, number of hosts, or a specific availability zone — never “all of production.” Start every new fault type at the smallest viable radius (a single canary instance or 1% of traffic) and only widen after the hypothesis has held at the smaller scope at least twice.

Staging vs. Production

Staging validates the mechanics of the experiment itself: does the fault actually inject, does the observation dashboard actually show the metric moving, does the rollback actually restore normal conditions within the target time. Production validates the thing you actually care about: whether the guarantee holds under real traffic patterns, real data skew, and real infrastructure topology. Treat a staging-only chaos program as incomplete — synthetic traffic rarely reproduces the concurrency patterns that break dedup logic in practice. Graduate to production only after the rollback path has been exercised at least 10 times in staging without a manual intervention.

Game-Day Cadence

Run a scheduled game day at minimum quarterly, covering the full fault taxonomy once per cycle. High-change-velocity services (recent deploys touching the dedup path, new consensus backend, TTL changes) should run a targeted experiment within one week of the change landing, not wait for the next quarterly cycle.

Tie Experiments to SLOs

Every experiment’s steady-state hypothesis must reference a real, already-alerting SLO — not an ad hoc threshold invented for the experiment. If monitoring dashboards do not already track duplicate_side_effect_rate and lease/lock health, build that dashboard before running chaos experiments against it; an experiment against an unmonitored guarantee only tells you the guarantee broke, not how quickly you would have noticed in production.

---

Related

• Observability & Operations for Idempotent Systems — parent section covering monitoring, distributed tracing, and chaos engineering for dedup layers
• Chaos-Testing Duplicate Request Handling — runbook scripting concurrent duplicate requests against a payment endpoint and asserting exactly one side effect
• Injecting Clock Skew to Validate TTL Safety — runbook skewing a node’s clock to prove fencing tokens prevent premature TTL/lease expiry damage
• Lock Timeout & Lease Management — TTL sizing and fencing token mechanics that chaos experiments are designed to validate
• Monitoring Idempotency Metrics and Dashboards — the dashboards and SLOs that chaos experiments must be tied to before they run in production
• Mitigating Thundering Herd During Retry Storms — server-side defenses against the retry-storm fault covered in this page’s taxonomy