Coordinating Deduplication with ZooKeeper

Part of: Consensus Algorithms for Deduplication

Apache ZooKeeper predates Raft-native stores like etcd but remains the dominant coordination service in JVM-heavy stacks — Kafka’s older controller quorum, Hadoop, Solr, and countless internal platforms already run a ZooKeeper ensemble. Rather than standing up a second consensus cluster, many teams reuse that ensemble to gate request deduplication: a single, atomic create call replaces a compare-and-swap that would otherwise need a database round trip. This page is a focused implementation runbook for the Consensus Algorithms for Deduplication pattern, and it assumes you already understand the linearizability guarantees and commit-then-execute model described there, plus the broader coordination contract in Distributed Coordination & Locking Strategies.

---

Problem statement and prerequisites

What you are implementing: a per-idempotency-key exclusive lock (or leader election) backed by ZooKeeper’s ephemeral-sequential znode recipe, so that only one node executes the business logic for a given key while competitors block on a watch instead of polling.

Prerequisites:

• A running ZooKeeper ensemble of at least 3 nodes (tolerates 1 failure) or 5 nodes (tolerates 2 failures) — never run an even-sized ensemble, since ZAB quorum math gains nothing from the extra node.
• Familiarity with distributed lock acquisition patterns in general, and why naive SETNX-style locks differ from ZooKeeper’s session-bound ephemeral nodes.
• A client library that exposes watches and multi-op transactions: Apache Curator for Java/JVM, or kazoo for Python.
• An understanding of lock timeout and lease management — ZooKeeper’s session timeout plays the same role as a Redis or etcd lease TTL, but it is renewed by heartbeats the client library sends automatically, not by application code.

---

The ZAB guarantee and why it matters for dedup

ZooKeeper’s replication protocol, ZAB (ZooKeeper Atomic Broadcast), elects a single leader per epoch and broadcasts all writes through that leader in strict order. Every follower applies writes in the exact sequence the leader committed them, which gives ZooKeeper two properties that matter for deduplication:

• Linearizable writes. A create("/dedup/req-", ..., EPHEMERAL_SEQUENTIAL) call either lands at a specific, globally-agreed sequence number or fails outright — there is no window where two clients believe they created the lowest-numbered node.
• Sequential consistency for reads. A client that just wrote to the ensemble always sees its own write on its next read. But a read against a lagging follower can return stale data relative to the global order unless the client calls sync() first. For dedup lock ownership checks, this means: always issue getChildren() against the same session that just created your znode, and if a second reader on a different session needs an authoritative view, have it call sync() before getChildren().

This is weaker than the fully linearizable reads a Raft-based store like etcd offers by default, but it is sufficient for the lock-ownership pattern below because ownership is always decided by the creating session’s own view of the child list, never by a third party guessing.

---

Znode structure and the watch chain

The canonical recipe creates one ephemeral-sequential child per contending node under a shared parent path, then has each node watch only the child immediately below its own sequence number — never the whole parent, and never every node ahead of it.

---

Step-by-step implementation

Step 1 — Create an ephemeral-sequential znode under /dedup/<key>

Each contending node creates a child znode under a parent path namespaced by the idempotency key. EPHEMERAL_SEQUENTIAL guarantees the node both disappears automatically on session loss and receives a monotonically increasing suffix assigned by the leader.

Java (Apache Curator)

CuratorFramework client = CuratorFrameworkFactory.newClient(
    "zk1:2181,zk2:2181,zk3:2181",
    new ExponentialBackoffRetry(1000, 3)
);
client.start();

String basePath = "/dedup/" + idempotencyKey;
client.createContainers(basePath); // idempotent parent creation

String myPath = client.create()
    .withMode(CreateMode.EPHEMERAL_SEQUENTIAL)
    .forPath(basePath + "/lock-");
// myPath e.g. "/dedup/req-42/lock-0000000002"

Python (kazoo)

from kazoo.client import KazooClient

zk = KazooClient(hosts="zk1:2181,zk2:2181,zk3:2181", timeout=12.0)
zk.start()

base_path = f"/dedup/{idempotency_key}"
zk.ensure_path(base_path)

my_path = zk.create(
    f"{base_path}/lock-",
    value=payload_hash.encode("utf-8"),
    ephemeral=True,
    sequence=True,
)
# my_path e.g. "/dedup/req-42/lock-0000000002"

Rather than hand-rolling this, both libraries ship a tested recipe: Curator’s InterProcessMutex and kazoo’s Lock implement steps 1 through 4 for you. Use the recipe in production; the raw calls below exist so you understand what the recipe is actually doing before you depend on it.

Step 2 — List children and determine ownership by lowest sequence number

List<String> children = client.getChildren().forPath(basePath);
Collections.sort(children); // lexicographic sort works because sequence is zero-padded to 10 digits

String myNode = myPath.substring(basePath.length() + 1);
if (children.get(0).equals(myNode)) {
    // This node holds the lock — proceed with the commit-then-execute
    // flow described in Consensus Algorithms for Deduplication.
    executeBusinessLogic(idempotencyKey);
} else {
    watchPredecessor(children, myNode);
}

children = sorted(zk.get_children(base_path))
my_node = my_path.rsplit("/", 1)[-1]

if children[0] == my_node:
    execute_business_logic(idempotency_key)
else:
    watch_predecessor(children, my_node)

Step 3 — Watch only the immediate predecessor znode

This is the step that distinguishes a production-grade recipe from a naive one. Setting a watch on the full child list under /dedup/<key> means every waiter is notified on every single acquisition and release — an O(n) notification storm against the ensemble for n waiters. Watching only the next-lowest sequence number bounds notification fan-out to exactly one client per release.

private void watchPredecessor(List<String> children, String myNode) throws Exception {
    int myIndex = children.indexOf(myNode);
    String predecessor = children.get(myIndex - 1);

    Stat stat = client.checkExists()
        .usingWatcher((WatchedEvent event) -> {
            if (event.getType() == Watcher.Event.EventType.NodeDeleted) {
                recheckOwnership(); // re-list and compare again — see Step 2
            }
        })
        .forPath(basePath + "/" + predecessor);

    if (stat == null) {
        recheckOwnership(); // predecessor already gone — race with our own listing
    }
}

def watch_predecessor(children, my_node):
    my_index = children.index(my_node)
    predecessor = children[my_index - 1]

    @zk.DataWatch(f"{base_path}/{predecessor}")
    def on_predecessor_change(data, stat, event):
        if event is not None and event.type == "DELETED":
            recheck_ownership()

Step 4 — Re-check ownership when the watch fires

Do not assume that a predecessor’s deletion automatically means you now hold the lock — always re-list and re-sort, because a fast-failing intermediate node could have been deleted concurrently. This is the same recheckOwnership / recheck_ownership call as Step 2; loop back to it until your znode is the lowest.

Step 5 — Detect and recover from session expiration

If the ZooKeeper client’s TCP connection to the ensemble is unreachable for longer than the negotiated session timeout, the ensemble expires the session and deletes every ephemeral znode it owned — including your lock znode, mid-operation, with no notification to the disconnected process until it reconnects.

client.getConnectionStateListenable().addListener((c, newState) -> {
    if (newState == ConnectionState.LOST) {
        // Session expired. Any in-flight business logic under this lock
        // must be treated as unsafe — abort, do not commit downstream effects,
        // and let the retry path re-enter with a fresh idempotency check.
        abortInFlightOperation(idempotencyKey);
    } else if (newState == ConnectionState.SUSPENDED) {
        // Connectivity lost but session not yet expired — pause, do not abort yet.
        pauseUntilReconnectedOrExpired();
    }
});

from kazoo.exceptions import SessionExpiredError

@zk.add_listener
def session_watcher(state):
    if state == "LOST":
        abort_in_flight_operation(idempotency_key)
    elif state == "SUSPENDED":
        pause_until_reconnected_or_expired()

try:
    zk.retry(zk.get, my_path)
except SessionExpiredError:
    abort_in_flight_operation(idempotency_key)

Set the session timeout deliberately: 12000 ms is a common default for payment-adjacent services (2× the tick time of 2000 ms, times a 3x safety margin), long enough to survive a garbage-collection pause but short enough that a genuinely dead node releases its lock quickly. Align this value with the lease TTL guidance used elsewhere in your coordination layer so that downstream idempotency records do not outlive the lock that was protecting them.

---

Verification and testing

Force a session expiry mid-operation and confirm the lock is released and reassigned

# Terminal 1: start a client holding the lock, then pause the process to
# simulate a GC stall or network partition longer than the session timeout.
kill -STOP $(pgrep -f dedup-worker)

# Terminal 2: confirm the ephemeral znode disappears once the session
# timeout elapses (12s in this example) and a second waiter is promoted.
zkCli.sh -server zk1:2181 ls /dedup/req-42
# Expect: lock-0000000001 has been removed; lock-0000000002 remains.

# Resume the stalled process — its client library must surface
# ConnectionState.LOST / SessionExpiredError, not silently keep running.
kill -CONT $(pgrep -f dedup-worker)

Verify watch fan-out is bounded to one notification per release

# Attach three waiters, release the lock holder, and confirm only one
# GetChildren/exists call fires against the ensemble — inspect via
# ZooKeeper's four-letter word stats.
echo stat | nc zk1 2181 | grep "Received"
# Compare the counter before and after a single release; it should
# increase by roughly 1 per waiter woken, not by (n-1) extra reads.

Confirm sequential consistency across sessions

zkCli.sh -server zk1:2181 sync /dedup/req-42
zkCli.sh -server zk1:2181 get /dedup/req-42/lock-0000000001
# sync() forces this session to catch up to the leader's latest committed
# state before the subsequent read — required before trusting a read
# issued from a session other than the one that created the lock znode.

---

Failure scenarios and debugging

---

SRE / observability checklist

1. zk_session_expired_total — Counter incremented on every ConnectionState.LOST / SessionExpiredError transition. Alert if the rate exceeds 1/min sustained over a 5-minute window — indicates GC pressure or network instability, not isolated incidents.
2. zk_watch_notifications_total by path prefix — confirms watch fan-out stays O(1) per release. A ratio that scales with waiter count is a regression back to full-child-list watching.
3. zk_ephemeral_nodes_total per /dedup/<key> prefix — Gauge; alert if any single key’s child count exceeds expected concurrency, which signals leaked or duplicated lock attempts.
4. zk_quorum_size vs zk_ensemble_size — Gauge pair; page immediately if the serving quorum drops below ⌊N/2⌋ + 1 for more than 10 seconds.
5. zk_operation_latency_ms — Histogram (p50/p99) for create, getChildren, and exists calls against the ensemble; a rising p99 above 200 ms precedes session timeouts under load.
6. Structured log fields on every lock transition — idempotency_key, znode_path, session_id, event (created | acquired | watch_fired | released | session_expired). Index on idempotency_key for incident triage.

---

Related

• Consensus Algorithms for Deduplication — parent: the commit-then-execute model and linearizability guarantees this ZooKeeper recipe implements.
• Distributed Coordination & Locking Strategies — the full coordination contract and failure-boundary map this pattern sits within.
• Raft-Backed Exactly-Once Deduplication Logs — sibling: the Raft/etcd equivalent, with a client-request-id session table instead of ephemeral znodes.
• Distributed Lock Acquisition Patterns — related: lighter-weight advisory locking when a full ZooKeeper ensemble is not justified.
• Implementing Redlock for High-Availability Deduplication — related: a Redis-quorum alternative with different latency and operational trade-offs.
• Lock Timeout & Lease Management — related: TTL sizing principles that apply directly to ZooKeeper session-timeout configuration.