Implementing Lease Heartbeat Renewal with etcd

Part of: Lock Timeout & Lease Management

An etcd lease is a time-to-live object that one or more keys can attach to. The lease holder keeps it alive by sending periodic keepalive requests over a long-lived gRPC stream; if the requests stop, the lease expires and every attached key is deleted atomically. This makes etcd leases a strong primitive for the renewable-ownership model described in Lock Timeout & Lease Management — a holder does not get a fixed deadline, it gets a lease it must actively keep alive, matching the broader guarantees laid out in Distributed Coordination & Locking Strategies.

This page is a runbook, not a conceptual overview. You will grant a lease, attach it to a key, run a renewal loop at TTL/3, extract a fencing token from the write, and handle the stream breaking mid-flight — in both Go and Python.

---

The keepalive stream lifecycle

The diagram below traces one lease across three actors: the client, the etcd lease service, and the key(s) attached to it. The critical failure path — a stalled keepalive stream leading to automatic revocation — is shown in the lower half.

---

Problem statement and prerequisites

What you are implementing: granting a bounded-TTL lease, attaching it to a key, keeping it alive over a gRPC stream at a TTL/3 cadence, extracting a fencing token from the write, and reacting correctly the moment the keepalive stream is lost — before the TTL window expires and a competitor is granted the same lock.

Prerequisites:

• You understand the renewable-lease model from Lock Timeout & Lease Management, specifically why a lease is renewed rather than reset.
• You are running a 3-node or 5-node etcd cluster reachable from your application (etcd1:2379, etcd2:2379, etcd3:2379 below).
• You have clientv3 (Go) or python-etcd3 (Python) available, and etcdctl for manual verification.
• Your protected resource can store and compare a fencing token before accepting a write — without resource-side enforcement, the renewal machinery below only prevents split ownership, not stale writes.

---

Step-by-step implementation

Step 1 — Grant a lease with a bounded TTL

The TTL is the maximum time the holder can go silent before etcd reclaims the lease. Size it with headroom above your heartbeat interval and etcd’s own raft election timeout (default 1000 ms).

Go (clientv3)

package lease

import (
    "context"
    "time"

    clientv3 "go.etcd.io/etcd/client/v3"
)

func NewClient() (*clientv3.Client, error) {
    return clientv3.New(clientv3.Config{
        Endpoints:   []string{"etcd1:2379", "etcd2:2379", "etcd3:2379"},
        DialTimeout: 5 * time.Second,
    })
}

func GrantLease(ctx context.Context, cli *clientv3.Client, ttlSeconds int64) (clientv3.LeaseID, error) {
    resp, err := cli.Grant(ctx, ttlSeconds)
    if err != nil {
        return 0, err
    }
    return resp.ID, nil
}

Step 2 — Attach the lease to the protected key

A lease with no keys attached is inert. Attach it in the same Put that writes the lock record so acquisition and lease attachment are a single round trip.

func AcquireLock(ctx context.Context, cli *clientv3.Client, key, ownerID string, leaseID clientv3.LeaseID) (int64, error) {
    putResp, err := cli.Put(ctx, key, ownerID, clientv3.WithLease(leaseID))
    if err != nil {
        return 0, err
    }
    // header.Revision is the fencing token — see Step 4
    return putResp.Header.Revision, nil
}

Step 3 — Start a LeaseKeepAlive stream renewing at TTL/3

clientv3’s KeepAlive opens a channel and automatically sends a renewal roughly every TTL/3. The channel must be drained continuously — if nothing reads from it, the client library still sends keepalives, but your code loses its only signal that the lease is alive.

func Heartbeat(ctx context.Context, cli *clientv3.Client, leaseID clientv3.LeaseID, onLost func()) error {
    ch, err := cli.KeepAlive(ctx, leaseID)
    if err != nil {
        return err
    }
    go func() {
        for ka := range ch {
            _ = ka // ka.TTL confirms the renewed lifetime; log it at debug level
        }
        // Channel closes when the context is cancelled OR the lease is lost.
        if ctx.Err() == nil {
            onLost() // context still live — this is a real lease loss
        }
    }()
    return nil
}

Python (python-etcd3) does not offer an automatic background renewal loop with the same ergonomics, so drive it explicitly with a daemon thread at the same TTL/3 cadence:

import etcd3
import threading

client = etcd3.client(host="etcd1", port=2379)
lease = client.lease(ttl=9)
client.put("/locks/payment-42", "owner-1", lease=lease)

def heartbeat(lease, interval_s, stop_event, on_lost):
    while not stop_event.wait(interval_s):
        try:
            lease.refresh()
        except Exception:
            on_lost()
            return

stop_event = threading.Event()
t = threading.Thread(
    target=heartbeat,
    args=(lease, 3, stop_event, lambda: print("lease lost — aborting critical section")),
    daemon=True,
)
t.start()

Step 4 — Use the Put revision as a fencing token

Every etcd write returns a header.Revision — a counter that increases monotonically across the entire keyspace, not just the key being written. This makes it a free fencing token: no separate counter, no extra round trip.

fencingToken := putResp.Header.Revision // e.g. 10482

Enforce it at the resource exactly as you would a Redis or Postgres generation counter — see handling stale locks for the resource-side compare-and-reject pattern applied to Redis and Postgres. For etcd specifically, store the accepted revision alongside the resource and reject any write whose token is lower:

UPDATE payments
SET    status = 'processing', fencing_revision = $1
WHERE  payment_id = $2
  AND  fencing_revision < $1;
-- 0 rows affected means the caller's revision is stale — reject the write

Do not cache a fencing token across a lease loss and reuse it later: if history has been compacted between the loss and the retry, a comparison against a purged revision returns ErrCompacted from etcd’s MVCC store. Treat ErrCompacted exactly like a stale-token rejection — fetch a fresh revision on every re-acquisition instead of reusing an old one.

Step 5 — Detect stream loss and abort before TTL expiry

The keepalive channel closing is the only signal you get — etcd does not push a “your lease is about to expire” notification. Treat closure as loss immediately; do not wait for a subsequent operation to fail.

func Heartbeat(ctx context.Context, cli *clientv3.Client, leaseID clientv3.LeaseID, onLost func()) error {
    ch, err := cli.KeepAlive(ctx, leaseID)
    if err != nil {
        return err
    }
    go func() {
        for range ch {
        }
        if ctx.Err() == nil {
            // Confirm before assuming — a slow reconnect can still succeed.
            if _, ttlErr := cli.Lease.TimeToLive(ctx, leaseID); ttlErr != nil {
                onLost()
            }
        }
    }()
    return nil
}

onLost should immediately stop the in-flight critical section, release any local resources, and require a fresh LeaseGrant + Put before resuming — never assume the old lease can be reused after the stream reconnects.

---

Verification and testing

Grant, attach, and watch expiry manually:

LEASE_ID=$(etcdctl lease grant 9 | awk '{print $2}')
etcdctl put /locks/payment-42 owner-1 --lease="$LEASE_ID"
etcdctl get /locks/payment-42

Simulate a stalled keepalive stream (GC pause):

etcdctl lease keep-alive "$LEASE_ID" &
KA_PID=$!
kill -STOP "$KA_PID"   # freeze the keepalive process — simulates a GC pause
sleep 12                # longer than the 9s TTL
etcdctl get /locks/payment-42   # expect empty output — key was deleted at TTL
kill -CONT "$KA_PID" 2>/dev/null; kill "$KA_PID" 2>/dev/null

Confirm automatic revocation and orphan detection:

etcdctl lease timetolive "$LEASE_ID" --keys
# Expected after expiry: "lease <id> already expired"

etcdctl lease list --keys
# Any lease with 0 attached keys and a shrinking TTL is a candidate orphan

Verify the fencing revision is monotonically increasing:

etcdctl put /locks/payment-42 owner-1 --lease="$LEASE_ID" -w json | grep '"revision"'
# Re-run after a second acquisition; the revision must be strictly greater

---

Failure scenarios and debugging

---

SRE / observability checklist

Emit or verify these signals before shipping etcd lease renewal to production:

1. etcd_lease_renewal_failures_total — Counter incremented on every failed KeepAlive/KeepAliveOnce call. Alert on rate > 3/min per lease-holding service.
2. lease_ttl_remaining_seconds — Gauge populated by periodically calling Lease.TimeToLive. A value trending toward zero without a corresponding renewal log entry indicates a stuck heartbeat loop.
3. etcd_lease_keepalive_stream_closed_total — Counter tagged by reason (context_cancelled | lease_expired | connection_error). Only context_cancelled should ever be non-zero during planned shutdowns.
4. Structured log fields on every lease event — lease_id, key, revision, ttl_s, event (granted | renewed | lost | revoked). Index on lease_id for incident triage.
5. etcd_server_has_leader and etcd_server_leader_changes_seen_total — cluster-level gauges/counters; a leader change during your renewal window is a leading indicator of spurious lease expiry, not just an etcd operations concern.
6. OTel span lease.heartbeat — attach lease.id, lease.ttl_s, and lease.fencing_revision to the span covering the protected critical section so a lost lease can be correlated with the exact operation it interrupted.

---

Related

• Lock Timeout & Lease Management — parent page covering TTL sizing, renewal cadence, and fencing token guarantees across backends.
• Handling Stale Locks in Distributed Systems — the resource-side fencing check pattern applied to Redis, Postgres, etcd, and DynamoDB.
• Raft-Backed Exactly-Once Deduplication Logs — how etcd’s underlying raft log provides the ordering guarantee that makes the revision counter safe to use as a fencing token.
• Mitigating Thundering Herd During Retry Storms — what happens downstream when a lease expiry triggers mass client retries against the same key.
• Implementing Exponential Backoff Without Overlapping Retries — backoff strategy for the re-acquisition attempts your onLost handler triggers.