Lock Timeout & Lease Management

Part of: Distributed Coordination & Locking Strategies

Unbounded locks in distributed systems are a silent reliability threat: a node that crashes, stalls in a GC pause, or loses its network path while holding a lock can block every competing worker indefinitely. Lock timeout and lease management addresses this by making ownership time-bounded, renewable, and cryptographically fenced — so expiry is safe, handoffs are clean, and stale holders cannot corrupt shared state.

---

Guarantee Model

A well-configured lease provides linearizable single-owner semantics within a bounded time window, with the following precise contract:

• At most one node holds the lock at any instant (mutual exclusion).
• Ownership transfers to a new holder at most TTL seconds after the current holder’s last successful heartbeat.
• Any write by a node that has lost the lease is rejected by downstream services via fencing token validation.

This guarantee breaks under two conditions:

1. Clock skew exceeding the safety margin. If a node’s wall clock runs fast by more than the safety margin baked into the TTL, it may believe the lease is still valid after the coordinator has already granted it to a successor.
2. Asymmetric network partitions where the coordination backend is reachable but downstream services are not. The holder can renew successfully yet fail to commit writes; a new holder acquires the lease and commits, then the old holder’s writes arrive out of order.

Mitigating both requires coordinator-relative timestamps (not wall-clock TTLs) and fencing token validation on every downstream write — covered in detail below.

---

Lease Lifecycle: State Diagram

The diagram below shows the full lifecycle of a lease from initial acquisition through renewal, expiry, and forced revocation.

---

Core Algorithm: TTL Calculation and Fencing Tokens

Step 1 — Calculate a Safe TTL

The TTL must cover worst-case execution time, not average-case. Use:

TTL_base = P99_work_latency_ms + P99_network_RTT_ms + safety_margin_ms

Concrete example for a payment-processing worker with P99 work latency of 800 ms, P99 network RTT of 40 ms, and a 200 ms safety margin:

TTL_base = 800 + 40 + 200 = 1040 ms  →  round up to 1500 ms

For geographically distributed deployments add a clock-drift buffer of 50–200 ms on top to account for NTP step-slew events. Never derive TTLs from arbitrary constants — recalculate them whenever the P99 latency budget changes.

Step 2 — Acquire with Backoff and Jitter

Distributed lock acquisition patterns fall into three categories: blocking (thread holds until granted), polling (client retry loop), and event-driven (watch/subscribe to availability notifications). For high-contention workloads, event-driven acquisition via etcd Watch or Redis keyspace notifications minimises idle CPU. Whichever strategy you use, the retry loop must apply exponential backoff with jitter:

backoff_ms = min(max_delay_ms, base_delay_ms × 2^attempt + random_ms(0, jitter_window_ms))

A practical starting point: base_delay = 50 ms, max_delay = 2000 ms, jitter_window = 100 ms.

Step 3 — Attach a Fencing Token

Every lock grant from the coordinator must include a monotonically increasing integer — the fencing token (also called an epoch counter or generation number). The token increments on every grant, not on every renewal. Downstream services must store the highest token they have seen and reject any write accompanied by a lower token:

-- PostgreSQL: atomic conditional write that enforces fencing
UPDATE resource_state
SET    payload        = $new_payload,
       fencing_token  = $token,
       updated_at     = now()
WHERE  resource_id    = $id
  AND  fencing_token  < $token;
-- 0 rows updated → reject, caller has lost the lease

This prevents the “slow writer” scenario: a node that survived a GC pause wakes up, believes it still holds the lease, and attempts to write — but the coordinator has already granted the lease (with a higher token) to a new holder.

Step 4 — Renew at TTL ÷ 3

Set the background renewal interval to TTL ÷ 3. This leaves two full retry windows before the lease expires even if one heartbeat is lost. Add ±15% jitter to prevent all workers from firing heartbeats in the same millisecond window:

// Go: lease renewal loop with jitter
func renewLoop(ctx context.Context, client LeaseClient, leaseID int64, ttlSec int64) {
    base := time.Duration(ttlSec/3) * time.Second
    for {
        jitter := time.Duration(rand.Int63n(int64(base/5))) - base/10 // ±10%
        select {
        case <-time.After(base + jitter):
            if err := client.KeepAlive(ctx, leaseID); err != nil {
                log.Error("renewal failed, entering drain", "err", err)
                return // caller transitions to DRAINING
            }
        case <-ctx.Done():
            return
        }
    }
}

---

Implementation Variants

Redis SET NX PX

Redis offers millisecond-granularity TTLs via SET key value NX PX ttl_ms. Renewal requires a Lua script to make the check-and-extend atomic:

-- Lua: atomic renewal (only extend if this node still owns it)
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("pexpire", KEYS[1], ARGV[2])
else
    return 0
end

Fencing tokens must be stored separately (e.g. in a sorted set or a dedicated counter key) because SET NX PX does not natively expose a monotonic grant counter. See using Redis SET NX for distributed request deduplication for the full key-layout pattern.

etcd Leases with gRPC KeepAlive

etcd leases are first-class objects: LeaseGrant returns a lease ID and the server tracks expiry server-side. The client calls LeaseKeepAlive over a persistent gRPC stream. The revision number returned by etcd Put serves directly as the fencing token — no additional counter is needed:

// Go: acquire an etcd lease and extract the revision as fencing token
resp, err := client.Grant(ctx, ttlSec)
// resp.ID is the lease handle; attach it to all subsequent Puts
putResp, err := client.Put(ctx, "/locks/order-123", nodeID,
    clientv3.WithLease(resp.ID))
fencingToken := putResp.Header.Revision // monotonically increasing

Connection loss terminates the KeepAlive stream, causing the coordinator to revoke the lease at TTL expiry without any explicit release call.

PostgreSQL Advisory Locks with Application Heartbeat

PostgreSQL session-scoped advisory locks (pg_try_advisory_lock) tie the lock to a database connection. If the connection drops, the lock is automatically released. For long-running operations that span multiple connection checkouts, application-level heartbeat rows in a distributed_leases table are more robust:

-- Schema: lease table with fencing token
CREATE TABLE distributed_leases (
    resource_id      TEXT PRIMARY KEY,
    holder_id        TEXT        NOT NULL,
    fencing_token    BIGINT      NOT NULL DEFAULT 0,
    expires_at       TIMESTAMPTZ NOT NULL,
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Acquire: atomic upsert that only wins if resource is free or expired
INSERT INTO distributed_leases (resource_id, holder_id, fencing_token, expires_at)
VALUES ($1, $2, 1, now() + ($3 || ' ms')::interval)
ON CONFLICT (resource_id) DO UPDATE
    SET holder_id     = EXCLUDED.holder_id,
        fencing_token = distributed_leases.fencing_token + 1,
        expires_at    = EXCLUDED.expires_at,
        updated_at    = now()
WHERE distributed_leases.expires_at < now();

-- Renew: extend only if this node still owns it
UPDATE distributed_leases
SET expires_at = now() + ($1 || ' ms')::interval,
    updated_at = now()
WHERE resource_id  = $2
  AND holder_id    = $3
  AND expires_at   > now();

This approach integrates cleanly with idempotency key storage and TTL management — both the lease record and the idempotency record can live in the same transaction, giving exactly-once semantics without a second round-trip.

Summary Comparison

---

Edge Cases and Failure Scenarios

---

Idempotency Key Coupling and Deduplication Windows

Lease boundaries must align with idempotency deduplication windows. The dedup window must span TTL + P99_processing_latency to cover the gap between lease expiry and final commit. When a request arrives, the flow is:

1. Check idempotency store for an existing key.
2. If the key exists and the associated fencing token matches the current lease token, return the cached response.
3. If the key exists but the token is stale (previous holder’s epoch), verify whether the operation completed before allowing re-acquisition.
4. If the key is absent, acquire the lease, execute, write the result with the fencing token, then release.

This coupling ensures that preventing race conditions in microservices relies on deterministic fenced state rather than probabilistic retry windows. For fintech workflows where exactly-once semantics are mandatory, pair the lease with a transactional outbox pattern so that the lease record and the outbox event are written in a single database transaction.

---

Operational Concerns

TTL Management

Re-derive the TTL every time the P99 work latency budget changes. Treat the TTL as a service-level parameter — store it in a config map or feature flag so it can be adjusted without a deployment. Document the derivation formula (P99_work + P99_network + safety_margin + drift_buffer) in the runbook so on-call engineers know which metrics to check when adjusting it.

Memory and Storage Budgeting

For Redis-backed leases, each lease key is typically 60–120 bytes of memory. At 10,000 concurrent leases that is at most 1.2 MB — negligible. For PostgreSQL lease tables, add a partial index on expires_at to support efficient expiry-based scans:

CREATE INDEX idx_leases_expires ON distributed_leases (expires_at)
WHERE expires_at > now();

Run a background cleanup job every 60 seconds to purge expired rows and keep the table small:

DELETE FROM distributed_leases WHERE expires_at < now() - INTERVAL '5 minutes';

SRE Alert Thresholds

Instrument the following metrics and wire them to your alerting layer:

• lease_acquisition_latency_p99_ms — alert if > 200 ms over a 5-minute window
• lease_renewal_failure_rate — alert if > 1% of renewals fail over 60 seconds
• lease_expiry_forced_total — alert if > 0 in a 1-minute window (indicates holder crashed or lost connectivity)
• fencing_token_increment_rate — sudden spikes indicate rapid ownership churn; alert on > 10× baseline
• stale_write_rejected_total — any non-zero count is a signal that a holder lost its lease mid-operation; investigate immediately

Emit all lease state transitions as structured log events with lease_id, holder_id, fencing_token, and ttl_ms fields so distributed traces can reconstruct the full ownership timeline during incident review.

Graceful Preemption and Worker Draining

Background workers must intercept SIGTERM and enter the DRAINING state immediately: stop accepting new work, complete in-flight operations within a configurable drain timeout (typically TTL × 2), checkpoint intermediate state, then release the lease. If the drain timeout is exceeded, force-release the lease and let the compensation layer handle reconciliation. Never allow a worker to exit without explicitly releasing or letting the TTL expire — orphaned leases block successors for up to one full TTL.

Post-failure reconciliation patterns — including audit trail generation and compensating transaction rollbacks — are covered in depth on handling stale locks in distributed systems.

---

Related

• Distributed Coordination & Locking Strategies — parent section covering the full coordination problem space including consensus, deadlock prevention, and lock acquisition
• Distributed Lock Acquisition Patterns — blocking vs polling vs event-driven acquisition strategies and how contention models affect timeout configuration
• Implementing Redlock for High-Availability Deduplication — multi-node quorum-based locking to survive coordinator partial failures
• Handling Stale Locks in Distributed Systems — recovery protocols, audit trails, and compensating transactions after lease expiry
• Preventing Race Conditions in Microservices — how lease fencing integrates with broader race-condition prevention patterns across service boundaries
• Retry Logic & Backoff Fundamentals — exponential backoff with jitter to safely retry failed acquisition attempts without thundering-herd effects