Injecting Clock Skew to Validate TTL Safety

Part of: Chaos Engineering for Idempotency

Clock skew is one of the hardest faults to reproduce and one of the most damaging to ignore. A single NTP step-adjustment on one node can silently invalidate every TTL and lease deadline computed from that node’s wall clock, causing renewal heartbeats to miss their window and leases to expire early — while the node itself believes nothing is wrong. This runbook deliberately skews a node’s clock, observes the resulting premature TTL and lease expiry, proves that fencing tokens prevent the skewed node from corrupting shared state, and remediates the root cause with monotonic clock sources.

---

Problem Statement and Prerequisites

What you are proving: that a node whose wall clock has been stepped forward cannot corrupt protected state, and that switching TTL/renewal math to a monotonic clock eliminates the premature expiry in the first place.

Prerequisites:

• You understand lock timeout and lease management — specifically how TTL is calculated and how renewal intervals are scheduled.
• Your lease implementation already returns a fencing token on every grant and your protected resource enforces it on every write.
• You can run the lease-holding process inside a container so you can scope the clock skew to that single process without affecting the coordination backend or the rest of the fleet.
• This is destructive to a live lease — run it in staging first, exactly as recommended in the chaos engineering methodology this runbook implements.

---

What Clock Skew Actually Breaks

---

Step-by-Step Implementation

Step 1 — Skew a lease-holding node’s clock with libfaketime

Run the lease-holding process inside a container with libfaketime preloaded so only that process’s time calls are affected — the coordination backend and every other node keep their real clock.

# Install libfaketime inside the container image, then run the lease-holding
# process with its wall clock stepped forward by 90 seconds at container start.
docker run --rm \
  -e LD_PRELOAD=/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1 \
  -e FAKETIME="+90s" \
  --name skewed-lease-holder \
  lease-worker:staging

# To step the clock again mid-run without restarting the container,
# exec in and rewrite the FAKETIME control file libfaketime reads from:
docker exec skewed-lease-holder sh -c 'echo "+90s" > /etc/faketime.rc'

For hosts where injecting LD_PRELOAD is not viable, step the container’s own clock directly (requires --cap-add SYS_TIME and a container-local clock, never the host clock):

docker run --rm --cap-add SYS_TIME --name skewed-lease-holder lease-worker:staging \
  sh -c 'date -s "+90 seconds" && exec /app/lease-worker'

Step 2 — Observe the renewal timer miss its window

Instrument the renewal loop to log both its own perceived deadline and the coordinator’s authoritative response, then watch the divergence appear the instant the fault is injected.

import time
import logging

logger = logging.getLogger("lease_renewal")

def renewal_deadline_wallclock(acquired_at_wallclock: float, ttl_seconds: float) -> float:
    """Vulnerable: derives the deadline from the node's own wall clock,
    which libfaketime or an NTP step can move without warning."""
    return acquired_at_wallclock + ttl_seconds

def check_and_renew(acquired_at_wallclock: float, ttl_seconds: float, renew_fn) -> None:
    now = time.time()  # wall clock — this is what the fault corrupts
    deadline = renewal_deadline_wallclock(acquired_at_wallclock, ttl_seconds)
    remaining = deadline - now
    logger.info("lease renewal check: remaining=%.2fs (wallclock now=%.2f)", remaining, now)
    if remaining < ttl_seconds / 3:
        renew_fn()
    if remaining < 0:
        logger.error("local clock believes lease already expired — was it skewed?")

Run the skewed container and confirm the log shows remaining jump negative immediately after the +90s step, even though only 90 seconds of wall-clock skew — not real elapsed time — has passed.

Step 3 — Prove the fencing token rejects the skewed node’s stale write

While Node A is skewed, force a competing node to acquire the lease, then attempt a write from the skewed node using its stale token.

# On a healthy second node/process, force-acquire the same resource
# after the coordinator has expired Node A's lease.
curl -s -X POST http://coordinator:8080/leases/order-lock/acquire \
  -d '{"holder":"node-b"}' | tee /tmp/node_b_grant.json
# Expect: {"token": 8, "ttl_ms": 10000}

# From the skewed Node A container, attempt the write it believes is still valid
docker exec skewed-lease-holder \
  curl -s -X POST http://resource-service:8080/orders/order-123/apply \
  -d '{"fencing_token": 7, "payload": "stale-write-from-node-a"}'
# Expected response: 409 Conflict {"error":"ErrStaleFencingToken","current_hwm":8}

-- Confirm at the database layer that the stale token was never applied
SELECT fencing_token, updated_at FROM order_state WHERE order_id = 'order-123';
-- Expected: fencing_token = 8, not 7 — Node A's write never landed

Step 4 — Remediate by deriving deadlines from a monotonic clock

Replace every wall-clock-based deadline calculation with a monotonic clock source (time.monotonic() in Python, CLOCK_MONOTONIC via runtime.nanotime()-style APIs in Go). Monotonic clocks never jump backward or forward due to NTP steps — they only ever move forward at a steady rate.

import time

class LeaseTimer:
    """Fixed: uses a monotonic clock so NTP steps and libfaketime injections
    on the wall clock cannot affect the renewal deadline calculation."""
    def __init__(self, ttl_seconds: float):
        self._acquired_at_monotonic = time.monotonic()
        self._ttl_seconds = ttl_seconds

    def remaining(self) -> float:
        elapsed = time.monotonic() - self._acquired_at_monotonic
        return self._ttl_seconds - elapsed

    def should_renew(self) -> bool:
        return self.remaining() < self._ttl_seconds / 3

package lease

import "time"

// Timer tracks lease expiry using time.Since, which is backed by the
// Go runtime's monotonic clock reading whenever the initial time.Time
// was captured with time.Now() — immune to wall-clock steps from NTP.
type Timer struct {
	acquiredAt time.Time
	ttl        time.Duration
}

func NewTimer(ttl time.Duration) *Timer {
	return &Timer{acquiredAt: time.Now(), ttl: ttl}
}

func (t *Timer) Remaining() time.Duration {
	elapsed := time.Since(t.acquiredAt) // monotonic-safe subtraction
	return t.ttl - elapsed
}

func (t *Timer) ShouldRenew() bool {
	return t.Remaining() < t.ttl/3
}

Note that Go’s time.Time retains a monotonic reading internally as long as it was created via time.Now(), and time.Since/subtraction between two such values automatically uses it — never serialize a time.Time to a wall-clock string and reconstruct it for deadline math, since that discards the monotonic component.

Step 5 — Verify the fix by re-running the skew injection

Re-run the exact fault from Step 1 against a build using the monotonic timer and confirm the renewal fires on schedule regardless of the wall-clock step.

docker run --rm \
  -e LD_PRELOAD=/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1 \
  -e FAKETIME="+90s" \
  --name monotonic-lease-holder \
  lease-worker:staging-monotonic-fix

docker logs -f monotonic-lease-holder | grep "lease renewal check"
# Expected: remaining stays consistent with real elapsed time; renewal
# fires at TTL/3 as scheduled, unaffected by the +90s wall-clock step.

---

Verification and Testing

# 1. Confirm no stale write reached the resource during the entire skew window
psql -c "SELECT COUNT(*) FROM order_state_audit WHERE fencing_token = 7 AND order_id = 'order-123';"
# expect: 0

# 2. Confirm the coordinator's fencing token high-water mark only ever increased
curl -s http://coordinator:8080/leases/order-lock/history | jq '.tokens'
# expect: strictly increasing sequence, e.g. [7, 8]

# 3. Confirm the monotonic-fixed build's renewal fired within the expected window
grep "renewal fired" monotonic-lease-holder.log | tail -1
# expect: a timestamp within TTL/3 (±15% jitter) of lease acquisition, not skewed by +90s

---

Failure Scenarios and Debugging

---

SRE / Observability Checklist

1. ntp_offset_seconds — Gauge per node from chronyc tracking or timedatectl. Alert if offset exceeds 50 milliseconds for any lease-holding node.
2. lease_clock_skew_detected_total — Counter incremented whenever a renewal check computes a remaining value inconsistent with real elapsed time (compare against a monotonic side-channel measurement).
3. stale_fencing_token_rejections_total — Counter on the resource side. During a deliberate skew test, this metric moving above zero is the expected, correct outcome — the opposite of most alerts.
4. lease_renewal_latency_ms — Histogram of time between scheduled and actual renewal fire time; skew or GC pressure both show up here as a fat tail.
5. Trace span lease.renew — Attach wallclock_now, monotonic_elapsed_ms, and ttl_ms attributes so a post-incident trace query can reconstruct exactly what each clock believed at the moment of expiry.
6. coordinator_ntp_offset_seconds — Separate gauge for the coordination backend itself; a skewed coordinator invalidates every lease it manages, not just one node’s.

---

Related

• Chaos Engineering for Idempotency — parent page covering the hypothesis-injection-observation loop and the full fault taxonomy this runbook exercises the clock-skew fault from
• Chaos Testing Duplicate Request Handling — companion runbook for the duplicate-delivery fault, using concurrent request bursts instead of clock manipulation
• Handling Stale Locks in Distributed Systems — the fencing token mechanics this runbook proves out under a live clock-skew fault
• Lock Timeout & Lease Management — TTL sizing and renewal interval formulas that must account for clock-drift budgets
• Implementing Redlock for High-Availability Deduplication — quorum-based locking whose skew resistance depends on testing every node in the quorum, not just one