Optimistic vs Pessimistic Concurrency Control

Part of: Distributed Coordination & Locking Strategies

Every concurrent write path faces the same underlying race: two callers read the same piece of state, both decide what to do next based on that read, and both attempt to commit a change. If nothing intervenes between the read and the write, the second commit can silently overwrite the first — a lost update. Pessimistic and optimistic concurrency control are the two structurally different answers to this check-then-act race, and choosing between them is one of the highest-leverage decisions in a system that mutates shared state under load.

Pessimistic concurrency control closes the race by acquiring an exclusive lock before any work happens: every other writer is forced to wait, so at the moment of mutation there is provably only one active writer. Optimistic concurrency control takes the opposite bet — it assumes conflicts are rare, lets every writer proceed unblocked, and detects a conflict only at commit time by checking whether the state changed since it was read. The loser is rejected and retries. Both approaches produce the same correctness guarantee (no lost updates); they differ entirely in when the serialisation point occurs and what it costs under contention.

This page is decision-oriented: it gives you the guarantee model for each approach, a side-by-side view of the control flow, four production implementation variants with code, and the failure modes each approach introduces under load. For the mechanics of acquiring and releasing a distributed mutex itself, see distributed lock acquisition patterns; for bounding how long a pessimistic lock may be held, see lock timeout and lease management.

---

Problem Framing: The Check-Then-Act Race

Consider a single database row representing an account balance, an inventory count, or an idempotency record. A naive read-modify-write cycle looks like:

1. SELECT balance FROM accounts WHERE id = 42;   -- read: balance = 100
2. compute new_balance = balance - 30;            -- application logic
3. UPDATE accounts SET balance = new_balance WHERE id = 42;  -- write

If two threads execute this sequence concurrently, both read balance = 100 at step 1, both compute new_balance = 70 at step 2, and the second UPDATE at step 3 silently overwrites the first — the debit that thread A applied is lost. Nothing in this sequence is atomic across the three steps; the database only guarantees atomicity of the individual SELECT and the individual UPDATE, not the read-modify-write cycle as a whole.

Both concurrency control strategies exist to close this gap. They differ in where they insert the serialisation:

• Pessimistic: insert a lock acquisition before step 1, so no other thread can even begin its read until the first thread finishes its write and releases.
• Optimistic: leave steps 1-2 unlocked, but make step 3 conditional on the state observed in step 1 being unchanged — if it changed, reject and retry.

The choice is not aesthetic. It is a direct trade against contention level, conflict probability, and latency budget. High contention with a narrow latency budget favours pessimistic locking, because retry storms under optimistic concurrency waste more wall-clock time than a bounded wait would. Low conflict probability with abundant retry headroom favours optimistic concurrency, because it avoids paying the lock-acquisition cost on every single request, most of which would never have conflicted anyway.

---

Guarantee Model

Pessimistic locking guarantee

A correctly implemented pessimistic lock provides strict serialisability at the resource level: at most one transaction holds the lock on a given key at any instant, and every other transaction attempting to acquire the same lock blocks (or fails fast) until it is released. The guarantee holds as long as:

• Every code path that touches the resource acquires the lock first — a single unlocked writer defeats the entire guarantee.
• The lock is scoped to exactly the resource being protected, not a coarser or unrelated key (over-broad locking causes unnecessary blocking; under-broad locking leaves a gap).
• The lock is released (or expires via a bounded TTL) even when the holder crashes mid-operation — see lock timeout and lease management for TTL sizing.

The guarantee degrades into a lock convoy: as concurrency rises, waiting transactions queue behind the lock holder, and total system throughput collapses toward the inverse of the lock hold time, regardless of how many CPU cores are available.

Optimistic concurrency guarantee

Optimistic concurrency control provides conflict-serialisable commits with client-visible failure: a write only succeeds if the version (or full row state) observed at read time is unchanged at commit time. The guarantee holds as long as:

• The version check and the write happen in a single atomic database operation (UPDATE ... WHERE version = ?, a Lua script, or a native ConditionExpression) — never a separate read-then-write pair.
• Every writer checks the return value (0 rows affected, ConditionalCheckFailedException, or a WATCH abort) and treats it as a hard conflict, not a silent no-op.
• A retry policy with bounded attempts and backoff exists, so a rejected writer does not simply fail the request.

The guarantee degrades into a retry storm: under high contention, most attempts conflict, each conflict triggers a retry, and the retries themselves increase contention further — a self-reinforcing spiral that can saturate the database with the same volume of failed writes it was trying to avoid locking for.

---

Control Flow Comparison

The diagram below contrasts the two control flows side by side for the same logical operation — updating an inventory count from two concurrent callers.

The left path serialises before any work begins — caller B cannot even start reading until caller A releases. The right path lets both callers work concurrently and only serialises at the final conditional write; the loser discovers the conflict after having done (and discarded) the work.

---

Implementation Variants

Variant 1 — Pessimistic: SELECT ... FOR UPDATE

Row-level locking in PostgreSQL and MySQL acquires an exclusive lock on the selected row for the duration of the transaction. Any other transaction attempting SELECT ... FOR UPDATE (or an UPDATE) on the same row blocks until the first transaction commits or rolls back.

BEGIN;

-- Acquire an exclusive row lock; blocks other FOR UPDATE / UPDATE on this row
SELECT balance FROM accounts WHERE id = 42 FOR UPDATE;

-- Application computes new_balance = balance - 30 here

UPDATE accounts SET balance = 70 WHERE id = 42;

COMMIT;  -- releases the lock

Set a statement timeout so a blocked transaction fails fast instead of queuing indefinitely behind a stalled holder:

SET LOCAL lock_timeout = '2000ms';  -- abort with an error after 2 seconds of waiting

This is the correct choice when the operation inside the lock is cheap and deterministic (a single row mutation), and when contention on the same row is expected to be frequent — for example, a hot inventory counter during a flash sale.

Variant 2 — Optimistic: Version-Column CAS

A version column turns the write into a compare-and-swap. The UPDATE only succeeds if the version matches what was read; a mismatch means someone else committed first.

-- Schema
ALTER TABLE accounts ADD COLUMN version INTEGER NOT NULL DEFAULT 0;

-- Read
SELECT balance, version FROM accounts WHERE id = 42;
-- balance = 100, version = 7

-- Conditional write: only succeeds if version is still 7
UPDATE accounts
SET    balance = 70,
       version = version + 1
WHERE  id = 42
  AND  version = 7;
-- 0 rows affected → conflict, re-read and retry
-- 1 row affected  → success

No lock is held between the read and the write — any number of readers can proceed concurrently, and only the write itself is atomic. This is the pattern covered in full runbook detail on implementing optimistic concurrency with version columns, including JPA, SQLAlchemy, and retry-with-backoff code.

Variant 3 — Optimistic: Redis WATCH / MULTI / EXEC

Redis provides optimistic locking natively: WATCH marks a key for conflict detection, and the subsequent MULTI/EXEC transaction aborts (returns nil) if any watched key changed between the WATCH and the EXEC.

import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def transfer(from_key: str, to_key: str, amount: int, max_retries: int = 5) -> bool:
    for attempt in range(max_retries):
        with r.pipeline() as pipe:
            try:
                pipe.watch(from_key)
                balance = int(pipe.get(from_key) or 0)
                if balance < amount:
                    pipe.unwatch()
                    return False
                pipe.multi()
                pipe.decrby(from_key, amount)
                pipe.incrby(to_key, amount)
                pipe.execute()  # raises WatchError if from_key changed since WATCH
                return True
            except redis.WatchError:
                continue  # someone else modified from_key; retry with fresh read
    return False  # exhausted retries — treat as a conflict, surface to caller

WATCH is cheap (no server-side lock object is created) but every retry re-executes the full read-compute-write cycle client-side, so it shares the retry-storm risk of any optimistic strategy under high contention.

Variant 4 — Optimistic: DynamoDB Conditional Writes

DynamoDB has no row-locking primitive; all concurrency control is optimistic, enforced via ConditionExpression evaluated atomically against the item at write time.

import boto3
from botocore.exceptions import ClientError

table = boto3.resource("dynamodb").Table("accounts")

def debit(account_id: str, amount: int, expected_version: int) -> bool:
    try:
        table.update_item(
            Key={"account_id": account_id},
            UpdateExpression="SET balance = balance - :amt, version = version + :one",
            ConditionExpression="version = :expected",
            ExpressionAttributeValues={
                ":amt": amount,
                ":one": 1,
                ":expected": expected_version,
            },
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False  # conflict — re-read version and retry
        raise

See DynamoDB conditional writes for deduplication for the equivalent pattern applied to idempotency-key state rather than account balances — the mechanism is identical.

Comparison Summary

---

Edge Cases and Failure Scenarios

---

Operational Concerns

Lock timeouts (pessimistic). Set lock_timeout (PostgreSQL) or innodb_lock_wait_timeout (MySQL) to 2 seconds for interactive request paths; reserve longer waits (10-30 seconds) only for background batch jobs where queuing is acceptable. Never leave the default (which can be unbounded) in a request-serving path.

Indexing (both). The row or key used as the lock/version target must be indexed — an unindexed WHERE id = ? forces a full table scan under the lock, extending hold time and worsening convoy risk. For version-column CAS, a composite index on (id, version) lets the database verify the predicate without a second lookup.

Memory and connection pool budgeting. Pessimistic locks held across a network round-trip (e.g. while your application computes the new value) hold a database connection idle for the same duration. Under 500 concurrent lock holders with a 50 ms average hold time, that is 25 connection-seconds per second of load — size the connection pool accordingly, or move computation outside the transaction to shrink hold time.

SRE alert thresholds. Instrument and alert on:

• lock_wait_time_p99_ms — alert if > 200 ms over a 5-minute window (pessimistic convoy signal)
• optimistic_conflict_rate — alert if > 15% of writes conflict over 60 seconds (retry storm signal)
• retry_attempts_p99 — alert if the 99th percentile of retries per logical write exceeds 3
• db_deadlocks_total — alert on any non-zero rate; deadlocks under pessimistic locking indicate inconsistent lock-acquisition ordering across code paths

Emit the version (or lock token) on every structured log line for the operation so incident review can reconstruct exactly which write won and which retried.

---

Related

• Distributed Coordination & Locking Strategies — parent section covering the full coordination problem space including consensus, lease management, and lock acquisition.
• Implementing Optimistic Concurrency with Version Columns — step-by-step runbook for adding a version column, handling conflicts, and retrying with backoff in Java, Python, and raw SQL.
• Distributed Lock Acquisition Patterns — blocking, polling, and event-driven strategies for acquiring the pessimistic locks described above.
• Lock Timeout & Lease Management — bounding pessimistic lock lifetimes so a crashed holder cannot stall competing writers indefinitely.
• Preventing Race Conditions in Microservices — applying optimistic conditional writes and thundering-herd mitigation across service boundaries rather than within a single database.
• Database Unique Constraints & Upserts — the storage-layer conditional-write primitives (INSERT ... ON CONFLICT) that underpin optimistic concurrency at the persistence layer.
• DynamoDB Conditional Writes for Deduplication — ConditionExpression-based optimistic concurrency for a lock-server-free, globally distributed deployment.