Part of: HTTP Method Semantics & Safety
GET and HEAD are the only two HTTP methods that RFC 9110 classifies as both safe and idempotent — a dual guarantee that makes them the foundation of reliable read paths in distributed systems. Understanding precisely what that guarantee covers, and where real infrastructure can fracture it, lets you write retry logic, configure proxies, and set observability thresholds with confidence rather than guesswork.
Prerequisites: You should understand the broad contract around HTTP Method Semantics & Safety and know why Idempotency Fundamentals & API Guarantees matter before working through the runbooks below.
What the RFC Actually Guarantees
RFC 9110 defines two orthogonal properties:
- Safe: The method has no observable effect on the state of the origin server. Clients (and intermediaries) may issue safe requests freely without worrying about unintended mutations.
- Idempotent: Issuing the same request multiple times produces the same effect as issuing it once. For
GETandHEAD, this collapses onto safety — because neither method changes state at all, the state after one request equals the state after n requests.
POST is neither safe nor idempotent. PUT and DELETE are idempotent but not safe. GET and HEAD sit alone in the safe-and-idempotent quadrant:
The guarantee is protocol-level only. RFC 9110 binds the origin server — it says nothing about CDN edge caches, audit logging pipelines, stateful load balancers, or read-replica lag. Production reliability requires you to preserve the spirit of the guarantee at every layer.
Step-by-Step: Confirming Your Read Path Honours the Guarantee
Work through these steps in order. Each step is independently verifiable.
Step 1 — Assert that your GET handlers contain zero state-mutating calls
Audit every GET route handler for database writes, cache invalidations, or enqueued side effects. A common violation is analytics event emission inside the handler:
# Python / FastAPI — WRONG: GET handler emitting a tracking event inline
@app.get("/products/{product_id}")
def get_product(product_id: str):
db.write_event("product_viewed", product_id) # violates safe-method contract
return db.fetch_product(product_id)
# Python / FastAPI — CORRECT: emit event asynchronously via a background task
from fastapi import BackgroundTasks
@app.get("/products/{product_id}")
def get_product(product_id: str, background_tasks: BackgroundTasks):
background_tasks.add_task(emit_event, "product_viewed", product_id)
return db.fetch_product(product_id)
// Go / net/http — CORRECT: defer analytics to a goroutine so the handler is read-only
func GetProduct(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "productID")
go analyticsQueue.Enqueue("product_viewed", id) // non-blocking, no return value used
product := db.FetchProduct(id)
json.NewEncoder(w).Encode(product)
}
Verify: Run grep -rn "db.write\|INSERT\|UPDATE\|DELETE\|cache.del" ./handlers/get_*.go (adapt for your language). The output should be empty.
Step 2 — Confirm Cache-Control directives are correct for the sensitivity level
Sensitive endpoints (financial account data, compliance records, PII) must bypass shared caches. Non-sensitive public data should use appropriate cache directives to avoid CDN stampedes.
# Bash — check Cache-Control on a live endpoint
curl -sI https://api.example.com/accounts/42 | grep -i cache-control
# Expected for sensitive data:
# cache-control: no-store, max-age=0
# Python — set correct Cache-Control in a Django view
from django.views.decorators.cache import never_cache
@never_cache
def account_detail(request, account_id):
return JsonResponse(get_account(account_id))
// Go — set Cache-Control header before writing the response
func AccountDetail(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "no-store, max-age=0")
id := chi.URLParam(r, "accountID")
json.NewEncoder(w).Encode(db.GetAccount(id))
}
Verify: Repeat the curl -sI check from two different geographic regions (use a VPN or a multi-region testing tool). Both responses must return identical Cache-Control headers.
Step 3 — Validate ETag consistency across retries
ETag divergence across consecutive GET retries to the same URL is a reliable signal that either (a) your read replicas are returning stale data, or (b) your CDN is constructing responses from inconsistent origin shards.
# Bash — fire 3 consecutive GETs and compare ETags
for i in 1 2 3; do
curl -sI https://api.example.com/orders/99 | grep -i etag
done
# All three lines must be identical if the resource has not changed.
# Python — programmatic ETag comparison across retries
import httpx
url = "https://api.example.com/orders/99"
responses = [httpx.head(url) for _ in range(3)]
etags = [r.headers.get("etag") for r in responses]
assert len(set(etags)) == 1, f"ETag divergence detected: {etags}"
If the assertion fails, proceed to the replica lag check in Step 4.
Step 4 — Detect replica lag during database failover
Read replicas return stale data during primary promotion windows. Add an explicit lag check before serving reads during topology changes.
-- PostgreSQL: reject reads if replica lag exceeds 3 seconds
SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp())) AS lag_seconds;
-- If lag_seconds > 3, route the request to the primary or return 503.
-- MySQL 8.0.22+ equivalent
SHOW REPLICA STATUS\G
-- Check Seconds_Behind_Source; abort if > 3.
# Python — replica lag guard in a repository layer
def get_with_replica_guard(query_fn, max_lag_seconds: float = 3.0):
lag = db_replica.scalar("SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))")
if lag is None or lag > max_lag_seconds:
return db_primary.execute(query_fn)
return db_replica.execute(query_fn)
Verify: Simulate a failover in a staging environment, fire 10 consecutive GET requests during the promotion window, and confirm that the lag guard routes all requests to the primary when lag_seconds > 3.
Step 5 — Strip injected headers at the proxy layer
Load balancers and API gateways often inject Set-Cookie, session tokens, or internal routing headers. Downstream cache layers may key on these, causing GET requests that should be cache-equivalent to miss:
# Nginx — strip injected correlation headers before routing to origin
location /api/ {
proxy_pass http://upstream;
# Remove headers that would pollute cache keys or appear as state mutations
proxy_set_header X-Session-Id "";
proxy_set_header X-AB-Variant "";
proxy_cache_key "$scheme$request_method$host$request_uri";
proxy_cache_lock on;
}
# Envoy — header mutation filter to strip tracking headers on GET/HEAD
http_filters:
- name: envoy.filters.http.lua
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.LuaPerRoute
inline_code: |
function envoy_on_request(request_handle)
if request_handle:headers():get(":method") == "GET" or
request_handle:headers():get(":method") == "HEAD" then
request_handle:headers():remove("x-session-id")
request_handle:headers():remove("x-ab-variant")
end
end
Verify: Use curl -v to inspect response headers before and after deploying the filter. Confirm neither X-Session-Id nor X-AB-Variant reaches the origin access logs.
Verification & Testing
After completing the steps above, run this checklist to confirm the read path is clean end-to-end:
- Idempotency smoke test. Fire the same
GETrequest 5 times within 500 ms. Compare response bodies with a hash (sha256sum). All hashes must match (assuming the resource has not changed). - Cache stampede simulation. Expire the CDN cache manually (
curl -X PURGE) and fire 20 concurrentGETrequests. Confirm origin receives exactly 1 request (all others are held behindproxy_cache_lock). - Replica lag injection. Use
tc netemto introduce 200 ms delay on the replica network interface during a test. Confirm the lag guard routes traffic to the primary. - Header strip confirmation. Replay a captured request with injected tracking headers through the proxy and verify the origin access log shows clean requests.
Failure Scenarios & Debugging
| Failure Scenario | Remediation Steps | Observability Hooks |
|---|---|---|
| CDN cache stampede: TTL expiry triggers simultaneous origin fetches, overwhelming DB read capacity | Enable proxy_cache_lock on in Nginx or collapse_buffering in Varnish; set stale-while-revalidate on non-sensitive endpoints |
nginx_upstream_requests_total spike; origin_latency_p99 > 500ms alert |
| Read replica lag returns stale data during primary promotion | Add per-request lag check (pg_last_xact_replay_timestamp); failover reads to primary if lag > 3s |
replication_lag_seconds gauge; stale_read_routed_to_primary_total counter |
| Stateful proxy injects session cookie that breaks cache key equivalence | Strip session/tracking headers at edge via proxy rewrite rule before proxy_cache_key evaluation |
cache_hit_ratio drops below 0.7 per endpoint; cache_bypass_reason=header_mismatch log field |
| Kubernetes readiness probe hits endpoint with synchronous DB write | Refactor probe target to a dedicated /healthz endpoint that performs only a connection ping |
kubelet_probe_success_total{probe_type="readiness"} below threshold; probe_latency_p99 > 200ms |
Envoy retry on retriable-4xx dispatches duplicate GET to origin |
Set retry_policy.num_retries: 0 for GET/HEAD unless origin explicitly validates dedup; use retriable-headers to restrict retry conditions |
envoy_cluster_upstream_rq_retry_total{method="GET"} non-zero alert |
SRE / Observability Checklist
Instrument these specific signals for every production GET/HEAD path:
get_etag_divergence_total— counter incremented when two consecutiveGETresponses for the same URL return differentETagvalues within a 100 ms window; page on any non-zero value.cache_hit_ratioper endpoint — gauge; alert when ratio drops below 0.65 (indicates cache bypass or stampede).replication_lag_seconds— gauge sourced frompg_last_xact_replay_timestamp(); alert at> 3.0during any active read workload.envoy_cluster_upstream_rq_retry_total{method="GET"}— counter; alert on non-zero rate to catch unintended retry amplification.- Trace span attribute
cache.hit: bool— tag everyGETspan; aggregate in dashboards to detect per-service cache regression. upstream_latency_p99per CDN PoP — alert whenp99 > 250msacross more than 2 PoPs simultaneously, indicating a potential stampede in progress.
Related
- HTTP Method Semantics & Safety — the parent reference covering all HTTP methods, safety classifications, and when to require explicit idempotency keys.
- Retry Logic & Backoff Fundamentals — how to configure exponential backoff with jitter so
GETretry storms do not overwhelm origin capacity. - Webhook Delivery Guarantees — safe-method guarantees interact with outbound
GETverification requests in webhook flows; see handling duplicate webhook deliveries for the full pattern.