Propagating Idempotency Keys Through Trace Context

Part of: Distributed Tracing for Deduplication

Once a request enters a multi-hop call graph — API gateway to order service to payment service — the idempotency key needs to travel with it so every hop’s dedup.check span can be correlated under the same business operation, even though each hop’s traceparent changes at every boundary. This runbook shows how to carry the key in OpenTelemetry baggage across Go and Java services, why the key must never be stuffed into traceparent itself, and how to verify the propagation actually reached the last hop.

Problem statement and prerequisites

What you are implementing: an OTel Baggage entry carrying the idempotency key hash, propagated via HTTP headers across every service hop, read and re-attached as a span attribute at each hop.

Prerequisites:

Step-by-step implementation

Step 1 — Extract incoming trace context and baggage (Go, API Gateway)

// Go: extract incoming context, including any propagated baggage
package gateway

import (
	"net/http"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"
)

var propagator = propagation.NewCompositeTextMapPropagator(
	propagation.TraceContext{},
	propagation.Baggage{},
)

func extractContext(r *http.Request) context.Context {
	return propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
}

Step 2 — Inject the idempotency key into baggage, not traceparent

// Go: add the idempotency key hash as a baggage member on the first hop
import (
	"context"
	"crypto/sha256"
	"encoding/hex"

	"go.opentelemetry.io/otel/baggage"
)

func withIdempotencyBaggage(ctx context.Context, rawKey string) (context.Context, error) {
	sum := sha256.Sum256([]byte(rawKey))
	keyHash := hex.EncodeToString(sum[:])[:16]

	member, err := baggage.NewMember("idempotency.key_hash", keyHash)
	if err != nil {
		return ctx, err
	}
	bag, err := baggage.New(member)
	if err != nil {
		return ctx, err
	}
	return baggage.ContextWithBaggage(ctx, bag), nil
}

Note what this deliberately does not do: it does not touch trace_id or span_id. Those remain whatever the OTel SDK generated for this request’s root span — a fresh, random 128-bit value that will be entirely different on the client’s next retry. The idempotency key hash rides alongside trace context as an independent, application-defined piece of state.

Step 3 — Propagate baggage across the next HTTP hop

// Go: outbound call to the order service, injecting both traceparent and baggage headers
func callOrderService(ctx context.Context, client *http.Client, url string) (*http.Response, error) {
	req, _ := http.NewRequestWithContext(ctx, "POST", url, nil)
	propagator.Inject(ctx, propagation.HeaderCarrier(req.Header))
	// req.Header now contains both "traceparent" and "baggage" headers
	return client.Do(req)
}

Step 4 — Read baggage in the downstream service and attach it as a span attribute (Java, Payment Service)

// Java: extract baggage on the receiving side and copy it onto the active span
import io.opentelemetry.api.baggage.Baggage;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.propagation.TextMapGetter;

public void handleIncomingRequest(HttpServletRequest request) {
    Context extractedContext = openTelemetry.getPropagators().getTextMapPropagator()
        .extract(Context.current(), request, HttpServletRequestGetter.INSTANCE);

    try (var scope = extractedContext.makeCurrent()) {
        String keyHash = Baggage.fromContext(extractedContext).getEntryValue("idempotency.key_hash");
        Span currentSpan = Span.current();
        if (keyHash != null) {
            currentSpan.setAttribute("idempotency.key_hash", keyHash);
        }
        processPayment(request);
    }
}

Step 5 — Re-inject baggage on any further outbound calls

Baggage does not propagate itself past a service that does not explicitly re-inject it on outbound requests — each hop must both extract on the way in and inject on the way out.

// Java: propagate baggage onward if the payment service calls a fraud-check service
import io.opentelemetry.context.propagation.TextMapSetter;

public HttpResponse callFraudCheck(HttpRequest.Builder builder) {
    openTelemetry.getPropagators().getTextMapPropagator()
        .inject(Context.current(), builder, HttpRequestBuilderSetter.INSTANCE);
    return httpClient.send(builder.build(), HttpResponse.BodyHandlers.ofString());
}

The diagram below shows the full three-hop picture: trace_id stays constant across every hop (it is the same trace), span_id changes at each hop (each service creates its own child span), and the idempotency key travels unchanged in baggage — a completely different propagation rule from either trace context field.

Baggage propagation across three hops API Gateway calls Order Service calls Payment Service. Each hop carries a traceparent header whose trace-id stays constant (7a3f9c...) but whose span-id changes at every hop (b1 then c2), alongside a baggage header carrying idempotency.key=f8d2e1 unchanged. A comparison at the bottom shows that traceparent's fixed 128-bit/64-bit hex fields have no room for a business key, while baggage is designed for arbitrary key-value pairs propagated unchanged hop to hop. API Gateway Order Service Payment Service traceparent: 00-7a3f9c…-b1-01 baggage: idempotency.key=f8d2e1 traceparent: 00-7a3f9c…-c2-01 baggage: idempotency.key=f8d2e1 trace_id constant, span_id changes each hop — baggage carries idempotency.key unchanged ✗ traceparent — wrong place 00-<trace-id>-<span-id>-01 fixed 128-bit / 64-bit hex fields — no room for a business identifier ✓ baggage — correct place idempotency.key=f8d2e1 W3C Baggage: arbitrary key/value pairs, propagated unchanged hop-to-hop

Why NOT to overload traceparent

The W3C Trace Context specification defines traceparent as <version>-<trace-id>-<parent-id>-<trace-flags>, where trace-id is a 128-bit value and parent-id a 64-bit value, both required to be generated with effectively random entropy. Three concrete reasons never to derive either from your idempotency key:

  1. Format incompatibility. Any compliant vendor, collector, or load balancer that parses traceparent assumes fixed hex-digit lengths. A key that doesn’t fit — or worse, one that does fit but collides with the randomness assumption — will either be rejected or silently corrupt trace sampling decisions made by intermediate systems.
  2. Lifetime mismatch. trace-id is regenerated on every new root span — every retry gets a new one by design. An idempotency key must survive across retries. Putting the key inside a field that resets on every attempt defeats the entire purpose of carrying it.
  3. No structured place for metadata. Even if you wanted to encode more than one field (key hash plus, say, a tenant ID), traceparent has no extensibility — tracestate allows vendor-specific key-value pairs but is explicitly documented as unsuitable for arbitrary application data at volume. Baggage is the mechanism the specification actually provides for this.

Correlating across services once baggage lands

With idempotency.key_hash attached as a span attribute at every hop (Step 4), a query against your tracing backend for idempotency.key_hash=f8d2e1 returns every span in every service that touched this logical operation — not just the spans within one trace_id. This is what lets you answer, “did the fraud-check service actually see this operation, or did the payment service short-circuit before calling it?” without grepping logs across three codebases.

Verification and testing

# Send a request with an idempotency key through the gateway and confirm baggage reaches the last hop
curl -X POST http://localhost:8080/checkout \
  -H "Idempotency-Key: 7c1e-order-8841" -d '{"amount":4200}'

# Query the tracing backend for the key_hash attribute across ALL services, not one trace
curl -s 'http://localhost:16686/api/traces?tags=%7B%22idempotency.key_hash%22%3A%22f8d2e1%22%7D' \
  | jq '[.data[].processes | to_entries[] | .value.serviceName] | unique'
# Expect: ["api-gateway", "order-service", "payment-service"] — confirms the key reached every hop
# Confirm baggage headers are actually on the wire between hops, not just in-process
tcpdump -A -i lo0 'tcp port 8081' | grep -i "baggage:"
# Expect a line like: baggage: idempotency.key_hash=f8d2e1
# Confirm an intermediate proxy is not stripping the baggage header
curl -v -H "baggage: idempotency.key_hash=test123" http://localhost:8081/internal/echo-headers 2>&1 | grep -i baggage
# If the echoed response is missing the header, the proxy or gateway config needs an explicit allow-list entry

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
An intermediate reverse proxy or API gateway strips the baggage header because it is not in an explicit allow-list of forwarded headers Add baggage (and traceparent) to the proxy’s forwarded-header allow-list explicitly — most gateways default to stripping unrecognized headers; verify with a header-echo test at each hop idempotency.key_hash present on first-hop spans but absent on downstream-hop spans in the same trace; cross-service correlation query returns fewer services than expected
Baggage payload exceeds the SDK’s per-entry or total size limit (commonly 180 bytes per entry, 8192 bytes total) and is silently truncated or dropped Keep the baggage value to the hashed key only (16 hex characters), never the raw key or additional metadata; if you need more context, add it as span attributes at each hop instead of growing the baggage payload SDK debug logs showing baggage entry exceeds limit, dropped; baggage_propagation_incomplete_total counter from a validating middleware
A retry generates a new idempotency key baggage value because the client-side key generation logic runs per-attempt instead of being cached once per logical operation Ensure the client (or the outermost gateway acting on the client’s behalf) generates the key once and reuses it verbatim on every retry — see idempotency key generation strategies for stable key derivation Two independent traces sharing the same operation but showing different idempotency.key_hash values — investigate client-side key caching first
Double-encoding: a service framework URL-encodes the baggage header value on inject and the receiving service does not decode it before comparing, so hash values appear to mismatch across hops despite being logically identical Standardize on the OTel SDK’s built-in Baggage propagator rather than hand-rolling header parsing; add a contract test that round-trips a known baggage value through the full gateway → downstream chain in CI key_hash_mismatch_total from a reconciliation job; spans showing percent-encoded characters (%3D) inside what should be a clean hex string

SRE / observability checklist

  1. Cross-service key correlation query — a saved query in your tracing backend for idempotency.key_hash=<value> should return spans from every service in the expected call graph; alert if a known-critical path (e.g. payment → fraud-check) is missing from the result set.
  2. baggage_propagation_incomplete_total — counter incremented by middleware whenever an expected baggage key is absent on an inbound request that should have carried it; alert if rate exceeds 0.1% of requests.
  3. Header allow-list drift detection — a periodic synthetic request through the full proxy chain with a test baggage value, asserting it survives to the last hop; alert immediately on any proxy config change that breaks this.
  4. traceparent and baggage both present on 100% of inter-service calls — validate via a collector-side span processor that inspects incoming request headers (where available) or via synthetic canary requests.
  5. Baggage payload size gauge — track baggage_entry_bytes per key to catch accidental growth (e.g. someone appending raw JSON instead of a hash) before it hits the SDK’s silent-drop limit.
  6. Trace-id-differs, key-hash-matches dashboard — a panel showing count of distinct trace_id values grouped by idempotency.key_hash, which should be greater than 1 whenever retries occurred, confirming the correlation model described on the parent page is working end to end.