Defining SLOs and Alerts for Deduplication Failures

Part of: Monitoring Idempotency Metrics & Dashboards

Raw metrics tell you a number changed; a service-level objective (SLO) tells you whether that change matters enough to wake someone up. This runbook defines a dedup-correctness SLI, sets an error budget, and writes multi-window burn-rate alerting rules so pages fire fast for severe outages and stay quiet during brief, self-resolving blips. It builds directly on the metrics taxonomy and the operational context in Observability & Operations for Idempotent Systems — read those first if the metric names below are unfamiliar.

Problem statement and prerequisites

What you are building: a formal SLI/SLO pair for deduplication correctness, an error-budget calculation, and a set of Prometheus alerting rules that page on fast budget burn and file a lower-urgency ticket on slow burn.

Prerequisites:

  • The counters from monitoring idempotency metrics and dashboards — specifically idempotency_requests_total and dedup_conflict_total — are already emitting.
  • You understand fencing tokens and lease expiry from lock timeout and lease management, since a stale-lock incident is one of the concrete events this SLI must catch.
  • A Prometheus Alertmanager instance with a PagerDuty receiver configured, or an equivalent routing layer.

Step-by-step implementation

Step 1 — Define the dedup-correctness SLI

The SLI is the proportion of requests that received a correct idempotent outcome: either a genuine cache hit returning the original recorded response, or a genuine miss processed exactly once. It excludes legitimate, correctly-resolved conflicts (two concurrent retries where one correctly waits and receives the other’s result) but counts as bad events: duplicate processing, a stale-fencing-token write that was not rejected, or a reconciliation record left stuck past its deadline.

SLI = 1 - (duplicate_processing_events + unrejected_stale_writes) / total_requests

Instrument the numerator as its own counter rather than deriving it after the fact from logs — deriving SLIs retroactively from unstructured logs is unreliable during an incident when you need the number immediately:

# Python: increment the bad-event counter at the one place duplicate processing can be detected
from prometheus_client import Counter

dedup_correctness_bad_events_total = Counter(
    "dedup_correctness_bad_events_total",
    "Requests that violated the deduplication correctness guarantee",
    ["reason", "route"],  # reason: "duplicate_processed" | "stale_write_accepted" | "reconciliation_timeout"
)

def record_bad_event(reason: str, route: str):
    dedup_correctness_bad_events_total.labels(reason=reason, route=route).inc()

Step 2 — Set the SLO target and compute the error budget

For a payment or fintech write path, a 99.95% correctness target is a reasonable starting point — tighter than a general availability SLO because a single bad event has outsized cost (a double charge, a duplicate shipment). Compute the budget explicitly rather than reasoning about the percentage alone:

SLO target:        99.95%
Error budget:       0.05%
Window:             30 days
Total requests/mo:  ~130,000,000 (assume 50 req/s average)
Budget in requests: 130,000,000 × 0.0005 = 65,000 bad events allowed per month
Budget in time (if fully unavailable): 30 days × 0.0005 = 21.6 minutes/month

Track budget consumption as a Prometheus recording rule so it is a first-class queryable signal, not a spreadsheet calculation redone by hand during an incident:

# recording_rules.yml
groups:
  - name: dedup-slo
    interval: 30s
    rules:
      - record: dedup:correctness:sli_ratio
        expr: >
          1 - (
            sum(rate(dedup_correctness_bad_events_total[5m]))
            /
            sum(rate(idempotency_requests_total[5m]))
          )
      - record: dedup:correctness:error_budget_remaining_ratio
        expr: >
          1 - (
            (1 - dedup:correctness:sli_ratio) / 0.0005
          )

Step 3 — Write multi-window, multi-burn-rate alerting rules

A single fixed threshold either pages too late on a severe outage or too often on noise. The standard pattern (as used for Google SRE-style SLOs) requires the burn rate to exceed a multiple of the sustainable rate across two windows simultaneously — a short window for speed, a long window to avoid paging on transient spikes:

# alert_rules.yml — multi-window burn-rate alerts for dedup correctness SLO (99.95% / 30d)
groups:
  - name: dedup-slo-burn-rate
    rules:
      # Fast burn: would exhaust the 30-day budget in ~2 hours if sustained — page immediately
      - alert: DedupCorrectnessFastBurn
        expr: >
          (
            dedup:correctness:sli_ratio5m < (1 - 14.4 * 0.0005)
            and
            dedup:correctness:sli_ratio1h < (1 - 14.4 * 0.0005)
          )
        for: 2m
        labels:
          severity: page
          team: payments-platform
        annotations:
          summary: "Dedup correctness SLO burning at 14.4x — budget exhausted in ~2h if sustained"
          runbook: "https://distributedrequest.com/observability-operations-for-idempotent-systems/monitoring-idempotency-metrics-and-dashboards/defining-slos-and-alerts-for-deduplication-failures/"

      # Slow burn: would exhaust the budget in ~5 days if sustained — ticket, not a page
      - alert: DedupCorrectnessSlowBurn
        expr: >
          (
            dedup:correctness:sli_ratio1h < (1 - 6 * 0.0005)
            and
            dedup:correctness:sli_ratio6h < (1 - 6 * 0.0005)
          )
        for: 15m
        labels:
          severity: ticket
          team: payments-platform
        annotations:
          summary: "Dedup correctness SLO burning at 6x — budget exhausted in ~5d if sustained"

The 5m/1h/6h ratio series referenced above are additional recording rules using the same dedup:correctness:sli_ratio expression with different rate() windows — add sli_ratio5m, sli_ratio1h, and sli_ratio6h as separate recorded names so the alert expressions can reference each window independently:

      - record: dedup:correctness:sli_ratio5m
        expr: 1 - (sum(rate(dedup_correctness_bad_events_total[5m])) / sum(rate(idempotency_requests_total[5m])))
      - record: dedup:correctness:sli_ratio1h
        expr: 1 - (sum(rate(dedup_correctness_bad_events_total[1h])) / sum(rate(idempotency_requests_total[1h])))
      - record: dedup:correctness:sli_ratio6h
        expr: 1 - (sum(rate(dedup_correctness_bad_events_total[6h])) / sum(rate(idempotency_requests_total[6h])))

Burn-rate multiplier reference

The 14.4 and 6 multipliers used above aren’t arbitrary — they come from how much of the monthly budget a sustained burn at that rate would consume within a given detection window. Use this table when tuning thresholds for a different SLO target or window:

Burn Rate Multiplier Budget Consumed in 1 Hour Time to Exhaust 30-Day Budget Typical Severity
14.4x ~2% ~2 hours Page immediately (for: 2m)
6x ~0.8% ~5 days Ticket, review within a business day (for: 15m)
3x ~0.4% ~10 days Ticket, review within the week
1x (sustainable rate) ~0.14% 30 days (exactly the budget) No alert — this is the target burn rate

Higher multipliers require shorter detection windows because they represent genuinely severe degradations that must page fast; lower multipliers can tolerate longer windows because a slow, sustained burn is still recoverable if caught within days rather than minutes. Recalculate this table whenever the SLO target or the measurement window changes — a 99.9% target over a 90-day window produces different absolute budgets even though the multipliers stay the same.

Step 4 — Route fast-burn pages to PagerDuty

Alertmanager routes on the severity label set above, sending page straight to PagerDuty with no delay and ticket to a lower-urgency channel:

# alertmanager.yml
route:
  receiver: default-ticket
  group_by: ["alertname", "team"]
  routes:
    - match:
        severity: page
      receiver: pagerduty-payments
      group_wait: 10s
      repeat_interval: 15m
    - match:
        severity: ticket
      receiver: jira-payments-backlog
      repeat_interval: 24h

receivers:
  - name: pagerduty-payments
    pagerduty_configs:
      - service_key: "$PAGERDUTY_INTEGRATION_KEY"
        severity: critical
        description: '{{ .CommonAnnotations.summary }}'
  - name: jira-payments-backlog
    webhook_configs:
      - url: "https://jira-webhook-bridge.internal/dedup-slo"

Step 5 — Codify the runbook link in the alert

Every page must carry a direct link to remediation steps so the responding engineer doesn’t start from zero. The runbook annotation above points back to this page; extend it in your real alert to also link the stale lock handling runbook when the bad-event reason label is stale_write_accepted.

Verification and testing

Confirm the SLI recording rules evaluate correctly

curl -s 'localhost:9090/api/v1/query?query=dedup:correctness:sli_ratio' | jq '.data.result'
# Expect a value very close to 1 (e.g. 0.9998) under normal operation

Inject synthetic bad events and confirm the fast-burn alert fires

# Simulate 200 duplicate-processing events in a short window to trigger a 14.4x burn
for i in $(seq 1 200); do
  curl -s -X POST localhost:9091/metrics/job/synthetic \
    --data-binary 'dedup_correctness_bad_events_total{reason="duplicate_processed",route="/v1/payments"} 1'
done
# Wait for the 2m "for" duration, then check Alertmanager
curl -s localhost:9093/api/v2/alerts | jq '.[] | select(.labels.alertname=="DedupCorrectnessFastBurn")'

Confirm error budget consumption is queryable

curl -s 'localhost:9090/api/v1/query?query=dedup:correctness:error_budget_remaining_ratio' | jq '.data.result[0].value[1]'
# A value near 1.0 means most of the monthly budget remains; near 0 means it is nearly exhausted

Confirm PagerDuty receives the fast-burn page

curl -s -H "Authorization: Token token=$PD_API_TOKEN" \
  "https://api.pagerduty.com/incidents?service_ids[]=$PD_SERVICE_ID" | jq '.incidents[0].title'

Failure scenarios and debugging

Failure Scenario Remediation Steps Observability Hooks
Fast-burn alert never fires despite a real outage Confirm both the 5m and 1h windows are wired into the and expression — a single-window rule pages too early on noise, but a misconfigured dual-window rule can also silently require both to breach simultaneously when only one has enough data yet dedup:correctness:sli_ratio5m and sli_ratio1h both queried independently in Prometheus to confirm they carry data
Alert flaps between firing and resolved every few minutes The for duration is too short relative to the natural noise in the ratio at low traffic volumes; increase for or widen the rate window so small sample sizes don’t cross the threshold repeatedly ALERTS metric transition count; Alertmanager notification log showing repeated fire/resolve pairs
Error budget appears exhausted but no user-facing incident occurred The bad-event counter is being incremented by a false positive in the detection logic itself (e.g. a reconciliation job flagging records that actually completed) — audit the reason label breakdown before assuming a real correctness violation dedup_correctness_bad_events_total broken down by (reason); cross-reference with the reconciliation backlog gauge
PagerDuty receives duplicate pages for the same underlying incident group_by in Alertmanager doesn’t include a dimension that differs across replicas firing the same rule; add the relevant label (e.g. route) to group_by or confirm repeat_interval is set high enough to avoid re-notification storms Alertmanager /api/v2/alerts/groups endpoint; PagerDuty incident dedup key matching the Alertmanager group_key
Slow-burn ticket never files even during a genuine multi-day degradation Confirm the 6h window recording rule has sufficient retention and isn’t being dropped by a retention policy shorter than 6 hours; verify the ticket receiver webhook is reachable and not silently failing alertmanager_notifications_failed_total counter; webhook endpoint health check separate from the alerting pipeline

SRE / observability checklist

  1. dedup_correctness_bad_events_total{reason,route} — Counter. The single most important metric on this page; confirm it increments at every place duplicate processing, stale-write acceptance, or reconciliation timeout can occur, not just one of them.
  2. dedup:correctness:sli_ratio5m / sli_ratio1h / sli_ratio6h — recording-rule Gauges powering the burn-rate windows; alert on any of them reporting NoData as if it were a budget breach.
  3. dedup:correctness:error_budget_remaining_ratio — Gauge surfaced on a leadership-visible dashboard; treat a value below 0.2 (20% of monthly budget remaining) as a trigger to freeze risky deploys to the dedup layer.
  4. ALERTS{alertname="DedupCorrectnessFastBurn"} — track fire/resolve transitions over time to catch flapping before it causes alert fatigue.
  5. PagerDuty incident annotations — confirm every fired alert carries the runbook link and a reason-specific remediation pointer, not a generic “investigate dashboards” message.
  6. Alertmanager alertmanager_notifications_failed_total — a failure in the paging pipeline itself is worse than a missed threshold; alert on this independently of the SLO burn-rate alerts.