Building an Idempotency Hit-Rate Dashboard in Grafana

Part of: Monitoring Idempotency Metrics & Dashboards

This is a focused runbook for building the single most useful panel in a deduplication dashboard: the idempotency hit rate. It walks through instrumenting hit/miss counters in Python and Go, writing the PromQL that turns them into a ratio, and wiring the result into a provisioned Grafana panel with an alert threshold. You should already understand the metrics taxonomy and cardinality rules from the parent page and the broader operational context in Observability & Operations for Idempotent Systems before wiring this into a production dashboard.

---

Problem statement and prerequisites

What you are building: a Grafana panel and backing PromQL query that shows the live percentage of requests served from the deduplication cache versus processed fresh, plus an alert that pages when it drops below a safe threshold.

Prerequisites:

• A Prometheus server scraping your application, and a Grafana instance with that Prometheus configured as a data source.
• The dedup layer already emits a distinguishable result for each check — you’re adding the counter increments if it doesn’t yet.
• Familiarity with the idempotency key storage layer your hit/miss decision reads from, and with using Redis SETNX for distributed request deduplication if Redis is your backing store.

---

How the panel derives its number

---

Step-by-step implementation

Step 1 — Instrument hit and miss counters

Increment a labeled counter at the exact point the dedup layer decides whether a request is a cache hit, a fresh miss, or a conflict with an in-flight duplicate.

Python (Flask middleware)

from prometheus_client import Counter

idempotency_requests_total = Counter(
    "idempotency_requests_total",
    "Idempotency check outcomes",
    ["result", "route", "store"],
)

def check_idempotency(key: str, route: str, store_client) -> str:
    existing = store_client.get(key)
    if existing is not None and existing["status"] == "PROCESSING":
        idempotency_requests_total.labels(result="conflict", route=route, store="redis").inc()
        return "conflict"
    if existing is not None:
        idempotency_requests_total.labels(result="hit", route=route, store="redis").inc()
        return "hit"
    idempotency_requests_total.labels(result="miss", route=route, store="redis").inc()
    return "miss"

Go (net/http handler)

package dedup

import "github.com/prometheus/client_golang/prometheus"

var requestsTotal = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "idempotency_requests_total",
        Help: "Idempotency check outcomes",
    },
    []string{"result", "route", "store"},
)

func init() {
    prometheus.MustRegister(requestsTotal)
}

func CheckIdempotency(key, route string, store Store) string {
    existing, found := store.Get(key)
    switch {
    case found && existing.Status == "PROCESSING":
        requestsTotal.WithLabelValues("conflict", route, "redis").Inc()
        return "conflict"
    case found:
        requestsTotal.WithLabelValues("hit", route, "redis").Inc()
        return "hit"
    default:
        requestsTotal.WithLabelValues("miss", route, "redis").Inc()
        return "miss"
    }
}

Expose the registry on /metrics in both cases — prometheus_client’s default start_http_server in Python, or promhttp.Handler() mounted at /metrics in Go.

Step 2 — Verify the raw counters are scraped

curl -s localhost:9100/metrics | grep idempotency_requests_total
# idempotency_requests_total{result="hit",route="/v1/payments",store="redis"} 1042
# idempotency_requests_total{result="miss",route="/v1/payments",store="redis"} 58

Step 3 — Add the recording rule

Pre-compute the ratio so every consumer (dashboard, alert, API) reads one cheap value instead of recomputing a division across every label combination on each query:

# recording_rules.yml
groups:
  - name: idempotency
    interval: 30s
    rules:
      - record: idempotency:hit_rate:ratio
        expr: >
          sum(rate(idempotency_requests_total{result="hit"}[5m]))
          /
          sum(rate(idempotency_requests_total[5m]))
      - record: idempotency:hit_rate:ratio_by_route
        expr: >
          sum by (route) (rate(idempotency_requests_total{result="hit"}[5m]))
          /
          sum by (route) (rate(idempotency_requests_total[5m]))

Reload Prometheus to pick up the new rule file:

curl -X POST localhost:9090/-/reload

Step 4 — Provision the Grafana panel

Add the panel to a dashboard JSON model so it’s version-controlled and reproducible rather than hand-built through the UI:

{
  "title": "Idempotency Hit Rate",
  "type": "timeseries",
  "datasource": { "type": "prometheus", "uid": "prometheus-primary" },
  "targets": [
    {
      "expr": "idempotency:hit_rate:ratio",
      "legendFormat": "overall",
      "refId": "A"
    },
    {
      "expr": "idempotency:hit_rate:ratio_by_route",
      "legendFormat": "{{route}}",
      "refId": "B"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "percentunit",
      "min": 0,
      "max": 1,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "value": null, "color": "red" },
          { "value": 0.9, "color": "yellow" },
          { "value": 0.97, "color": "green" }
        ]
      }
    }
  }
}

Import via grafana-cli or the HTTP API so the panel deploys alongside the service rather than drifting from source control:

curl -X POST http://grafana:3000/api/dashboards/db \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -d @dashboard.json

Step 5 — Add dashboard template variables

Hard-coding a single route or store into the panel query forces a separate dashboard per service. Add Grafana template variables so one dashboard covers the whole fleet, and operators can pivot without editing JSON:

{
  "templating": {
    "list": [
      {
        "name": "route",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "prometheus-primary" },
        "query": "label_values(idempotency_requests_total, route)",
        "multi": true,
        "includeAll": true
      },
      {
        "name": "store",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "prometheus-primary" },
        "query": "label_values(idempotency_requests_total, store)",
        "multi": true,
        "includeAll": true
      }
    ]
  }
}

Reference the variables in the panel query with idempotency:hit_rate:ratio_by_route{route=~"$route"} so selecting a subset in the dropdown filters every panel on the dashboard simultaneously, without needing a second dashboard definition per team.

Step 6 — Add the alert rule

Grafana-managed alerting evaluates the same recording rule with a hold duration to avoid paging on brief blips:

# grafana alert rule (provisioned YAML)
apiVersion: 1
groups:
  - orgId: 1
    name: idempotency-alerts
    folder: Dedup
    interval: 1m
    rules:
      - uid: idempotency-hit-rate-low
        title: Idempotency hit rate below 90%
        condition: C
        data:
          - refId: A
            datasourceUid: prometheus-primary
            model: { expr: "idempotency:hit_rate:ratio", instant: true }
          - refId: C
            datasourceUid: __expr__
            model: { type: threshold, expression: A, conditions: [{ evaluator: { type: lt, params: [0.9] } }] }
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "Idempotency hit rate has been below 90% for 10 minutes"

---

Verification and testing

Confirm the raw counter increments under load

# Send 100 identical requests with the same idempotency key
for i in $(seq 1 100); do
  curl -s -X POST localhost:8080/v1/payments \
    -H "Idempotency-Key: test-fixed-key-001" \
    -d '{"amount": 100}' > /dev/null
done
curl -s localhost:9100/metrics | grep 'result="hit"'
# Expect ~99 hits and 1 miss for a fixed key across 100 requests

Confirm the recording rule evaluates

curl -s 'localhost:9090/api/v1/query?query=idempotency:hit_rate:ratio' | jq '.data.result'
# Expect a single series with a value between 0 and 1

Confirm the Grafana panel renders the same number

curl -s -H "Authorization: Bearer $GRAFANA_API_KEY" \
  "http://grafana:3000/api/dashboards/uid/<dashboard-uid>" | jq '.dashboard.panels[0].title'

Force a low hit rate and confirm the alert fires

# Generate 1000 unique keys (all misses) to drop the ratio below 0.9
for i in $(seq 1 1000); do
  curl -s -X POST localhost:8080/v1/payments \
    -H "Idempotency-Key: unique-$i" -d '{"amount": 1}' > /dev/null
done
# Check Grafana alerting state
curl -s -H "Authorization: Bearer $GRAFANA_API_KEY" \
  http://grafana:3000/api/alertmanager/grafana/api/v2/alerts | jq '.[].labels'

---

Failure scenarios and debugging

---

SRE / observability checklist

1. idempotency_requests_total{result,route,store} — Counter. Confirm all three label values are bounded (fixed route templates, fixed store names) before shipping to production.
2. idempotency:hit_rate:ratio — recording rule Gauge. Alert if it reports no data for more than 2 evaluation intervals — an absent rule is worse than a low one because it hides the outage.
3. prometheus_rule_group_last_duration_seconds — confirm recording rule evaluation stays well under the 30s interval; a slow rule delays every downstream consumer.
4. Grafana alert state (ALERTING / OK / NoData) — export via the Alertmanager API and pipe into your paging tool; treat NoData as equivalent to ALERTING for this specific rule.
5. Deploy markers overlaid on the panel — annotate the dashboard with deploy timestamps so a hit-rate dip that correlates with a release is obvious without cross-referencing a separate deploy log.
6. process_start_time_seconds — correlate counter resets with restarts to distinguish a real regression from noise introduced by a rolling deploy.

---

Related

• Monitoring Idempotency Metrics & Dashboards — parent page covering the full metrics taxonomy, cardinality rules, and Datadog/CloudWatch variants
• Defining SLOs and Alerts for Deduplication Failures — turning this hit-rate threshold into a formal SLO with an error budget and burn-rate alerts
• Idempotency Key Storage & TTL Management — the storage layer whose PROCESSING/COMPLETE states this dashboard’s hit/miss/conflict logic reads
• Using Redis SETNX for Distributed Request Deduplication — the Redis-backed store pattern used in the Python and Go examples above
• Observability & Operations for Idempotent Systems — the broader operational section covering tracing and chaos engineering alongside metrics