fix(proxy): expose persistent savings metrics (#1647)

aivinay · web-flow · commit 5fe4e7b19530 · 2026-07-01T23:28:12.000-05:00
## Description Closes #1616 Expose the proxy's durable `persistent_savings.lifetime` totals through `/metrics` so Prometheus/Grafana scrapes can read the same lifetime savings counters already visible in `/stats` and `/stats-history`. The existing runtime counters remain process-local: `headroom_tokens_saved_total` still resets with the proxy process. New `headroom_persistent_savings_*` counters are sourced from the `SavingsTracker` lifetime block. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Export durable lifetime savings counters from `PrometheusMetrics.export()`: - `headroom_persistent_savings_requests_total` - `headroom_persistent_savings_tokens_saved_total` - `headroom_persistent_savings_input_tokens_total` - `headroom_persistent_savings_input_cost_usd_total` - `headroom_persistent_savings_compression_savings_usd_total` - Add a restart regression proving runtime counters reset while persistent savings counters remain available from the same savings file. - Extend the existing `/stats-history` restart test with `/metrics` endpoint assertions. - Update metrics docs to distinguish runtime `headroom_tokens_saved_total` from lifetime `headroom_persistent_savings_tokens_saved_total`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text Local focused checks: $ rtk /usr/bin/env HEADROOM_REQUIRE_RUST_CORE=false PYTHONPATH=. /tmp/headroom-1616-testenv/bin/python -m pytest tests/test_proxy_cache_ttl_metrics.py::test_prometheus_metrics_export_includes_extended_fields tests/test_proxy_cache_ttl_metrics.py::test_prometheus_export_includes_persistent_savings_after_restart 2 passed, 1 warning in 0.19s $ rtk /tmp/headroom-1616-testenv/bin/python -m ruff check headroom/proxy/prometheus_metrics.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_savings_history.py All checks passed! $ rtk /tmp/headroom-1616-testenv/bin/python -m ruff format --check headroom/proxy/prometheus_metrics.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_savings_history.py 3 files already formatted $ rtk git diff --check # no output GitHub Actions: All non-skipped checks passed on PR #1647, including lint, build, build-wheel, test (1-4), test-agno, test-extras, test-dashboard-ui, docker-native-e2e, docker-init-e2e, docker-wrap-e2e, security checks, merge-conflicts, and PR governance. ``` ## Real Behavior Proof - Environment: local macOS worktree, throwaway Python env at `/tmp/headroom-1616-testenv`, `PYTHONPATH=.`. - Exact command / steps: recorded a compressed request through `PrometheusMetrics.record_request()`, re-created `PrometheusMetrics` with the same `SavingsTracker` path, then exported `/metrics` text. - Observed result: runtime counters are zero after re-creating the metrics object, while `headroom_persistent_savings_tokens_saved_total` and related persistent counters still expose the durable lifetime values. - Not tested: full server-level pytest locally, because the local build is blocked by the known native `headroom._core`/`esaxx-rs` build issue (`fatal error: 'cstdint' file not found`). The app-level `/metrics` assertions passed in GitHub Actions. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes This intentionally does not rename or hydrate the existing runtime `headroom_tokens_saved_total` counter. That preserves the current process-local semantics and gives external dashboards a dedicated lifetime series that maps directly to `/stats.persistent_savings`. `mypy headroom` was not run as a standalone local command. CHANGELOG is N/A for this narrow proxy metrics fix unless maintainers prefer an entry.
diff --git a/docs/content/docs/metrics.mdx b/docs/content/docs/metrics.mdx
@@ -72,6 +72,9 @@ headroom_requests_total{mode="optimize"} 1234
 # HELP headroom_tokens_saved_total Total tokens saved
 headroom_tokens_saved_total 5678900
 
+# HELP headroom_persistent_savings_tokens_saved_total Durable lifetime input tokens saved by proxy compression
+headroom_persistent_savings_tokens_saved_total 5678900
+
 # HELP headroom_compression_ratio Compression ratio histogram
 headroom_compression_ratio_bucket{le="0.5"} 890
 headroom_compression_ratio_bucket{le="0.7"} 1100
@@ -253,7 +256,8 @@ When the budget is exceeded, requests return a budget exceeded error, the `/stat
 
 | Metric | What It Tells You | Target |
 |--------|-------------------|--------|
-| `tokens_saved_total` | Total cost savings | Higher is better |
+| `headroom_tokens_saved_total` | Runtime tokens saved since this proxy process started | Higher is better |
+| `headroom_persistent_savings_tokens_saved_total` | Durable lifetime tokens saved from `/stats.persistent_savings` | Higher is better |
 | `compression_ratio_avg` | Efficiency | 0.7--0.9 typical |
 | `cache_hit_rate` | Cache effectiveness | >20% is good |
 | `latency_p99` | Performance impact | &lt;10ms |
@@ -265,7 +269,8 @@ Example Prometheus queries for a Grafana dashboard:
 
 | Panel | PromQL |
 |-------|--------|
-| Tokens Saved | `headroom_tokens_saved_total` |
+| Runtime Tokens Saved | `headroom_tokens_saved_total` |
+| Lifetime Tokens Saved | `headroom_persistent_savings_tokens_saved_total` |
 | Compression Ratio (median) | `histogram_quantile(0.5, headroom_compression_ratio_bucket)` |
 | Request Latency (p99) | `histogram_quantile(0.99, headroom_latency_seconds_bucket)` |
 | Cache Hit Rate | `headroom_cache_hits_total / (headroom_cache_hits_total + headroom_cache_misses_total)` |
diff --git a/docs/content/docs/proxy.mdx b/docs/content/docs/proxy.mdx
@@ -151,11 +151,14 @@ curl http://localhost:8787/metrics
 ```
 headroom_requests_total{mode="optimize"} 1234
 headroom_tokens_saved_total 5678900
+headroom_persistent_savings_tokens_saved_total 5678900
 headroom_compression_ratio_bucket{le="0.5"} 890
 headroom_latency_seconds_bucket{le="0.01"} 800
 headroom_cache_hits_total 456
 ```
 
+`headroom_tokens_saved_total` is the runtime counter for the current proxy process. Use `headroom_persistent_savings_tokens_saved_total` for durable lifetime savings that match `/stats.persistent_savings`.
+
 ### `POST /v1/messages`
 
 Anthropic API format. The proxy compresses messages, forwards to Anthropic, and returns the response.
diff --git a/headroom/proxy/prometheus_metrics.py b/headroom/proxy/prometheus_metrics.py
@@ -825,6 +825,7 @@ async def export(self) -> str:
             stage_timing_count_snapshot = dict(self.stage_timing_count)
             stage_timing_max_snapshot = dict(self.stage_timing_max)
         async with self._lock:
+            lifetime_savings = self.savings_tracker.snapshot()["lifetime"]
             lines: list[str] = []
             _append_metric(
                 lines,
@@ -896,6 +897,44 @@ async def export(self) -> str:
                 help_text="Tokens saved by optimization",
                 value=self.tokens_saved_total,
             )
+            _append_metric(
+                lines,
+                name="headroom_persistent_savings_requests_total",
+                metric_type="counter",
+                help_text="Durable lifetime requests recorded by the proxy savings tracker",
+                value=lifetime_savings["requests"],
+            )
+            _append_metric(
+                lines,
+                name="headroom_persistent_savings_tokens_saved_total",
+                metric_type="counter",
+                help_text="Durable lifetime input tokens saved by proxy compression",
+                value=lifetime_savings["tokens_saved"],
+            )
+            _append_metric(
+                lines,
+                name="headroom_persistent_savings_input_tokens_total",
+                metric_type="counter",
+                help_text="Durable lifetime input tokens recorded by the proxy savings tracker",
+                value=lifetime_savings["total_input_tokens"],
+            )
+            _append_metric(
+                lines,
+                name="headroom_persistent_savings_input_cost_usd_total",
+                metric_type="counter",
+                help_text="Durable lifetime input spend in USD estimated by the proxy savings tracker",
+                value=lifetime_savings["total_input_cost_usd"],
+            )
+            _append_metric(
+                lines,
+                name="headroom_persistent_savings_compression_savings_usd_total",
+                metric_type="counter",
+                help_text=(
+                    "Durable lifetime compression savings in USD estimated by the "
+                    "proxy savings tracker"
+                ),
+                value=lifetime_savings["compression_savings_usd"],
+            )
             # NOTE: per-strategy compression breakdown is tracked
             # internally on `self.compressions_by_strategy` and
             # `self.tokens_saved_by_strategy` (populated by
diff --git a/tests/test_proxy_cache_ttl_metrics.py b/tests/test_proxy_cache_ttl_metrics.py
@@ -9,6 +9,7 @@
 from headroom.observability import reset_headroom_tracing, reset_otel_metrics
 from headroom.proxy.cost import CostTracker, build_prefix_cache_stats
 from headroom.proxy.prometheus_metrics import PrometheusMetrics
+from headroom.proxy.savings_tracker import SavingsTracker
 
 
 def test_prometheus_metrics_tracks_observed_ttl_buckets() -> None:
@@ -79,8 +80,10 @@ def test_prefix_cache_stats_include_observed_ttl_mix() -> None:
     assert stats["totals"]["observed_ttl_buckets"]["1h"]["tokens"] == 45
 
 
-def test_prometheus_metrics_export_includes_extended_fields() -> None:
-    metrics = PrometheusMetrics()
+def test_prometheus_metrics_export_includes_extended_fields(tmp_path) -> None:
+    metrics = PrometheusMetrics(
+        savings_tracker=SavingsTracker(path=str(tmp_path / "proxy_savings.json"))
+    )
 
     asyncio.run(
         metrics.record_request(
@@ -105,6 +108,11 @@ def test_prometheus_metrics_export_includes_extended_fields() -> None:
 
     exported = asyncio.run(metrics.export())
 
+    assert "headroom_requests_total 1" in exported
+    assert "headroom_tokens_saved_total 5" in exported
+    assert "headroom_persistent_savings_requests_total 1" in exported
+    assert "headroom_persistent_savings_tokens_saved_total 5" in exported
+    assert "headroom_persistent_savings_input_tokens_total 100" in exported
     assert "headroom_latency_ms_count 1" in exported
     assert 'headroom_transform_timing_ms_sum{transform="router"} 4.5' in exported
     assert 'headroom_waste_signal_tokens_total{signal="json_bloat"} 7' in exported
@@ -113,6 +121,31 @@ def test_prometheus_metrics_export_includes_extended_fields() -> None:
     assert "headroom_cache_bust_tokens_lost_total 11" in exported
 
 
+def test_prometheus_export_includes_persistent_savings_after_restart(tmp_path) -> None:
+    savings_path = tmp_path / "proxy_savings.json"
+    metrics = PrometheusMetrics(savings_tracker=SavingsTracker(path=str(savings_path)))
+
+    asyncio.run(
+        metrics.record_request(
+            provider="openai",
+            model="gpt-4o",
+            input_tokens=120,
+            output_tokens=20,
+            tokens_saved=40,
+            latency_ms=12.5,
+        )
+    )
+
+    reloaded = PrometheusMetrics(savings_tracker=SavingsTracker(path=str(savings_path)))
+    exported = asyncio.run(reloaded.export())
+
+    assert "headroom_tokens_saved_total 0" in exported
+    assert "headroom_requests_total 0" in exported
+    assert "headroom_persistent_savings_requests_total 1" in exported
+    assert "headroom_persistent_savings_tokens_saved_total 40" in exported
+    assert "headroom_persistent_savings_input_tokens_total 120" in exported
+
+
 def test_streaming_parser_extracts_anthropic_ttl_bucket_usage() -> None:
     from headroom.proxy.server import HeadroomProxy, ProxyConfig
 
diff --git a/tests/test_proxy_savings_history.py b/tests/test_proxy_savings_history.py
@@ -866,6 +866,12 @@ def test_stats_history_persists_across_restarts_and_stats_stays_compatible(tmp_p
         assert stats_data["persistent_savings"]["lifetime"]["tokens_saved"] == 40
         assert stats_data["persistent_savings"]["storage_path"] == str(savings_path)
 
+        metrics = client.get("/metrics")
+        assert metrics.status_code == 200
+        assert "headroom_tokens_saved_total 40" in metrics.text
+        assert "headroom_persistent_savings_tokens_saved_total 40" in metrics.text
+        assert "headroom_persistent_savings_requests_total 1" in metrics.text
+
         history = client.get("/stats-history")
         assert history.status_code == 200
         history_data = history.json()
@@ -907,6 +913,12 @@ def test_stats_history_persists_across_restarts_and_stats_stays_compatible(tmp_p
         assert history.json()["lifetime"]["tokens_saved"] == 40
         assert history.json()["display_session"]["requests"] == 1
 
+        metrics = client.get("/metrics")
+        assert metrics.status_code == 200
+        assert "headroom_tokens_saved_total 0" in metrics.text
+        assert "headroom_persistent_savings_tokens_saved_total 40" in metrics.text
+        assert "headroom_persistent_savings_requests_total 1" in metrics.text
+
         _record_request(client, model="gpt-4o", tokens_saved=15)
 
         updated = client.get("/stats-history").json()
@@ -922,6 +934,12 @@ def test_stats_history_persists_across_restarts_and_stats_stays_compatible(tmp_p
         assert updated["series"]["daily"][0]["total_input_tokens_delta"] == 240
         assert updated["series"]["daily"][0]["total_input_cost_usd_delta"] == pytest.approx(0.48)
 
+        metrics = client.get("/metrics")
+        assert metrics.status_code == 200
+        assert "headroom_tokens_saved_total 15" in metrics.text
+        assert "headroom_persistent_savings_tokens_saved_total 55" in metrics.text
+        assert "headroom_persistent_savings_requests_total 2" in metrics.text
+
         full = client.get("/stats-history?history_mode=full").json()
         assert full["history_summary"]["mode"] == "full"
         assert full["history_summary"]["stored_points"] == 2
