fix(proxy): expose persistent savings metrics#1647
Merged
JerrettDavis merged 1 commit intoJul 2, 2026
Conversation
Contributor
PR governanceThis PR follows the template and is marked ready for human review. |
02f2384 to
6f47b98
Compare
JerrettDavis
approved these changes
Jul 1, 2026
JerrettDavis
left a comment
Collaborator
There was a problem hiding this comment.
Looks good. The new persistent counters preserve the existing process-local metric semantics while exposing the durable lifetime totals under explicit names, and the restart coverage catches the important regression case.
JerrettDavis
approved these changes
Jul 2, 2026
Merged
chopratejas
pushed a commit
that referenced
this pull request
Jul 3, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>0.29.0</summary> ## [0.29.0](v0.28.0...v0.29.0) (2026-07-03) ### Features * **proxy:** add --lossless no-CCR mode with format-native compaction ([#1721](#1721)) ([c75ebde](c75ebde)) * **stats:** surface Codex WS compression counters in /stats summary ([#1680](#1680)) ([2fe19c3](2fe19c3)) * **transforms:** adaptive Otsu KEEP/DROP threshold (+ land relevance split on main) ([#1726](#1726)) ([eea667a](eea667a)) ### Bug Fixes * **bedrock:** fail fast when session-token auth lacks botocore ([#1553](#1553)) ([54cfa36](54cfa36)) * **bedrock:** route ARNs via converse, named AWS profiles, and au. re… ([#1456](#1456)) ([7d87aa2](7d87aa2)) * **ccr:** honor workspace dir for sqlite store ([#1564](#1564)) ([96e1dfe](96e1dfe)) * **claude:** surface Remote Control proxy incompatibility ([#1610](#1610)) ([4bf7f92](4bf7f92)) * **cli:** stop advertising unwired compression tuning env vars in banner ([#1634](#1634)) ([d5bf98d](d5bf98d)) * **codex:** avoid duplicate headroom provider config ([#1431](#1431)) ([ddd4adf](ddd4adf)) * **compression:** reject lossy unmarked tool output in unit router path ([#1479](#1479)) ([de24cd5](de24cd5)) * **cortex-code:** migrate to current Cortex REST API endpoints + add e2e benchmarks ([#1474](#1474)) ([f00ace6](f00ace6)) * **dashboard:** align token savings headline denominator ([#1653](#1653)) ([646e705](646e705)) * **dashboard:** derive per-project setup URL from live origin ([#1511](#1511)) ([e035aef](e035aef)) * **detection:** contain unidiff panic on orphaned +++ target line ([#1548](#1548)) ([e386c09](e386c09)) * **evals:** CJK-aware F1 tokenization + token estimation ([#1527](#1527)) ([99a8540](99a8540)) * **install:** close parent log fd in start_detached_agent ([#1576](#1576)) ([816cb85](816cb85)) * **install:** use Windows-safe PID liveness probe in runtime_status ([#1544](#1544)) ([#1560](#1560)) ([6b227b9](6b227b9)) * **learn:** aggregate verbosity baselines across projects instead of overwriting ([#1288](#1288)) ([27a5468](27a5468)) * **mcp:** show lifetime totals and label rolling session scope in headroom_stats ([#1428](#1428)) ([1c0e152](1c0e152)) * **memory:** cap local embedder CPU thread oversubscription ([#198](#198)) ([#1559](#1559)) ([b84afbf](b84afbf)) * **memory:** singleflight LocalBackend init to stop cold-start races ([#1691](#1691)) ([bec47a1](bec47a1)) * **openclaw:** detect uv-installed headroom binary in ~/.local/bin ([#1459](#1459)) ([adaeb88](adaeb88)) * **opencode:** preserve custom OpenAI gateway paths ([#1596](#1596)) ([c19347c](c19347c)) * **opencode:** route native providers + load transport plugin, fix Serena context ([#1573](#1573)) ([ad0034f](ad0034f)) * preserve anthropic passthrough tool order ([#1427](#1427)) ([a932247](a932247)) * **proxy/auth:** match real Anthropic OAuth token prefix (sk-ant-oat) ([#1672](#1672)) ([8cddf9b](8cddf9b)) * **proxy:** expose persistent savings metrics ([#1647](#1647)) ([5fe4e7b](5fe4e7b)) * **proxy:** fail open when kompress saturation would exhaust pre-upstream budget ([#1430](#1430)) ([15ac650](15ac650)) * **proxy:** handle streaming CCR retrieval ([#1451](#1451)) ([d337e3b](d337e3b)) * **proxy:** include system/tools/sampling in cache key ([#1473](#1473)) ([312129a](312129a)) * **proxy:** preserve Responses passthrough bytes ([#1598](#1598)) ([2a34a82](2a34a82)) * **proxy:** strip Codex lite header on the HTTP /responses path ([#1663](#1663)) ([9fbd47b](9fbd47b)) * **proxy:** wire --compression-max-workers / HEADROOM_COMPRESSION_MAX_WORKERS ([#1632](#1632)) ([814ffa3](814ffa3)) * **savings:** count cache-read tokens in input cost estimate ([#1429](#1429)) ([72ade37](72ade37)) * skip Magika backend on x86 CPUs without AVX2 ([#1162](#1162)) ([64783d8](64783d8)) * **transforms/content-router:** route grep/log output away from HTML extractor ([#1719](#1719)) ([0d18ef2](0d18ef2)) * **transforms:** bound native content detection with a Windows watchdog ([#575](#575)) ([#1563](#1563)) ([95abca3](95abca3)) * Vertex AI support for Claude Code with ANTHROPIC_VERTEX_BASE_URL ([#1393](#1393)) ([cff7247](cff7247)) * **wrap:** detach the shared proxy on Windows so it survives an ungraceful agent close ([#1464](#1464)) ([6cba441](6cba441)) * **wrap:** preserve custom Vertex base URL ([#1477](#1477)) ([75427bb](75427bb)) * **wrap:** remove rtk instructions from Codex AGENTS.md on unwrap ([#1604](#1604)) ([c9d717c](c9d717c)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
JerrettDavis
pushed a commit
to peterlodri-sec/headroom
that referenced
this pull request
Jul 14, 2026
## Description Closes headroomlabs-ai#1616 Expose the proxy's durable `persistent_savings.lifetime` totals through `/metrics` so Prometheus/Grafana scrapes can read the same lifetime savings counters already visible in `/stats` and `/stats-history`. The existing runtime counters remain process-local: `headroom_tokens_saved_total` still resets with the proxy process. New `headroom_persistent_savings_*` counters are sourced from the `SavingsTracker` lifetime block. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Export durable lifetime savings counters from `PrometheusMetrics.export()`: - `headroom_persistent_savings_requests_total` - `headroom_persistent_savings_tokens_saved_total` - `headroom_persistent_savings_input_tokens_total` - `headroom_persistent_savings_input_cost_usd_total` - `headroom_persistent_savings_compression_savings_usd_total` - Add a restart regression proving runtime counters reset while persistent savings counters remain available from the same savings file. - Extend the existing `/stats-history` restart test with `/metrics` endpoint assertions. - Update metrics docs to distinguish runtime `headroom_tokens_saved_total` from lifetime `headroom_persistent_savings_tokens_saved_total`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text Local focused checks: $ rtk /usr/bin/env HEADROOM_REQUIRE_RUST_CORE=false PYTHONPATH=. /tmp/headroom-1616-testenv/bin/python -m pytest tests/test_proxy_cache_ttl_metrics.py::test_prometheus_metrics_export_includes_extended_fields tests/test_proxy_cache_ttl_metrics.py::test_prometheus_export_includes_persistent_savings_after_restart 2 passed, 1 warning in 0.19s $ rtk /tmp/headroom-1616-testenv/bin/python -m ruff check headroom/proxy/prometheus_metrics.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_savings_history.py All checks passed! $ rtk /tmp/headroom-1616-testenv/bin/python -m ruff format --check headroom/proxy/prometheus_metrics.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_savings_history.py 3 files already formatted $ rtk git diff --check # no output GitHub Actions: All non-skipped checks passed on PR headroomlabs-ai#1647, including lint, build, build-wheel, test (1-4), test-agno, test-extras, test-dashboard-ui, docker-native-e2e, docker-init-e2e, docker-wrap-e2e, security checks, merge-conflicts, and PR governance. ``` ## Real Behavior Proof - Environment: local macOS worktree, throwaway Python env at `/tmp/headroom-1616-testenv`, `PYTHONPATH=.`. - Exact command / steps: recorded a compressed request through `PrometheusMetrics.record_request()`, re-created `PrometheusMetrics` with the same `SavingsTracker` path, then exported `/metrics` text. - Observed result: runtime counters are zero after re-creating the metrics object, while `headroom_persistent_savings_tokens_saved_total` and related persistent counters still expose the durable lifetime values. - Not tested: full server-level pytest locally, because the local build is blocked by the known native `headroom._core`/`esaxx-rs` build issue (`fatal error: 'cstdint' file not found`). The app-level `/metrics` assertions passed in GitHub Actions. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes This intentionally does not rename or hydrate the existing runtime `headroom_tokens_saved_total` counter. That preserves the current process-local semantics and gives external dashboards a dedicated lifetime series that maps directly to `/stats.persistent_savings`. `mypy headroom` was not run as a standalone local command. CHANGELOG is N/A for this narrow proxy metrics fix unless maintainers prefer an entry.
JerrettDavis
pushed a commit
to peterlodri-sec/headroom
that referenced
this pull request
Jul 14, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>0.29.0</summary> ## [0.29.0](headroomlabs-ai/headroom@v0.28.0...v0.29.0) (2026-07-03) ### Features * **proxy:** add --lossless no-CCR mode with format-native compaction ([headroomlabs-ai#1721](headroomlabs-ai#1721)) ([c75ebde](headroomlabs-ai@c75ebde)) * **stats:** surface Codex WS compression counters in /stats summary ([headroomlabs-ai#1680](headroomlabs-ai#1680)) ([2fe19c3](headroomlabs-ai@2fe19c3)) * **transforms:** adaptive Otsu KEEP/DROP threshold (+ land relevance split on main) ([headroomlabs-ai#1726](headroomlabs-ai#1726)) ([eea667a](headroomlabs-ai@eea667a)) ### Bug Fixes * **bedrock:** fail fast when session-token auth lacks botocore ([headroomlabs-ai#1553](headroomlabs-ai#1553)) ([54cfa36](headroomlabs-ai@54cfa36)) * **bedrock:** route ARNs via converse, named AWS profiles, and au. re… ([headroomlabs-ai#1456](headroomlabs-ai#1456)) ([7d87aa2](headroomlabs-ai@7d87aa2)) * **ccr:** honor workspace dir for sqlite store ([headroomlabs-ai#1564](headroomlabs-ai#1564)) ([96e1dfe](headroomlabs-ai@96e1dfe)) * **claude:** surface Remote Control proxy incompatibility ([headroomlabs-ai#1610](headroomlabs-ai#1610)) ([4bf7f92](headroomlabs-ai@4bf7f92)) * **cli:** stop advertising unwired compression tuning env vars in banner ([headroomlabs-ai#1634](headroomlabs-ai#1634)) ([d5bf98d](headroomlabs-ai@d5bf98d)) * **codex:** avoid duplicate headroom provider config ([headroomlabs-ai#1431](headroomlabs-ai#1431)) ([ddd4adf](headroomlabs-ai@ddd4adf)) * **compression:** reject lossy unmarked tool output in unit router path ([headroomlabs-ai#1479](headroomlabs-ai#1479)) ([de24cd5](headroomlabs-ai@de24cd5)) * **cortex-code:** migrate to current Cortex REST API endpoints + add e2e benchmarks ([headroomlabs-ai#1474](headroomlabs-ai#1474)) ([f00ace6](headroomlabs-ai@f00ace6)) * **dashboard:** align token savings headline denominator ([headroomlabs-ai#1653](headroomlabs-ai#1653)) ([646e705](headroomlabs-ai@646e705)) * **dashboard:** derive per-project setup URL from live origin ([headroomlabs-ai#1511](headroomlabs-ai#1511)) ([e035aef](headroomlabs-ai@e035aef)) * **detection:** contain unidiff panic on orphaned +++ target line ([headroomlabs-ai#1548](headroomlabs-ai#1548)) ([e386c09](headroomlabs-ai@e386c09)) * **evals:** CJK-aware F1 tokenization + token estimation ([headroomlabs-ai#1527](headroomlabs-ai#1527)) ([99a8540](headroomlabs-ai@99a8540)) * **install:** close parent log fd in start_detached_agent ([headroomlabs-ai#1576](headroomlabs-ai#1576)) ([816cb85](headroomlabs-ai@816cb85)) * **install:** use Windows-safe PID liveness probe in runtime_status ([headroomlabs-ai#1544](headroomlabs-ai#1544)) ([headroomlabs-ai#1560](headroomlabs-ai#1560)) ([6b227b9](headroomlabs-ai@6b227b9)) * **learn:** aggregate verbosity baselines across projects instead of overwriting ([headroomlabs-ai#1288](headroomlabs-ai#1288)) ([27a5468](headroomlabs-ai@27a5468)) * **mcp:** show lifetime totals and label rolling session scope in headroom_stats ([headroomlabs-ai#1428](headroomlabs-ai#1428)) ([1c0e152](headroomlabs-ai@1c0e152)) * **memory:** cap local embedder CPU thread oversubscription ([headroomlabs-ai#198](headroomlabs-ai#198)) ([headroomlabs-ai#1559](headroomlabs-ai#1559)) ([b84afbf](headroomlabs-ai@b84afbf)) * **memory:** singleflight LocalBackend init to stop cold-start races ([headroomlabs-ai#1691](headroomlabs-ai#1691)) ([bec47a1](headroomlabs-ai@bec47a1)) * **openclaw:** detect uv-installed headroom binary in ~/.local/bin ([headroomlabs-ai#1459](headroomlabs-ai#1459)) ([adaeb88](headroomlabs-ai@adaeb88)) * **opencode:** preserve custom OpenAI gateway paths ([headroomlabs-ai#1596](headroomlabs-ai#1596)) ([c19347c](headroomlabs-ai@c19347c)) * **opencode:** route native providers + load transport plugin, fix Serena context ([headroomlabs-ai#1573](headroomlabs-ai#1573)) ([ad0034f](headroomlabs-ai@ad0034f)) * preserve anthropic passthrough tool order ([headroomlabs-ai#1427](headroomlabs-ai#1427)) ([a932247](headroomlabs-ai@a932247)) * **proxy/auth:** match real Anthropic OAuth token prefix (sk-ant-oat) ([headroomlabs-ai#1672](headroomlabs-ai#1672)) ([8cddf9b](headroomlabs-ai@8cddf9b)) * **proxy:** expose persistent savings metrics ([headroomlabs-ai#1647](headroomlabs-ai#1647)) ([5fe4e7b](headroomlabs-ai@5fe4e7b)) * **proxy:** fail open when kompress saturation would exhaust pre-upstream budget ([headroomlabs-ai#1430](headroomlabs-ai#1430)) ([15ac650](headroomlabs-ai@15ac650)) * **proxy:** handle streaming CCR retrieval ([headroomlabs-ai#1451](headroomlabs-ai#1451)) ([d337e3b](headroomlabs-ai@d337e3b)) * **proxy:** include system/tools/sampling in cache key ([headroomlabs-ai#1473](headroomlabs-ai#1473)) ([312129a](headroomlabs-ai@312129a)) * **proxy:** preserve Responses passthrough bytes ([headroomlabs-ai#1598](headroomlabs-ai#1598)) ([2a34a82](headroomlabs-ai@2a34a82)) * **proxy:** strip Codex lite header on the HTTP /responses path ([headroomlabs-ai#1663](headroomlabs-ai#1663)) ([9fbd47b](headroomlabs-ai@9fbd47b)) * **proxy:** wire --compression-max-workers / HEADROOM_COMPRESSION_MAX_WORKERS ([headroomlabs-ai#1632](headroomlabs-ai#1632)) ([814ffa3](headroomlabs-ai@814ffa3)) * **savings:** count cache-read tokens in input cost estimate ([headroomlabs-ai#1429](headroomlabs-ai#1429)) ([72ade37](headroomlabs-ai@72ade37)) * skip Magika backend on x86 CPUs without AVX2 ([headroomlabs-ai#1162](headroomlabs-ai#1162)) ([64783d8](headroomlabs-ai@64783d8)) * **transforms/content-router:** route grep/log output away from HTML extractor ([headroomlabs-ai#1719](headroomlabs-ai#1719)) ([0d18ef2](headroomlabs-ai@0d18ef2)) * **transforms:** bound native content detection with a Windows watchdog ([headroomlabs-ai#575](headroomlabs-ai#575)) ([headroomlabs-ai#1563](headroomlabs-ai#1563)) ([95abca3](headroomlabs-ai@95abca3)) * Vertex AI support for Claude Code with ANTHROPIC_VERTEX_BASE_URL ([headroomlabs-ai#1393](headroomlabs-ai#1393)) ([cff7247](headroomlabs-ai@cff7247)) * **wrap:** detach the shared proxy on Windows so it survives an ungraceful agent close ([headroomlabs-ai#1464](headroomlabs-ai#1464)) ([6cba441](headroomlabs-ai@6cba441)) * **wrap:** preserve custom Vertex base URL ([headroomlabs-ai#1477](headroomlabs-ai#1477)) ([75427bb](headroomlabs-ai@75427bb)) * **wrap:** remove rtk instructions from Codex AGENTS.md on unwrap ([headroomlabs-ai#1604](headroomlabs-ai#1604)) ([c9d717c](headroomlabs-ai@c9d717c)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #1616
Expose the proxy's durable
persistent_savings.lifetimetotals through/metricsso Prometheus/Grafana scrapes can read the same lifetime savings counters already visible in/statsand/stats-history.The existing runtime counters remain process-local:
headroom_tokens_saved_totalstill resets with the proxy process. Newheadroom_persistent_savings_*counters are sourced from theSavingsTrackerlifetime block.Type of Change
Changes Made
PrometheusMetrics.export():headroom_persistent_savings_requests_totalheadroom_persistent_savings_tokens_saved_totalheadroom_persistent_savings_input_tokens_totalheadroom_persistent_savings_input_cost_usd_totalheadroom_persistent_savings_compression_savings_usd_total/stats-historyrestart test with/metricsendpoint assertions.headroom_tokens_saved_totalfrom lifetimeheadroom_persistent_savings_tokens_saved_total.Testing
pytest)ruff check .)mypy headroom)Test Output
Real Behavior Proof
/tmp/headroom-1616-testenv,PYTHONPATH=..PrometheusMetrics.record_request(), re-createdPrometheusMetricswith the sameSavingsTrackerpath, then exported/metricstext.headroom_persistent_savings_tokens_saved_totaland related persistent counters still expose the durable lifetime values.headroom._core/esaxx-rsbuild issue (fatal error: 'cstdint' file not found). The app-level/metricsassertions passed in GitHub Actions.Review Readiness
Checklist
Screenshots (if applicable)
N/A
Additional Notes
This intentionally does not rename or hydrate the existing runtime
headroom_tokens_saved_totalcounter. That preserves the current process-local semantics and gives external dashboards a dedicated lifetime series that maps directly to/stats.persistent_savings.mypy headroomwas not run as a standalone local command. CHANGELOG is N/A for this narrow proxy metrics fix unless maintainers prefer an entry.