fix(dashboard): revamp the web dashboard UI, merge upstream/main#2
Closed
khaosdoctor wants to merge 367 commits into
Closed
fix(dashboard): revamp the web dashboard UI, merge upstream/main#2khaosdoctor wants to merge 367 commits into
khaosdoctor wants to merge 367 commits into
Conversation
## Description
`SavingsTracker`'s two numeric-coercion helpers (`_coerce_int`,
`_coerce_float`) are the trust boundary every persisted savings counter
routes through, but they caught only `TypeError` and `ValueError`. Two
non-finite gaps slipped through:
1. **Uncaught `OverflowError` on load → proxy won't start.**
`json.loads` accepts bare `NaN`/`Infinity`, so a `proxy_savings.json`
holding a non-finite value flows `_sanitize_state` → `_coerce_int(inf)`
→ `int(float('inf'))`, which raises `OverflowError`. `_load_state` only
catches `JSONDecodeError`/`OSError`, so it escapes
`SavingsTracker.__init__` and the proxy fails to boot. (`float(10**400)`
raises `OverflowError` too.)
2. **`NaN`/`Infinity` passthrough → dashboard-breaking JSON.**
`float('nan')`/`float('inf')` never raise, so `_coerce_float` returned
them verbatim. They poison arithmetic/comparisons and serialize back to
`NaN`/`Infinity` literals — invalid JSON that the dashboard's
`JSON.parse` rejects. One bad write poisons every later start.
Fix at the trust boundary (~4 LOC): both helpers now also catch
`OverflowError`; `_coerce_float` rejects non-finite floats via
`math.isfinite`. Coercion fails open to safe defaults, so a poisoned
field loads as `0` (correct fail-open, not data loss).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `_coerce_int`: added `OverflowError` to the caught exceptions (every
non-finite dies inside `int()` as `ValueError` for nan or
`OverflowError` for inf).
- `_coerce_float`: added `OverflowError` to the caught exceptions and
now rejects non-finite results via `math.isfinite` before returning,
failing open to the default.
- Added `import math`.
- Added 2 tests in `tests/test_proxy_savings_history.py` (a unit test
for the helpers and an integration test for the
poisoned-`proxy_savings.json` startup-crash vector).
- CHANGELOG entry under `Unreleased → Fixed`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy_savings_history.py -k reject_non_finite # unmodified source (RED)
E OverflowError: cannot convert float infinity to integer
headroom/proxy/savings_tracker.py:109: in _coerce_int -> return max(int(value), 0)
$ pytest tests/test_proxy_savings_history.py # after fix
======================== 22 passed, 1 warning in 36.64s ========================
$ pytest tests/test_proxy_project_savings.py tests/test_proxy_cache_ttl_metrics.py
======================== 30 passed, 1 warning in 8.57s =========================
$ ruff check .
All checks passed!
$ ruff format --check headroom/proxy/savings_tracker.py tests/test_proxy_savings_history.py
2 files already formatted
$ mypy headroom
Success: no issues found in 406 source files
$ python rbp_nonfinite.py # manual real-behavior run
1) raw file has NaN/Infinity literals: True
SavingsTracker constructed OK; lifetime = {'requests': 1, 'tokens_saved': 0, 'compression_savings_usd': 0.0, 'total_input_tokens': 0, 'total_input_cost_usd': 0.0}
all lifetime values finite: True
2) persisted file has NO NaN/Infinity literal: True
persisted lifetime finite: True
persisted lifetime = {'requests': 2, 'tokens_saved': 40, 'compression_savings_usd': 0.0001, 'total_input_tokens': 100, 'total_input_cost_usd': 0.00025}
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0), Python 3.13.13, isolated worktree
venv (`uv sync --extra dev`), `HF_HUB_OFFLINE=1
LITELLM_LOCAL_MODEL_COST_MAP=true`.
- Exact command / steps: reproduced the crash on unmodified source
(`pytest ... -k reject_non_finite`), then after the fix ran a standalone
script that writes a `proxy_savings.json` containing `NaN`/`Infinity`,
constructs `SavingsTracker`, and calls
`record_request(total_input_tokens=float('inf'),
total_input_cost_usd=float('nan'))` before re-reading the persisted
file.
- Observed result: BEFORE — `OverflowError: cannot convert float
infinity to integer` at `headroom/proxy/savings_tracker.py:109`,
escaping construction. AFTER — construction succeeds; poisoned lifetime
loads as all-finite `0`; after the non-finite `record_request` the
persisted file contains no `NaN`/`Infinity` literal and every lifetime
value is finite (`tokens_saved: 40, total_input_tokens: 100,
total_input_cost_usd: 0.00025`).
- Not tested: no live end-to-end proxy HTTP run against a real provider
(exercised the tracker's public API directly); did not add an
`allow_nan=False` guard in `_save_locked` or inf-guard the
`_estimate_*_usd` cost helpers (see Additional Notes).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A — no user-visible UI change.
## Additional Notes
- **Considered and skipped** (kept the diff to one logical change):
`json.dumps(..., allow_nan=False)` in `_save_locked` would add a *new*
crash path — it raises `ValueError`, but `_save_locked` only catches
`OSError`, so a slipped-through non-finite would crash the write instead
of failing open. After this fix no non-finite reaches the payload.
Inf-guarding the `_estimate_*_usd` cost helpers is unnecessary —
realistic token counts × per-token cost cannot overflow to `inf`.
- Documentation checklist item is N/A (no docs beyond the CHANGELOG
entry).
- Pre-push `make ci-precheck` flakes on the unrelated Rust latency
benchmark (`classify_under_10us_per_call`) under machine load; this is a
Python-only change, so the push used `--no-verify` (CI re-runs it on
clean hardware).
… floor) (headroomlabs-ai#1771) ## Description The compression acceptance gate rejected any compression that saved less than ~15% (`min_ratio` interpolated 0.85 at low context pressure → 0.65 under pressure). That floor was a crude proxy for "big enough to justify busting the prefix cache," but it dropped genuine token savings — notably lossless code/log folds that shrink <15% (the `ratio_too_high` rejections). This makes the gate accept **any real shrink** (`ratio < 1.0`): any token saved is worth taking. The two guards that actually protect correctness are untouched: - **Reversibility gate** — lossy, unmarked tool output still stays verbatim (accuracy; headroomlabs-ai#1307). - **Net-cost policy** (`HEADROOM_NET_COST_POLICY=1`, opt-in) — precisely accounts for the prefix-cache-bust economics (savings × expected-reads vs one-time suffix re-write) when a session wants that protection. Lowering the two values back to `0.85`/`0.65` restores the savings floor. Closes # ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Performance improvement ## Changes Made - `ContentRouterConfig.min_ratio_relaxed`: `0.85 → 1.0` - `ContentRouterConfig.min_ratio_aggressive`: `0.65 → 1.0` - Gate now accepts any `compression_ratio < 1.0` at every context pressure; reversibility + net-cost guards unchanged. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality (existing gate-mechanism tests cover it; they pass explicit `min_ratio` values and are unaffected) - [ ] Manual testing performed ### Test Output ```text # content-router + compression suites (default-config paths): tests/test_transforms/test_content_router.py ......... 139 passed # broad compression sweep (-k compress/router/crush/kompress/lossless/ccr/savings/...): 1 failed, 1940 passed, 54 skipped in 181.25s # the 1 failure = test_lossless_mode::test_router_lossless_search_no_marker_and_recoverable # — a local fastembed-cache state-leak flake; passes in isolation (1 passed in 3.15s), # and is in lossless mode which bypasses this gate entirely. ruff check headroom/transforms/content_router.py -> All checks passed! mypy headroom/transforms/content_router.py -> Success: no issues found ``` ## Real Behavior Proof - Environment: local worktree, Python 3.12, `PYTHONPATH` pinned to the branch checkout. - Exact command / steps: ran the content-router acceptance-gate suites and a compression-adjacent sweep against the branch; verified the flaky test passes in isolation. - Observed result: gate-mechanism tests (explicit `min_ratio`) unaffected; no default-floor test regressed; blocks that previously produced `ratio_too_high` at ratios in `[0.85, 1.0)` are now accepted. - Not tested: no live end-to-end proxy run was performed for this specific change; the behavioral effect (more `router:*` acceptances, fewer `ratio_too_high`) is inferred from the gate logic + suite. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective (existing gate tests cover the mechanism) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (handled at release time) ## Additional Notes Deliberate tradeoff (discussed and chosen): without the net-cost policy enabled, accepting sub-15% wins can be net-negative on prompt-cached sessions, because compressing a block invalidates the cached suffix (a one-time re-write, at 1.25× on Anthropic). If that shows up in practice, enable `HEADROOM_NET_COST_POLICY=1` (the precise economics guard) or restore a floor by lowering the two `min_ratio_*` values.
…ate (headroomlabs-ai#1772) ## Description Unit-mismatch bug in the compression acceptance gate. `router.apply()` computes `compression_ratio` from `len(text.split())` (word count), but a **lossless** search/log fold (`compact_lossless`) saves **bytes** by collapsing a repeated path prefix into a single heading — word count stays flat or even *rises* (the heading adds a word). So the gate saw `ratio ≥ 1.0` and discarded every free, byte-recoverable win as `ratio_too_high`. (Raising the floor to 1.0 in headroomlabs-ai#1771 did **not** fix this — the word-ratio was already ≥ 1.0.) Measure lossless results (those whose `strategy_chain` carries a `lossless_*` entry) by **byte ratio** at the gate and in the result cache — the real saving. Lossy strategies are unchanged (word count tracks their token savings), and the reversibility gate is untouched (`LOG`/`SEARCH`/`DIFF` aren't in `LOSSY_UNMARKED_STRATEGIES`). The excluded-tool and bash-search paths already bypass this gate via `continue`; this fixes the **main strategy dispatch** (the lossless-mode `LOG`/`SEARCH`/`DIFF` path). Follow-up to headroomlabs-ai#1771. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - At the `apply()` acceptance gate: compute `accept_ratio` = byte ratio for lossless results (`strategy_chain` has `lossless_*`), else the existing word ratio. Gate + result-cache entry now use `accept_ratio`. - Added an end-to-end regression test that drives the full `router.apply()` path. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text tests/test_lossless_mode.py::test_router_apply_accepts_lossless_search_byte_measured PASSED tests/test_content_router_tool_role_reversibility.py .......... (10 passed) # broader (pre-move) sweep on the same change: tests/test_lossless_mode.py / test_transforms/test_content_router.py / test_lossless_excluded_compaction.py / test_bash_search_lossless_fold.py — 121 passed ruff check headroom/transforms/content_router.py -> All checks passed! mypy headroom/transforms/content_router.py -> Success: no issues found ``` ## Real Behavior Proof - Environment: local worktree, Python 3.12, `PYTHONPATH` pinned to the branch. - Exact command / steps: new regression test constructs a single-file grep result, runs it through `ContentRouter(lossless=True).apply(...)`, and asserts the tool output is byte-smaller and recovers exactly (`search_unheading(out) == original`). - Observed result: before this fix the fold was rejected (`out == original`, counted `ratio_too_high`); after, it's applied (`len(out) < len(original)`, marker-free, byte-exact recovery). The test also asserts the fold's word count is ≥ the original's, so the test is meaningless if "fixed" by word count. - Not tested: no live end-to-end proxy run; validated via the full `apply()` path in unit tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (handled at release time) ## Additional Notes Why prior tests missed it: `compress()` and `_apply_strategy_to_content` return the folded result directly and never touch the `apply()` acceptance gate, so the existing lossless-mode unit tests (which call those) passed while the real proxy path silently discarded the fold. The new test exercises `apply()` end-to-end.
…eadroomlabs-ai#1716) ## Description `headroom wrap opencode` (and `headroom install opencode`) injects a `provider.headroom` block into the OpenCode config, but the block contained **no `models` map**. OpenCode only resolves `<provider>/<model>` ids that are listed in a custom provider's `models` map, so every documented `headroom/*` model (see `plugins/opencode/README.md`) failed with: ```text Error: Model not found: headroom/claude-sonnet-4-6. ``` This PR adds the model map (mirroring `DEFAULT_MODELS` in `plugins/opencode/src/provider.ts` and the README table) via a single shared `headroom_provider_entry()` helper used by all three injection sites. It also fixes a latent bug in the TS helper `createHeadroomProvider`, which prefixed model **keys** with `headroom/` — OpenCode would have registered them as `headroom/headroom/<id>`. Not addressed here (flagged for maintainers): the `headroom-opencode` npm package referenced by the plugin docs is not published to npm (registry 404), so the transparent-transport interception path (which would capture `github-copilot/*` traffic in the dashboard) still depends on a locally built `plugins/opencode/dist/entry.opencode.js`. With this fix, the documented `headroom/*` provider route works, so wrapped OpenCode traffic is proxied and recorded when users select `headroom/*` models. Closes headroomlabs-ai#1657 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/providers/opencode/config.py`: added `HEADROOM_OPENCODE_MODELS` (claude-sonnet-4-6, claude-opus-4-6, claude-haiku-4-5-20251001, gpt-4o, gpt-4.1 — same names/limits as the TS plugin) and a `headroom_provider_entry(port)` helper that includes the `models` map; `_render_provider_block` and `inject_opencode_provider_config` now use it instead of duplicating the provider dict. - `headroom/providers/opencode/runtime.py`: `build_opencode_config_content` reuses `headroom_provider_entry()` so `OPENCODE_CONFIG_CONTENT` exposes the models too. - `plugins/opencode/src/provider.ts`: `createHeadroomProvider` no longer prefixes model keys with `headroom/` (OpenCode namespaces model ids by provider key; keys must be bare ids). - `tests/test_providers_opencode_config.py`: assertions that the injected provider block and `build_opencode_config_content` output contain a `models` map with bare-id keys including `claude-sonnet-4-6`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_providers_opencode_config.py -q 1 failed, rest passed — test_build_launch_env_with_project is a pre-existing Windows-only failure (json.dumps escapes backslashes in the plugin path); it fails identically on upstream/main without this change and passes on Linux. $ ruff check headroom/providers/opencode tests/test_providers_opencode_config.py All checks passed! $ ruff format --check . 5 files already formatted $ mypy headroom --ignore-missing-imports Success (notes only, no errors) $ cd plugins/opencode && npm run typecheck && npm test tsc --noEmit: OK Test Files 2 passed (2) Tests 13 passed (13) ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, Node v26.3.0, this branch with the Rust core built locally. - Exact command / steps: `python -c "from headroom.providers.opencode.runtime import build_opencode_config_content; import json; print(json.dumps(build_opencode_config_content(port=8787, include_mcp=False)['provider']['headroom'], indent=1))"` - Observed result: the generated `headroom` provider block now contains `"models"` with bare-id keys (`claude-sonnet-4-6`, `claude-opus-4-6`, `claude-haiku-4-5-20251001`, `gpt-4o`, `gpt-4.1`), each with name and context/output limits; previously the block had no `models` key, which is exactly why OpenCode returned `Model not found: headroom/claude-sonnet-4-6`. - Not tested: a live `opencode run` round-trip against a real OpenCode install (no OpenCode binary in this environment); dashboard event capture for `github-copilot/*` models via the transport plugin (blocked on the unpublished `headroom-opencode` artifact, see Description). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - Docs: `plugins/opencode/README.md` already documents these models; no doc change needed. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Description Wire output shaping for OpenAI Responses traffic across HTTP `/v1/responses` and Codex WebSocket `response.create` frames. The change adds provider-specific shaping for `instructions`, `reasoning.effort`, and `text.verbosity` while keeping Anthropic request mutation separate. Review follow-up: merged byte-faithful `/v1/responses` forwarding from headroomlabs-ai#1557 and marks shaped HTTP Responses payloads as `body_mutated=True`, so retry forwarding sends the shaped body instead of the original raw bytes. ## Type of Change - [ ] Bug fix (non-breaking change fixes an issue) - [x] New feature (non-breaking change adds functionality) - [ ] Breaking change (fix or feature would cause existing functionality change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added OpenAI Responses output shaping for `instructions`, `reasoning.effort`, and `text.verbosity`. - Wired shaping into `/v1/responses` HTTP and Codex WebSocket `response.create` paths. - Preserved `x-headroom-bypass` and `HEADROOM_OUTPUT_HOLDOUT` behavior. - Added output-shaper transform labels for verbosity, text verbosity, reasoning effort, holdout control, and strata. - Updated output-savings conversation keys for Responses payloads and WS `response.create` envelopes. - Counted WS frame payload tokens when assigning output-savings strata. - Merged byte-faithful `/v1/responses` forwarding from headroomlabs-ai#1557 and kept shaped HTTP bodies on the mutated-forwarding path. - Added tests for classification, shaping, holdout, bypass, labels, WS strata, and byte-faithful forwarding compatibility. - Updated `CHANGELOG.md` for OpenAI Responses output-shaping support. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_openai_codex_ws_lifecycle.py tests/test_openai_responses_output_shaper.py tests/test_codex_responses_passthrough_bytes.py tests/test_output_shaper.py tests/test_output_savings.py -q 110 passed, 1 warning in 1.49s $ uv run --extra dev ruff check headroom/proxy/handlers/openai.py tests/test_openai_responses_output_shaper.py tests/test_codex_responses_passthrough_bytes.py tests/test_openai_codex_ws_lifecycle.py tests/test_output_shaper.py tests/test_output_savings.py All checks passed! $ git diff --check No whitespace errors. ``` ## Real Behavior Proof - Environment: local macOS checkout, branch `output-shaper-openai-responses`. - Exact command / steps: ran targeted pytest, ruff, and diff checks listed above. - Observed result: targeted tests passed with an existing FastAPI TestClient deprecation warning; ruff passed; diff check passed. - Not tested: full repository test suite, live OpenAI traffic, browser dashboard rendering, full `mypy headroom`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows project's style guidelines - [x] I performed self-review of my code - [x] I commented my code, particularly in hard-to-understand areas - [x] I made corresponding changes to documentation - [x] My changes generate no new warnings - [x] I added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I updated `CHANGELOG.md` if applicable ## Screenshots N/A ## Additional Notes - Non-applicable Type Change items are left unchecked. - The pytest warning comes from `fastapi.testclient` importing Starlette TestClient and was not introduced by this change. - `CHANGELOG.md` includes entries for OpenAI Responses output-shaping support and byte-faithful `/v1/responses` forwarding compatibility. --------- Co-authored-by: obchain <riteshnikhoriya94@gmail.com>
## Description The request-path compression executor currently uses asyncio-style I/O sizing for CPU-bound Kompress work. When `compression_max_workers` is unset, `HeadroomProxy.__init__` resolves the pool to `min(32, cpu * 4)`, so an eight-core host can run 32 simultaneous compression workers that all contend for real CPU. This changes only the automatic request-path default to one worker per reported CPU while preserving the existing explicit override path from `--compression-max-workers` and `HEADROOM_COMPRESSION_MAX_WORKERS`. Closes headroomlabs-ai#1635 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Cap the automatic request-path compression executor default at `max(1, os.cpu_count() or 1)`. - Preserve explicit `compression_max_workers` values, including the existing clamp to at least one worker. - Keep CLI help, `ProxyConfig` comments, and nearby test documentation aligned with the CPU-bound default. - Update the focused compression executor regression so the default contract documents CPU-bound sizing, and keep the existing Codex compression stress guard stable when p50 rounds to zero. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_codex_ws_compression_scheduler.py tests/test_proxy_compression_executor.py tests/test_cli_proxy_improvements.py::TestCompressionMaxWorkers -q`) - [x] Linting passes (`uv run ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py tests/test_cli_proxy_improvements.py tests/test_proxy_compression_executor.py tests/test_codex_ws_compression_scheduler.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_codex_ws_compression_scheduler.py tests/test_proxy_compression_executor.py tests/test_cli_proxy_improvements.py::TestCompressionMaxWorkers -q 16 passed, 1 skipped, 1 warning in 6.13s $ uv run ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py tests/test_cli_proxy_improvements.py tests/test_proxy_compression_executor.py tests/test_codex_ws_compression_scheduler.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python environment from `uv sync --extra dev`, no provider credentials needed. - Exact command / steps: construct `HeadroomProxy` with `compression_max_workers=None`, inspect `proxy.compression_max_workers` and `/health` `runtime.compression_executor`. - Observed result: the automatic request-path pool resolves to reported CPU count, while explicit overrides still resolve to the configured value and report `source: explicit`. - Not tested: multi-session wall-clock benchmark under live Kompress load. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No `CHANGELOG.md` edit: this repo generates changelog entries from conventional commits. This intentionally does not touch the background compression executor surface covered by headroomlabs-ai#1633.
…-ai#1802) ## Description Codex `/v1/responses` WebSocket frames could spend the full global compression timeout before falling through unchanged, then report only a generic `compression_exception` reason. That made a recoverable timeout look like an opaque compression failure and left Codex users waiting around 30 seconds for frames that did not produce useful compression. This keeps the existing compression executor, adds a Codex WS-specific compression timeout bound, and records timeout fallback distinctly from other compression exceptions. Closes headroomlabs-ai#922. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Bound Codex Responses WebSocket frame compression with a WS-specific timeout. - Pass that timeout through the existing compression executor instead of adding a parallel executor path. - Record timeout passthrough with `compression_timeout` instead of the generic compression exception reason. - Preserve generic `compression_exception` for non-timeout failures. - Add coverage for first-frame timeout bounds, timeout reason logging, generic exception preservation, and later-frame failed metrics. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_openai_codex_ws_lifecycle.py tests/test_codex_ws_compression_scheduler.py tests/test_compression_observability.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py tests/test_codex_ws_compression_scheduler.py tests/test_compression_observability.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_openai_codex_ws_lifecycle.py -q 22 passed in 1.38s uv run pytest tests/test_codex_ws_compression_scheduler.py tests/test_compression_observability.py -q 14 passed, 1 skipped in 3.63s uv run pytest tests/test_openai_codex_ws_lifecycle.py tests/test_codex_ws_compression_scheduler.py tests/test_compression_observability.py -q 36 passed, 1 skipped in 2.09s uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py tests/test_codex_ws_compression_scheduler.py tests/test_compression_observability.py All checks passed! uv run ruff format headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py --check 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows, Python through the project `uv` environment. - Exact command / steps: run the focused Codex WS timeout regressions with small monkeypatched timeout values. - Observed result: base uses the global timeout path or reports only generic `compression_exception`; head passes with Codex WS timeout bounded to the smaller WS cap, logs `compression_timeout` for timeout fallback, preserves `compression_exception` for non-timeout failures, and records failed metrics for later-frame timeout fallback. - Not tested: live Codex Desktop traffic against paid OpenAI credentials. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No changelog entry is needed for this request-path bug fix. Type checking was not part of the focused local validation for this Python-only change. Live Codex Desktop validation is not included because the regression is covered at the handler boundary.
) ## Description Unified diff content could skip compression when the router reached the DIFF strategy with no question context. `DiffCompressor.compress()` defaulted omitted context to an empty string, but explicit `None` still crossed into the Rust boundary and raised before any compression result could be produced. The router also had a DEBUG-only crash path because it measured `len(context)` before DIFF dispatch. This normalizes `None` at the router entry and at the DIFF wrapper boundary so direct and routed diff compression both send a string context to Rust. Closes headroomlabs-ai#1798. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Normalize `None` context to `""` before router debug logging and compression dispatch. - Normalize `None` context to `""` again before calling the Rust diff compressor. - Add regressions for explicit `None`, omitted context, non-empty context preservation, and DEBUG-enabled router DIFF dispatch. - Keep DIFF fallback behavior unchanged so patch-shaped content is not routed through a lossy fallback. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_transforms/test_diff_compressor.py tests/test_transforms/test_content_router.py -q`) - [x] Linting passes (`uv run ruff check headroom/transforms/diff_compressor.py headroom/transforms/content_router.py tests/test_transforms/test_diff_compressor.py tests/test_transforms/test_content_router.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_transforms/test_diff_compressor.py tests/test_transforms/test_content_router.py -q 86 passed in 3.09s uv run pytest tests/test_transforms/test_content_router.py -q 55 passed in 2.84s uv run ruff check headroom/transforms/diff_compressor.py headroom/transforms/content_router.py tests/test_transforms/test_diff_compressor.py tests/test_transforms/test_content_router.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python through the project `uv` environment. - Exact command / steps: run the new DIFF context regressions against base and head. - Observed result: base fails explicit `None` at the fake Rust boundary with `AssertionError: Rust diff compressor received None context`; head passes explicit `None`, omitted context, non-empty context, and DEBUG-enabled router dispatch. - Not tested: native Rust internals beyond the Python wrapper boundary. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No changelog entry is needed for this narrow wrapper and router bug fix. Type checking was not part of the focused local validation for this Python-only change.
## Description Adds provider-only HTTP proxy configuration for upstream LLM calls without setting process-wide proxy environment variables. `--http-proxy` and `HEADROOM_HTTP_PROXY` are scoped to the proxy server's provider HTTPX clients, and HTTP/2 is disabled for those clients when the proxy is set so HTTPS provider APIs can tunnel through CONNECT. Using process env vars such as `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, or `NO_PROXY` would also affect HTTPX, but those vars are inherited by tool executions, so this keeps proxy routing out of the global environment. Closes: N/A ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `--http-proxy` with `HEADROOM_HTTP_PROXY` fallback. - Passed the proxy URL only into provider HTTPX clients. - Disabled provider HTTP/2 when the proxy is configured. - Preserved the new setting through direct server startup and multi-worker config serialization. - Documented the flag/env var and why global `HTTP_PROXY`-style vars are not suitable for provider-only routing. - Added an Unreleased changelog entry. - Added coverage for CLI/env wiring, worker serialization, and HTTPX client options. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen pytest tests/test_cli_proxy_improvements.py tests/test_proxy_scalability.py ============================== 72 passed in 5.95s ============================== $ uv run --frozen ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py tests/test_cli_proxy_improvements.py tests/test_proxy_scalability.py All checks passed! $ uv run --frozen mypy headroom --ignore-missing-imports Success: no issues found in 406 source files $ env -u HTTP_PROXY -u http_proxy npm --prefix docs run types:check [MDX] generated files in 6.351916000000074ms Generating route types... [MDX] generated files in 5.813166999999794ms ✓ Types generated successfully $ git diff --check # no output ``` ## Real Behavior Proof - Environment: local provider setup that requires outbound LLM traffic through an HTTP proxy - Exact command / steps: ran focused pytest, Ruff, mypy, docs `types:check`, and `git diff --check` after rebasing the branch onto `origin/main`; reviewed the docs and changelog diffs; actively used the new proxy setting locally for a provider that requires proxied egress - Observed result: CLI/env/config tests passed; static checks passed; docs type generation passed; local provider traffic can be routed through the provider-only proxy setting without exporting global proxy variables to tool executions - Not tested: broad provider matrix across every supported upstream ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. CLI/backend/docs update only. ## Additional Notes The branch keeps implementation, docs, changelog, and formatting changes in separate commits.
…adroomlabs-ai#1817) ## Description The proxy wrote the full savings state to disk on every request: a `json.dumps` of up to 5000 history entries plus a blocking `os.fsync`, run under the shared metrics event loop. Concurrent sessions queued behind whichever request was mid-save. This batches the write so the hot path stops paying that cost every time. Serialize is the dominant part of that cost (about 57% in measurement) and it holds the GIL, so moving the write to a worker thread can't overlap it with the loop, and batching only the `fsync` caps the win at about 28%. Cutting how often the whole state is written is the lever that helps. Durability holds where it matters. `/stats`, `/stats-history`, and CSV export read in-memory state, so they never go stale. The on-disk file only feeds restart-survival: graceful shutdown flushes the tail, and a hard crash loses at most 24 requests' lifetime delta on the proxy path. A flush still does the durable temp-write, `fsync`, and atomic rename, only less often. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `SavingsTracker` gains `save_flush_every` (default 1, so direct and CLI callers keep persisting on every call). A counter throttles the existing `_save_locked`, and `flush()` forces a write. - The proxy constructs the tracker with `save_flush_every=25` at its one production construction site (`prometheus_metrics.py`). Graceful shutdown flushes the tail (`server.py`). - Every write is a full-state snapshot, so a skipped save loses nothing: the next write is a complete replacement. `_save_locked` resets the throttle counter only after a durable write (and in the stateless branch), so a transient write failure leaves the counter untouched and the next record retries instead of waiting a fresh window. - Tests: one existing savings test that read the on-disk file mid-session now flushes first. New tests prove the batched final on-disk state equals the immediate (`flush_every=1`) state on identical inputs, that a failed `mkstemp` retries on the next record rather than consuming a full window, and that `HeadroomProxy.shutdown()` flushes the tracker so a graceful stop never drops the batched tail. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Regression across the fix's full blast radius: the touched savings suite plus # every test exercising savings_tracker, prometheus_metrics, or the server.py # shutdown surface (one construction site, one flush call site, confirmed repo-wide). $ uv run pytest tests/test_proxy_savings_history.py tests/test_proxy_project_savings.py \ tests/test_backend_streaming_cache_metrics.py tests/test_pricing_litellm.py \ tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_hooks_regression.py \ tests/test_compression_observability.py tests/test_observability_metrics.py \ tests/test_prometheus_stage_timing_concurrency.py tests/test_request_outcome.py \ tests/test_telemetry_context.py tests/test_provider_codex_runtime.py \ tests/test_proxy_eager_preload_bind.py tests/test_proxy_pipeline_lifecycle.py \ tests/test_proxy_scalability.py tests/test_proxy_warmup.py \ tests/test_proxy/test_bedrock_passthrough.py -q 195 passed $ uv run ruff check . All checks passed! $ uv run ruff format --check . 1044 files already formatted $ uv run mypy headroom Success: no issues found in 406 source files ``` ## Real Behavior Proof - Environment: Python 3.13.13, macOS (Apple M4, APFS), isolated worktree venv, `HF_HUB_OFFLINE=1 LITELLM_LOCAL_MODEL_COST_MAP=true`. Branch `fix/savings-tracker-batch-save` at `47c6ce9d`, 3 commits on `upstream/main` `e8151f05`. - Exact command / steps: extracted the pre-fix git blobs (`e8151f05` base, `ddfd6626` batch-only) into standalone modules and ran the new tests' logic against them for failing-before proof. Booted the real app via `create_app()` + `TestClient`, drove 10 `record_request` calls, then exited the lifespan to trigger the real `HeadroomProxy.shutdown()` flush. Ran a 3-trial N=1000-call micro-benchmark seeding a `SavingsTracker` with a full 5000-entry history for `save_flush_every=1` against `=25`, counting `os.fsync` syscalls. - Observed result: BEFORE (`save_flush_every=1`) was 4.975 ms/call with 1000 fsync syscalls. AFTER (`save_flush_every=25`) was 1.090 ms/call with 40 fsyncs, a 4.56x speedup and exactly 25x fewer fsyncs. Base `e8151f05` rejects `save_flush_every` with `TypeError` and saves on 10 of 10 calls. The pre-retry blob `ddfd6626` fails the retry test with `AssertionError` at `assert path.exists()` after the 6th call, while HEAD `47c6ce9d` passes it. A real ASGI-lifespan shutdown persisted all 10 buffered requests that were absent from disk before shutdown. - Not tested: the hard-crash loss window, bounded to at most 24 requests by design, is not reproduced with a real crash. Absolute per-call timing varies by hardware, though the fsync reduction is deterministic and exact. The end-to-end shutdown-flush proof above was an ad hoc real run, and a dedicated `shutdown()` to `flush()` unit guard ships in this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. Proxy-internal persistence change, no user-facing surface. ## Additional Notes No issue filed. It surfaces to users as the proxy feeling slow under load rather than a nameable bug, so there was nothing to link. Docs and CHANGELOG left unchecked: the flag is internal and the default behavior is unchanged, so nothing user-facing moved. Touches the same file as headroomlabs-ai#1764 (parent-dir fsync) but the changes don't overlap, so it rebases cleanly whichever lands first. Pushed with `git push --no-verify`: the `make ci-precheck` pre-push hook runs `pip install -e .`, which fails with "No module named pip" in the uv-managed worktree venv (environment quirk, not the diff). All Rust tests (846+) and the Python suite (195) passed in that same hook run before the pip step. --------- Co-authored-by: Omar Gerardo <ogerardo@MacBook-Air.local>
…#1804) ## Description The session dashboard repeats the same savings and performance numbers in adjacent places. `proxy_compression_saved` appears in several captions and detail rows, and average overhead and TTFB appear both in the hero area and again in Performance without adding new context. This narrows the non-hero dashboard presentation so repeated session metrics have one visible home plus decomposition where it adds information. It leaves `/stats`, savings math, cache attribution, and the hero proxy savings card unchanged. Refs headroomlabs-ai#960 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Removed redundant non-hero session-view captions that restated proxy-compression token counts without adding a new dimension. - Kept canonical homes for proxy compression and token usage details. - Preserved Performance range context while avoiding adjacent restatement of hero averages. - Added a static dashboard regression for repeated session metrics. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy_dashboard_stats_cache.py -q`) - [x] Linting passes (`uv run ruff check tests/test_proxy_dashboard_stats_cache.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy_dashboard_stats_cache.py -q 12 passed, 1 skipped, 1 warning in 19.24s $ uv run ruff check tests/test_proxy_dashboard_stats_cache.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python environment from `uv sync --extra dev`, browserless dashboard HTML inspection. - Exact command / steps: load `get_dashboard_html()` in the focused dashboard stats test and assert removed duplicate captions stay removed while canonical metric owners remain present. - Observed result: session-view repeated savings and performance labels no longer duplicate the same numbers without context. - Not tested: full browser screenshot and history-view de-duplication. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No `CHANGELOG.md` edit: this repo generates changelog entries from conventional commits. This intentionally avoids the hero proxy savings card already covered by headroomlabs-ai#927 and headroomlabs-ai#1649, and it does not fold provider cache discount into Headroom-value savings.
## Description Fixes OpenCode Headroom MCP configuration across wrap, MCP install/status/uninstall, and persistent install docs/CLI. OpenCode was being configured to use a remote HTTP MCP endpoint at `/mcp`, but the Headroom proxy does not expose MCP there. The correct OpenCode configuration is a local stdio MCP server that runs `headroom mcp serve`. Closes headroomlabs-ai#1380 ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [x] Documentation update - [x] Tests ## Changes Made - Changed OpenCode MCP registration to emit `type: "local"` with `command: ["headroom", "mcp", "serve"]`. - Changed OpenCode MCP environment serialization from `env` to OpenCode's `environment` key, while still reading legacy `env` entries. - Removed generated remote `/mcp` entries from OpenCode wrap/runtime config. - Made `wrap opencode --no-mcp` skip persistent `mcp.headroom` injection. - Kept provider-only OpenCode config injection from writing MCP; MCP persistence is owned by the registrar path. - Made `headroom mcp status` and `headroom mcp uninstall` use the registrar lifecycle so OpenCode is covered. - Added `opencode` to persistent install `--target` choices. - Clarified OpenCode persistent install docs to use `--scope provider` for direct `opencode.json` edits. - Added regression coverage for registrar serialization, wrap behavior, runtime config, provider-scope install, MCP CLI lifecycle, and install target parsing. ## Testing - [x] `rtk .venv/bin/python -m pytest tests/test_mcp_registry tests/test_cli/test_mcp.py tests/test_cli/test_wrap_opencode.py tests/test_providers_opencode_config.py tests/test_providers_opencode_install.py tests/test_install -q` - [x] Result after absorbing headroomlabs-ai#1381 overlap: `263 passed, 1 skipped` - [x] Targeted Ruff check passed for the changed Python/test files. - [x] Targeted Ruff format check passed for the changed Python/test files. - [x] Isolated HOME smoke tests with real `opencode mcp list --pure`. ## Real Behavior Proof - `headroom mcp install --agent opencode --proxy-url http://127.0.0.1:9000 --force` against an isolated HOME wrote a valid local OpenCode MCP entry with `environment.HEADROOM_PROXY_URL`. - `opencode mcp list --pure` against that isolated HOME connected to `headroom mcp serve`. - `headroom wrap opencode --prepare-only --no-rtk --no-serena --port 9001` wrote local MCP plus provider config. - `headroom wrap opencode --prepare-only --no-rtk --no-serena --no-mcp --port 9002` wrote provider config without `mcp.headroom`. - Generated runtime `OPENCODE_CONFIG_CONTENT` was accepted by `opencode mcp list --pure`; `include_mcp=False` reported no MCP servers. - `headroom mcp status` detected the isolated OpenCode config and read the custom proxy URL. - `headroom mcp uninstall` removed `mcp.headroom` from the isolated OpenCode config while leaving provider config intact. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review
## Description `headroom wrap copilot --subscription` can currently trust the token-exchange host for individual Copilot seats, which routes newer responses-API models like `gpt-5.4` to `api.individual.githubcopilot.com` and reproduces the transient `502` retry loop from issue headroomlabs-ai#1694. This normalizes that public individual-seat host back to the generic Copilot API host while preserving dedicated business or explicitly pinned hosts. Closes headroomlabs-ai#1694. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Normalized exchanged Copilot subscription hosts through the existing public-host classifier instead of trusting the raw token-exchange payload. - Added a regression proving `api.individual.githubcopilot.com` downgrades to `https://api.githubcopilot.com` for subscription routing. - Added a wrap-level regression proving subscription launches export the normalized host into the proxy env. - Preserved business-host and explicit `GITHUB_COPILOT_API_URL` routing behavior. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_copilot_auth.py -q`) - [x] Unit tests pass (`uv run pytest tests/test_cli/test_wrap_copilot.py -q`) - [x] Linting passes (`uv run ruff check headroom/copilot_auth.py tests/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_copilot_auth.py -q 57 passed, 1 warning in 0.34s uv run pytest tests/test_cli/test_wrap_copilot.py -q 31 passed, 1 warning in 0.32s uv run ruff check headroom/copilot_auth.py tests/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py All checks passed! uv run ruff format --check headroom/copilot_auth.py tests/test_copilot_auth.py tests/test_cli/test_wrap_copilot.py 3 files already formatted ``` ## Real Behavior Proof - Environment: Windows, Python `uv` environment, mocked Copilot token-exchange and wrap launch surfaces. - Exact command / steps: run the focused Copilot auth and wrap regression tests after teaching subscription token-exchange routing to normalize the public individual-seat host. - Observed result: exchanged subscription tokens that advertise `https://api.individual.githubcopilot.com` now route through `https://api.githubcopilot.com`, while business-host and explicit-host pin cases stay unchanged. - Not tested: a live GitHub Copilot subscription request against the upstream service. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes This is intentionally scoped to host selection for exchanged Copilot subscription tokens. It does not change token discovery, token pinning, or non-subscription OAuth routing.
## Description During proxy shutdown, an in-flight retrying request can currently stay asleep inside `_retry_request()` and keep the client socket hanging until the retry timer expires or an external supervisor kills the process. This wires retry backoff to a proxy-scoped shutdown event so shutdown interrupts those waits immediately and returns a clear `503` response instead of leaving the request stalled. Closes headroomlabs-ai#1821. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a proxy-scoped shutdown event in `headroom/proxy/server.py`. - Cleared that event at startup and set it at shutdown before teardown proceeds. - Replaced both retry-backoff sleeps with a helper that wakes on either timeout or shutdown. - Returned a shutdown `503` with `retry-after: 0` when shutdown interrupts retry backoff. - Stopped the shutdown interruption logs from falling back to the raw upstream URL when no safe path string is available. - Added focused regressions for retry-backoff interruption and shutdown event signaling. - Updated the existing Retry-After tests to observe the new shutdown-aware wait helper instead of the old raw sleep hook. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy_handler_helpers.py tests/test_proxy_pipeline_lifecycle.py -q`) - [x] Unit tests pass (`uv run pytest tests/test_proxy_retry_429.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/server.py tests/test_proxy_handler_helpers.py tests/test_proxy_retry_429.py tests/test_proxy_pipeline_lifecycle.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_proxy_handler_helpers.py tests/test_proxy_pipeline_lifecycle.py -q 32 passed, 1 warning in 13.05s uv run pytest tests/test_proxy_retry_429.py -q 10 passed, 1 warning in 1.12s uv run ruff check headroom/proxy/server.py tests/test_proxy_handler_helpers.py tests/test_proxy_retry_429.py tests/test_proxy_pipeline_lifecycle.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, project `uv` environment, focused proxy retry and shutdown regressions. - Exact command / steps: copy the updated shutdown regression files into a detached `origin/main` worktree and run `tests/test_proxy_handler_helpers.py` plus `tests/test_proxy_pipeline_lifecycle.py`, then rerun those files on this branch and separately rerun `tests/test_proxy_retry_429.py` after updating the existing Retry-After tests to patch the shutdown-aware wait helper. - Observed result: base fails because retry backoff still returns the original `429` and `shutdown()` leaves the retry event unset; head passes the focused file, preserves the existing Retry-After assertions, and returns a shutdown `503` with `retry-after: 0` while signaling retry waiters during shutdown. - Not tested: live systemd-managed shutdown on Linux or a full VS Code / Claude Code session. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes This is intentionally scoped to retry backoff during shutdown. It does not try to cancel unrelated in-flight request work or change the broader retry policy outside shutdown.
…A7 lossy-after-fold (headroomlabs-ai#1818) ## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…mlabs-ai#1840) ## Description Strips dangling terminal-style model suffixes like `[1m]` from Anthropic-compatible model ids before Headroom forwards `/v1/messages` upstream. Closes headroomlabs-ai#1812 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Generalized `sanitize_anthropic_model_id()` so the existing dangling ANSI-style suffix cleanup applies to Anthropic-compatible non-Claude models, including `glm-5.2[1m]`. - Added a provider-level regression for `glm-5.2[1m] -> glm-5.2`. - Added a `/v1/messages` handler regression that captures the upstream request body and verifies Headroom forwards `glm-5.2`, not `glm-5.2[1m]`. ## Testing - [x] Unit tests pass (`pytest`) — focused local tests and full CI test matrix passed - [x] Linting passes (`ruff check .`) — local Ruff and CI lint passed - [x] Type checking passes (`mypy headroom`) — local mypy and CI lint passed - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ rtk proxy env HEADROOM_REQUIRE_RUST_CORE=false PYTHONPATH=/Users/vinaygupta/Desktop/git/headroom-fix-1812-1m-model-suffix /tmp/headroom-1812-testenv/bin/python -c '<inject local headroom._core test stub; pytest.main(["tests/test_providers/test_anthropic.py", "tests/test_proxy_anthropic_model_sanitization.py"])>' ============================= test session starts ============================== platform darwin -- Python 3.13.11, pytest-9.1.1, pluggy-1.6.0 -- /private/tmp/headroom-1812-testenv/bin/python collected 17 items tests/test_providers/test_anthropic.py::TestAnthropicModelSanitization::test_sanitize_model_id_removes_ansi_escape_sequences PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelSanitization::test_sanitize_model_id_removes_displayed_style_suffix PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelSanitization::test_sanitize_model_metadata_cleans_nested_model_ids PASSED tests/test_providers/test_anthropic.py::TestAnthropicTokenCounting::test_count_text_fallback PASSED tests/test_providers/test_anthropic.py::TestAnthropicTokenCounting::test_count_messages_basic PASSED tests/test_providers/test_anthropic.py::TestAnthropicTokenCounting::test_count_text_allows_literal_special_tokens PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_claude_sonnet PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_claude_opus PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_strips_ansi_model_suffix PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_get_context_limit_claude_5_family PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_supports_model_known PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_supports_model_prefix PASSED tests/test_providers/test_anthropic.py::TestAnthropicModelLimits::test_token_counter_cache_uses_sanitized_model_id PASSED tests/test_providers/test_anthropic.py::TestAnthropicCostEstimation::test_estimate_cost_basic PASSED tests/test_providers/test_anthropic.py::TestAnthropicCostEstimation::test_pricing_lookup_strips_ansi_model_suffix PASSED tests/test_providers/test_anthropic.py::TestAnthropicCostEstimation::test_pricing_claude_5_family PASSED tests/test_proxy_anthropic_model_sanitization.py::test_anthropic_messages_strips_local_1m_model_suffix_before_forwarding PASSED ======================== 17 passed, 3 warnings in 2.11s ======================== $ rtk uvx ruff check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py tests/test_proxy_anthropic_model_sanitization.py All checks passed! $ rtk uvx ruff format --check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py tests/test_proxy_anthropic_model_sanitization.py 3 files already formatted ``` The normal editable test command was attempted but did not reach test execution in this local checkout because the native extension build failed: ```text $ rtk uv run pytest tests/test_providers/test_anthropic.py tests/test_proxy_anthropic_model_sanitization.py × Failed to build `headroom-ai @ file:///Users/vinaygupta/Desktop/git/headroom-fix-1812-1m-model-suffix` warning: esaxx-rs@0.1.10: src/esaxx.cpp:620:10: fatal error: 'cstdint' file not found error: failed to run custom build command for `esaxx-rs v0.1.10` ``` ## Real Behavior Proof - Environment: local macOS worktree from current upstream `main`; Python 3.13.11 throwaway test environment; `HEADROOM_REQUIRE_RUST_CORE=false`; in-memory `headroom._core` stub used only to avoid the local missing native extension during Python-level tests. - Exact command / steps: POST a TestClient `/v1/messages` request with `{"model": "glm-5.2[1m]", ...}` and replace `_retry_request` with a test double that records the upstream body. - Observed result: the recorded upstream request body contains `{"model": "glm-5.2"}` and `mutation_reasons == ["sanitize_model_id"]`, so the mutated JSON body is serialized instead of forwarding the original bytes. - Not tested: live Z.AI credentials/provider call; full local pytest; local `mypy headroom`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes - All non-skipped GitHub Actions checks are green after the rebase onto `main`; skipped jobs are path-gated. - No code comments were added because the fix reuses the existing sanitizer and mutation-tracking path. - Documentation and CHANGELOG updates are N/A for this narrow proxy compatibility fix. - The local pytest warnings were from the throwaway environment/test tooling (`asyncio_mode`, Starlette TestClient deprecation, and the existing AnthropicProvider no-client warning), not from the changed code path.
) ## Description Pin the top-level Docker Compose proxy service to Headroom's canonical writable workspace under the existing `headroom_workspace` named volume. Closes headroomlabs-ai#1835 The dashboard's durable savings/history data is loaded from `proxy_savings.json` via `HEADROOM_WORKSPACE_DIR`; logs, session stats, TOIN, config, and default workspace state are also derived from that root. The top-level compose file already mounted `/home/nonroot/.headroom`, but it relied on image/user home resolution instead of exporting the canonical workspace env. This makes the official compose contract explicit and matches the Docker-native compose/runtime path behavior. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Set `HOME=/home/nonroot` for the top-level compose proxy service. - Set `HEADROOM_WORKSPACE_DIR=/home/nonroot/.headroom` and `HEADROOM_CONFIG_DIR=/home/nonroot/.headroom/config` so dashboard savings/history, logs, config, memory state, session stats, and TOIN resolve into the persisted named volume. - Added a regression test that locks the top-level compose persistence wiring. ## Testing - [x] Unit tests pass (`pytest`) — focused local tests and full CI test matrix passed - [x] Linting passes (`ruff check .`) — local Ruff and CI lint passed - [x] Type checking passes (`mypy headroom`) — local mypy and CI lint passed - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ rtk pytest tests/test_docker_compose_persistence.py Pytest: 1 passed $ rtk pytest tests/test_docker_compose_persistence.py tests/test_paths.py Pytest: 76 passed $ rtk uvx ruff check tests/test_docker_compose_persistence.py All checks passed! $ rtk docker compose config services: headroom-proxy: environment: HEADROOM_CONFIG_DIR: /home/nonroot/.headroom/config HEADROOM_HOST: 0.0.0.0 HEADROOM_WORKSPACE_DIR: /home/nonroot/.headroom HOME: /home/nonroot volumes: - type: volume source: headroom_workspace target: /home/nonroot/.headroom ``` Attempted broader proxy stats-history coverage, but this local checkout does not have the native extension built: ```text $ rtk pytest tests/test_docker_compose_persistence.py tests/test_paths.py tests/test_proxy_savings_history.py::test_stats_history_persists_across_restarts_and_stats_stays_compatible ModuleNotFoundError: No module named 'headroom._core' ``` Attempted project-managed Ruff, but `uv run` tried to build the editable package first and hit the known local native build issue before Ruff could execute: ```text $ rtk uv run ruff check tests/test_docker_compose_persistence.py error: failed to run custom build command for `esaxx-rs v0.1.10` fatal error: 'cstdint' file not found ``` ## Real Behavior Proof - Environment: local clean clone at current upstream `main`, branch `fix/1835-docker-compose-persistence`. - Exact command / steps: `rtk docker compose config` from the repo root. - Observed result: Compose renders `HOME`, `HEADROOM_WORKSPACE_DIR`, and `HEADROOM_CONFIG_DIR` under `/home/nonroot/.headroom`, and the `headroom_workspace` named volume targets that same path. - Not tested: full Docker image build or live `docker compose up` restart cycle; full pytest/mypy not run locally because this checkout lacks the built `headroom._core` extension. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes - All non-skipped GitHub Actions checks are green after the rebase onto `main`; skipped jobs are path-gated. - The dashboard's recent request table is still an in-memory tail and is expected to be empty after a proxy restart. This PR targets durable dashboard savings/history and other workspace-backed files. - `HEADROOM_LOG_FILE=/home/nonroot/.headroom/requests.jsonl` remains an optional operator setting; persisted request JSONL is not replayed into the dashboard after restart. - The docs/CHANGELOG checklist items are N/A for this narrow compose configuration fix.
…ions (headroomlabs-ai#1768) (headroomlabs-ai#1837) ## Description `headroom wrap claude` writes `env.ANTHROPIC_BASE_URL` (or the foundry/vertex variant) into a project's `.claude/settings.local.json` so daemon-spawned Claude Code workers route through the local Headroom proxy. Removal only happened in the wrap process's `finally:` block. An unclean exit — `SIGKILL`, OOM, reboot, or terminal/tmux close (`SIGHUP`, which was not caught; only `SIGINT`/`SIGTERM` were) — skipped that cleanup, so the entry persisted indefinitely. Every subsequent bare `claude` in that project then routed to the dead port and hung indefinitely retrying it. Closes headroomlabs-ai#1768 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `_write_claude_wrap_base_url` now optionally stamps a sidecar marker (`.claude/.headroom_wrap_marker.json`) recording the writer's pid/identity, the port, and the true prior value — kept out of `settings.local.json` itself so Headroom bookkeeping never shows up as a stray key in a file Claude Code's own config loader parses. - A shared `_identity_mismatch` helper (factored out of the existing `_marker_pid_reused` proxy-client-refcounting logic) lets a marker be judged stale: missing/invalid pid, dead pid, or a live pid whose identity doesn't match the recorded one (PID reuse after a crash). - `claude()` now checks for — and self-heals — a stale marker immediately before writing a fresh entry, restoring the recorded prior value instead of trusting a leftover from a dead session. - `claude()` now also registers a `SIGHUP` handler (guarded via `hasattr`, since Windows has none) alongside the existing `SIGTERM` handler, so terminal-close triggers the same cleanup/restore path. - `headroom unwrap claude` now reads the marker's recorded prior value before restoring, instead of unconditionally deleting the key — so a user's own pre-existing `ANTHROPIC_BASE_URL` (set before ever running `wrap`) isn't blindly wiped. - `headroom doctor` gained a new check (`check_wrap_marker_staleness`) that flags a stale project-local marker and points at `headroom unwrap claude` to clean it up — separate from the existing global-settings `check_claude_routing` check. - (Unrelated, pre-existing on `main`) reformatted `headroom/proxy/handlers/openai.py`, `tests/test_openai_codex_ws_lifecycle.py`, `tests/test_output_shaper.py` — whitespace/indentation only, no logic change — since they were already failing `ruff format --check .` on `main` before this branch touched anything, and the repo-wide lint gate blocks on it. Out of scope: `wrap --worktree` — no such flag or multi-worktree `.claude` handling exists anywhere in `wrap.py` today; not adding new surface for an aspirational scenario the issue mentions but that isn't implemented. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_cli/test_wrap_claude_base_url.py tests/test_cli/test_unwrap_claude.py tests/test_cli/test_wrap_stale_marker.py -q 42 passed $ pytest tests/test_cli -q 512 passed, 1 failed (test_wrap_codex_prepare_only_registers_serena_when_uvx_exists — confirmed to fail identically on a clean checkout of main with no changes applied; test-order flake, unrelated to this PR) $ ruff check . All checks passed! $ ruff format --check . 1047 files already formatted $ mypy headroom/cli/wrap.py headroom/cli/doctor.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: local checkout, Python 3.13, Windows. - Exact command / steps: wrote a base_url entry + marker via `_write_claude_wrap_base_url("https://github.com/khaosdoctor/headroom/pull/...,%20port=8787")`, then overwrote the marker's recorded pid with a value guaranteed not to be a live process (simulating the crash from the issue's own repro: `headroom wrap claude -- -p ok & ; kill -9 <wrap-pid>`). Ran `headroom.cli.doctor.check_wrap_marker_staleness()` against that path, then called `_check_and_clear_stale_wrap_marker()` (the same check `claude()` now runs before writing a fresh entry). - Observed result: `doctor`'s check correctly reports `WARN` naming the dead pid/port and pointing at `headroom unwrap claude`. The stale-check call then self-heals: in the "nothing existed before wrap" case the leaked entry is removed; in a second run seeded with a real pre-existing `ANTHROPIC_BASE_URL` (set before `wrap` ever ran), that original value is recovered instead of being deleted. In both cases the marker file is cleared afterward. - Not tested: actual OS-level signal delivery (`kill -HUP` against a real running `headroom wrap claude` subprocess) — the SIGHUP registration is exercised via a source-inspection test instead of a live signal, since spawning/killing the real CLI subprocess isn't practical in this environment; verified E2E via CI's `wrap-native` jobs (Ubuntu/macOS) which passed. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — CLI/backend fix, no UI surface. ## Additional Notes - Documentation checklist item left unchecked: no user-facing docs currently describe wrap's settings.local.json write/cleanup behavior in enough detail to need updating; happy to add a troubleshooting note if maintainers want one. - `wrap --worktree` handling is out of scope (see Changes Made) — flagging in case maintainers want it tracked as a separate follow-up issue. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…ical — stop token-mode cache busting (headroomlabs-ai#1850) The freeze path (both providers) emits the agent's ORIGINAL bytes for a frozen message, but the provider cached whatever we FORWARDED last turn (the compressed form). Forwarding original then mismatches the cached prefix and busts it from that point — re-creating the whole suffix. Measured on a real SWE-bench run: 100% of attributed misses were prefix_change, ~56% of ALL cache-writes were bust-induced (2.8M tokens), driving cache_create +150% and cost +41% vs baseline. Cache mode already avoided this via _extract_cache_stable_delta (replay the previously-forwarded prefix, compress only the delta). Token mode called apply(frozen_count) directly, which forwards original for the frozen region. Fix: add a shared, provider-agnostic overlay_cached_prefix() that replays the previously-forwarded (cached, compressed) prefix byte-identical, append-only guarded and idempotent, and apply it in BOTH the Anthropic and OpenAI handlers right before forwarding. This makes freezing byte-identical in every mode, so the only remaining difference between "token" and "cache" mode is how large a mutable (still-compressible) tail each leaves — not whether the frozen prefix busts the cache. Tests: - test_cache_prefix_overlay.py: the helper (replay, append-only guard, idempotence). - test_cross_turn_cache_safety.py: the invariant that was missing — drive the REAL tracker + freeze + overlay over multiple append-only turns against a simulated provider prefix cache and assert the forwarded prefix stays byte-identical turn-over-turn. Load-bearing: it fails (detects the bust) without the overlay. ## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. -->
…droomlabs-ai#1538) ## Description Fix `pip install headroom-ai` on Intel Mac (`x86_64-apple-darwin`). Source installs failed because `ort-sys 2.0.0-rc.12` (transitive via `fastembed`) does not ship prebuilt ONNX Runtime binaries for that target, causing maturin/cargo to exit during the wheel build. This PR mirrors the existing Windows fix: build the Rust core with `ort-load-dynamic`, pin `ORT_DYLIB_PATH` to the pip `onnxruntime` native library at import time, and publish Intel macOS wheels from CI. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `crates/headroom-core/Cargo.toml`: use `ort-load-dynamic` for `x86_64-apple-darwin` instead of `ort-download-binaries-rustls-tls`. - `headroom/_ort.py`: extend the ORT dylib pin hook to Intel macOS (`darwin` + `x86_64`), resolving `libonnxruntime*.dylib` from the pip `onnxruntime` package. - `.github/workflows/release.yml` and `.github/workflows/rust.yml`: add `macos-15-intel` / `x86_64-apple-darwin` wheel matrix entries. - `tests/test_release_workflows.py` and `tests/test_transforms/test_ort_dylib.py`: update/add coverage for the new target and dylib pin behavior. - `README.md`: note that prebuilt wheels are published for Intel macOS. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_transforms/test_ort_dylib.py \ tests/test_release_workflows.py::test_fastembed_uses_dynamic_ort_on_windows \ tests/test_release_workflows.py::test_build_wheels_matrix_includes_intel_macos_with_dynamic_ort -q .......................... [100%] 10 passed in 0.18s $ maturin build --release -o /tmp/headroom-dist 📦 Built wheel for abi3 Python ≥ 3.10 to /tmp/headroom-dist/headroom_ai-0.27.0-cp310-abi3-macosx_10_12_x86_64.whl $ python3.11 -m venv /tmp/hr-venv && /tmp/hr-venv/bin/pip install . Successfully built headroom-ai Successfully installed headroom-ai-0.27.0 $ cd /tmp && /tmp/hr-venv/bin/python -c "import headroom; import headroom._core; print('ok')" version 0.27.0 _core ok ``` ## Real Behavior Proof - Environment: macOS `x86_64-apple-darwin`, Python 3.11.5, Rust 1.95.0 - Exact command / steps: Reproduced the reported failure with `pip install headroom-ai` (sdist build dies in `ort-sys` for `x86_64-apple-darwin`); after this patch ran `maturin build --release`, then `pip install .` in a clean venv, then `python -c "import headroom._core"`. - Observed result: Before fix, cargo/maturin exit 101 on missing ORT prebuilts; after fix, wheel build succeeds and `headroom._core` imports cleanly (`version 0.27.0`, `_core ok`). - Not tested: `macos-15-intel` GitHub Actions wheel matrix row (will be validated by CI after merge). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - ML features (magika detection, fastembed embeddings) still require `onnxruntime` at runtime on Intel Mac. Users should install `headroom-ai[proxy]` or `pip install onnxruntime`; `_ort.py` auto-pins `ORT_DYLIB_PATH` when that package is present. - Apple Silicon (`aarch64-apple-darwin`) behavior is unchanged: it continues to bundle ORT via `ort-download-binaries-rustls-tls`. - Lint/mypy not re-run locally in this pass; targeted pytest + maturin/pip install proof covers the changed surface. --------- Co-authored-by: Bor <you@example.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…eadroomlabs-ai#1513) ## Description `GET /v1/models` (and other buffered passthrough routes) returned an opaque HTTP **502** when an OpenAI-compatible upstream closed a pooled keep-alive connection mid-response, surfacing `httpx.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)`. The same upstream answers a direct `curl` with 200 because curl opens a fresh connection per call, while Headroom reuses pooled keep-alive connections — so the first request issued on a stale connection fails even though the upstream is healthy. The fix makes the buffered passthrough path retry once on a fresh connection (exactly what curl does), and return a clear error only if the upstream is genuinely sending an incomplete response. Closes headroomlabs-ai#1112 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom.proxy.helpers.request_with_transient_retry(client, *, request_id=None, max_retries=1, **request_kwargs)`: issues a buffered httpx request and retries on a **fresh connection** when (and only when) `httpx.RemoteProtocolError` is raised. Every other exception (`ConnectError`, timeouts, status errors) propagates immediately, so existing handling is unchanged. Documented as buffered-only (a streamed response can't be safely replayed once bytes reach the client). - Route `OpenAIHandlerMixin.handle_passthrough` through the helper, and add an `except httpx.RemoteProtocolError` arm that returns a clear `502` with error type `upstream_protocol_error` when the protocol error persists across the retry (instead of letting the raw error surface as an opaque/unhandled 502). - Add `tests/test_proxy_passthrough_transient_retry.py` (helper unit tests + handler-level tests covering the exact issue path). - Add a `CHANGELOG.md` entry under `Unreleased → Fixed`. Scope note: streaming `/v1/responses` is intentionally **out of scope** for this change — a streamed response cannot be safely retried after the first byte has been delivered to the client. The helper is written reusable so a streaming-aware follow-up can build on it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/helpers.py headroom/proxy/handlers/openai.py tests/test_proxy_passthrough_transient_retry.py All checks passed! $ mypy headroom/proxy/helpers.py --ignore-missing-imports Success: no issues found in 1 source file $ pytest tests/test_proxy_passthrough_transient_retry.py -q tests/test_proxy_passthrough_transient_retry.py ....... [100%] 7 passed in 0.27s # no regressions in the surrounding passthrough/handler suites: $ pytest tests/test_proxy_passthrough_transient_retry.py tests/test_proxy_handler_helpers.py \ tests/test_proxy_byte_faithful_forwarding.py \ tests/test_proxy/test_compression_failure_action.py tests/test_proxy_copilot_auth_hooks.py -q 80 passed, 1 warning in 6.88s ``` ## Real Behavior Proof Reproduced against a **real local TCP server** (no mocks) that speaks HTTP/1.1 and, when armed, emits a chunked body then closes the socket **without** the terminating `0\r\n\r\n` — the exact condition that makes httpx raise the `incomplete chunked read` error from this issue. - Environment: macOS arm64, Python 3.12, httpx 0.28.1 (same httpx major as the report), real loopback sockets via `asyncio.start_server`. - Exact command / steps: start the local server; (1) issue a single buffered request — the pre-fix `handle_passthrough` behaviour; (2) issue the same request through `request_with_transient_retry` — the fix. Verbatim: `python repro_1112.py`. - Observed result: BEFORE the fix a single request raises `httpx.RemoteProtocolError` ("incomplete chunked read") which `handle_passthrough` surfaced as an opaque HTTP 502; AFTER the fix the same request returns **HTTP 200** (the retry opened a fresh connection, mirroring a direct `curl`). Full terminal output: ```text upstream listening on http://127.0.0.1:62374/v1/models BEFORE (single buffered request, pre-fix behaviour): raised httpx.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read) -> handle_passthrough surfaced this as an opaque HTTP 502 AFTER (request_with_transient_retry, the fix): HTTP 200 body={"object":"list","data":[]} -> first attempt hit the incomplete chunked read, retry on a fresh connection returned 200 (mirrors a direct curl) ``` The log line `Upstream closed connection mid-response (...incomplete chunked read); retrying on a fresh connection (attempt 1/1)` fires on the recovered request, confirming the retry path is what produced the 200. - Not tested: real third-party upstreams (LiteLLM/vLLM/etc.) — the local server reproduces the precise httpx error deterministically; the streaming `/v1/responses` path is intentionally out of scope (a streamed response cannot be safely retried after the first byte reaches the client). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - No new dependencies (httpx is already a proxy dependency), so no supply-chain justification is required. - The retry is deliberately narrow: only `httpx.RemoteProtocolError` is retried, capped at one retry, so a genuinely-down upstream still fails fast via the existing `ConnectError`/timeout path. - "Documentation" checklist item refers to the `CHANGELOG.md` entry; no user-facing docs pages needed for this internal resilience fix.
… stops busting (headroomlabs-ai#1852) Follow-up to headroomlabs-ai#1850. Two residual cache-bust sources, both `cache_control`-related: 1. **Guard too strict.** `overlay_cached_prefix` decided "is this turn an append-only extension?" by comparing whole message dicts — including `cache_control`. Clients (Claude Code, litellm) move the cache breakpoint to the newest message every call, so a marker landing in the frozen prefix made the guard fail, the overlay skip its replay, and the raw freeze forward ORIGINAL bytes over the cached COMPRESSED prefix → partial bust (the ~42% residual on the a10 run, `prefix_change=0`). Fix: run the append-only guard on **content only** (strip `cache_control` before comparing) — content is what the provider's cache keys on. 2. **Marker accumulation.** The overlay replays the markers that rode on each turn's then-newest message, so `cache_control` blocks pile up ~1/turn; Anthropic hard-errors at >4 total. Fix: `normalize_message_cache_control` strips every message-level marker and re-places a single ephemeral breakpoint on the last block (one breakpoint caches the whole prefix; cache is content-keyed so re-placing never busts). Wired into the Anthropic handler after the overlay. **Per-provider (deliberately scoped):** - **Anthropic**: `cache_control` markers → both fixes apply. - **OpenAI**: AUTOMATIC prefix caching, no markers → overlay (byte-identity) only; normalize is NOT applied (Anthropic markers on an OpenAI request would be wrong). - **Bedrock**: serves Claude via the pipeline but has no cachePoint/freeze-replay path → not affected; a cachePoint analog would be needed if caching is expanded. - **Gemini**: explicit Cache API (`cachedContent`), no inline markers/freeze → N/A. > Stacked on headroomlabs-ai#1850 — review that first; the diff against `main` includes its overlay + `has_new_ccr_markers` work. ## Description Keeps the freeze overlay's cache-safety intact against real clients that relocate the `cache_control` breakpoint each turn, and prevents `cache_control` blocks from accumulating past Anthropic's 4-marker limit. See the two fixes above. Closes #<!-- none --> — follow-up to headroomlabs-ai#1850 (no separate issue). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/prefix_tracker.py`: append-only guard in `overlay_cached_prefix` now compares **content only** (ignores `cache_control`); new `normalize_message_cache_control()` collapses message-level markers to a single ephemeral breakpoint on the last block. - `headroom/proxy/handlers/anthropic.py`: apply `normalize_message_cache_control` after the overlay (Anthropic only). - `tests/test_cache_control_move_bust.py`: reproduces the moved-marker bust + proves both fixes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (local, see below) ### Test Output ```text $ pytest tests/test_cache_control_move_bust.py -q ....... [100%] 7 passed in 0.19s # broader cache-safety suite (overlay + cross-turn + CCR deferred + openai/anthropic cache-stability + helpers) $ pytest tests/test_cache_control_move_bust.py tests/test_cache_prefix_overlay.py \ tests/test_cross_turn_cache_safety.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py \ tests/test_proxy_handler_helpers.py tests/test_proxy_openai_cache_stability.py \ tests/test_proxy_anthropic_cache_stability.py -q 91 passed, 2 warnings in 29.98s $ ruff check . # ruff 0.15.17 (CI-pinned) All checks passed! $ ruff format --check . # ruff 0.15.17 1057 files already formatted $ mypy headroom --ignore-missing-imports Success: no issues found (changed modules: prefix_tracker, anthropic, openai, helpers) ``` ## Real Behavior Proof - **Environment:** local (`.venv`, Python 3.12), ruff 0.15.17 / mypy pinned to CI versions. - **Exact command / steps:** `tests/test_cache_control_move_bust.py` drives the REAL tracker + freeze + `overlay_cached_prefix` + `normalize_message_cache_control` across multiple append-only turns where the client moves the `cache_control` breakpoint each turn. - **Observed result:** with a moved marker in the frozen prefix, the content-only guard keeps the overlay replaying (forwarded prefix stays byte-identical → no bust); `cache_control` blocks stay ≤4 across many turns and content is never altered. The reproduction test fails without the fix and passes with it. - **Not tested (this PR):** the end-to-end a10 SWE-bench run is the field observation motivating fix #1 (~42% residual, `prefix_change=0`); not re-run here. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Stacked on headroomlabs-ai#1850; land that first. Docs/CHANGELOG untouched (behavioral cache-safety fix; no user-facing surface change). N/A: no screenshots (no UI).
headroomlabs-ai#1606) ## Description OpenAI-compatible `/v1/chat/completions` requests didn't receive the same proxy savings/profile kwargs as the other compression paths. The live chat handler (`handle_openai_chat` in `headroom/proxy/handlers/openai.py`) called `openai_pipeline.apply()` with only `model_limit` / `context` / `frozen_message_count` / `biases` / `compression_policy` — it never passed `proxy_pipeline_kwargs(self.config)`. So when the proxy runs with `HEADROOM_SAVINGS_PROFILE=agent-90`, the effective config reports user/system-message compression and `target_ratio=0.10`, but the real chat path silently dropped all of it. OpenAI-compatible clients such as OpenCode kept protecting user messages and missed the configured profile. For contrast, `handlers/anthropic.py` passes `**proxy_pipeline_kwargs(self.config)` to every `apply()` call, and so does the dedicated OpenAI compress endpoint in this same module — only the two chat-completions `apply()` sites were missing it. Closes headroomlabs-ai#1534 ## Fix Add `**proxy_pipeline_kwargs(self.config)` to both chat-path `apply()` calls (the token-mode branch and the non-token branch): ```python lambda: self.openai_pipeline.apply( messages=messages, model=model, model_limit=context_limit, context=extract_user_query(messages), frozen_message_count=openai_frozen_count, biases=_hook_biases, compression_policy=compression_policy, **proxy_pipeline_kwargs(self.config), # ← added ) ``` `proxy_pipeline_kwargs` is already imported in the module and is the exact helper the Anthropic handler and the OpenAI compress endpoint use, so the chat path now matches them. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/handlers/openai.py`: pass `**proxy_pipeline_kwargs(self.config)` on both `apply()` call sites in `handle_openai_chat` (token-mode and non-token branches). - `tests/test_proxy/test_openai_chat_savings_profile.py`: new regression test driving the chat handler with `savings_profile="agent-90"` and asserting the profile knobs reach `apply()`. - `CHANGELOG.md`: Bug Fixes entry under Unreleased. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output The new test drives the real chat handler through the `create_app` + `TestClient` harness with a recording `apply()` stub. Before the fix it captures exactly the five kwargs the issue describes (no profile knobs); after the fix the profile knobs are present: ```text # before the fix (openai.py reverted, test kept) E AssertionError: assert None is True E + where None = {...}.get('compress_user_messages') # captured kwargs were: biases, compression_policy, messages, model, # model_limit, context, frozen_message_count — no profile knobs FAILED tests/test_proxy/test_openai_chat_savings_profile.py::test_chat_completions_threads_savings_profile_kwargs_into_apply # after the fix tests\test_proxy\test_openai_chat_savings_profile.py . ======================== 1 passed, 1 warning in 39.44s ======================== ``` No regression in the existing chat backend-path suite: ```text $ uv run pytest tests/test_proxy/test_openai_backend_path.py ======================== 5 passed, 1 warning in 15.78s ======================== $ uv run ruff check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_chat_savings_profile.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, headroom built from this branch (`uv sync --extra dev`), proxy config `savings_profile="agent-90"`, `optimize=True`, `backend="anyllm"` with a mocked OpenAI upstream. - Exact command / steps: started the app with `create_app(config)`, replaced `proxy.openai_pipeline.apply` with a recording stub, and POSTed a real `/v1/chat/completions` request with a large user message so the compression decision fires. Inspected the kwargs the handler actually passed to `apply()`. - Observed result: before the fix the recorded `apply()` kwargs were `{biases, compression_policy, messages, model, model_limit, context, frozen_message_count}` — no profile knobs. After the fix the same call also carries `compress_user_messages=True`, `compress_system_messages=True`, `target_ratio=0.10`, `min_tokens_to_compress=120` (the agent-90 profile), matching the issue's "Expected". - Not tested: did not stand up a real OpenAI/OpenCode upstream end-to-end (no live key in this environment); the upstream is mocked and the assertion is on the kwargs the proxy threads into the compression pipeline, which is exactly what the bug was about. Did not run the full `mypy headroom` pass (two-line kwarg addition, no new types). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Two-line change plus comments; no new dependencies. Reuses the existing `proxy_pipeline_kwargs` helper, so behavior is consistent across Anthropic, the OpenAI compress endpoint, and now the OpenAI chat path. - @chopratejas flagging you for review — this aligns the OpenAI chat path with the savings-profile handling the other providers already had. Co-authored-by: JD Davis <mxjerrett@gmail.com>
…s-ai#1645) ## Description Under concurrent load with large request bodies, `/v1/messages` returns **HTTP 502**. A single upstream HTTP/2 stream reset poisons the shared h2 connection and raises `RemoteProtocolError` (`StreamReset`) / `LocalProtocolError` on every other in-flight stream: ``` ERROR [hr_...] Request failed: RemoteProtocolError: <StreamReset stream_id:35, error_code:1, remote_reset:True> ERROR [hr_...] Request failed: LocalProtocolError: 39 INFO event=proxy_inbound_response ... status=502 duration_ms=78712 ``` These are transport errors, but they weren't in the proxy's retry paths — the non-streaming `_retry_request` caught `(ConnectError, TimeoutException, HTTPStatusError)` and the streaming connect loop caught `(ConnectError, ConnectTimeout, PoolTimeout)`. So a stream reset skipped retry entirely and fell through to the broad handler catch as a `502`, with no reconnect. This broadens both retry paths to treat any `httpx.TransportError` — which includes the h2 `Local`/`RemoteProtocolError` — as retryable, so the poisoned connection is dropped and the request re-sent on a fresh one. Closes headroomlabs-ai#1639 > Scope note: the issue also mentions `HEADROOM_HTTP2` being ignored on the `headroom install agent run` launch path. That's a separate config-plumbing gap; I've kept this PR to the 502-cascade fix (which makes the chain self-recover regardless of the env workaround) and am happy to follow up on the env plumbing separately. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/server.py` (`_retry_request`): the retry `except` now catches `(httpx.TransportError, httpx.HTTPStatusError)` instead of `(ConnectError, TimeoutException, HTTPStatusError)`. `TransportError` is the common base of ConnectError, the timeout family, and the protocol/network errors — so h2 stream resets are retried with backoff. - `headroom/proxy/handlers/streaming.py`: the streaming connect-retry loop and its terminal handler now catch `httpx.TransportError`. The retry runs before any body byte is forwarded to the client (only `build_request` + `send(stream=True)` are inside the loop), so re-sending is safe. On exhaustion the terminal handler still emits a clean `event: error` SSE instead of letting the reset bubble up as a 502. The mid-stream handler was left as-is (already covered by its `except Exception`, and not safe to retry once bytes have been sent). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_h2_stream_reset_retry.py -q 4 passed $ pytest tests/test_proxy_streaming_resilience.py tests/test_mid_turn_steering.py \ tests/test_proxy_streaming_ratelimit_headers.py tests/test_streaming_usage_parser.py \ tests/test_proxy_byte_faithful_forwarding.py -q 87 passed, 1 skipped $ ruff check <changed files> && ruff format --check <changed files> All checks passed! / 3 files already formatted $ mypy headroom/proxy/server.py headroom/proxy/handlers/streaming.py --ignore-missing-imports Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS (arm64), Python 3.14 venv, editable install of this branch. - Exact command / steps: ran `pytest tests/test_h2_stream_reset_retry.py` — the tests drive the real `_retry_request` and `_stream_response` with `http_client.post` / `http_client.send` set to raise `httpx.RemoteProtocolError("<StreamReset ...>")` on the first attempt and return a good response on the second. - Observed result: non-streaming — the request is retried and returns the `200` response (`post` awaited twice); on unconditional resets it re-raises after `retry_max_attempts` (no silent hang). Streaming — the reset on `send()` is retried and the upstream SSE (`message_start`…) is forwarded with no `connection_error` event (`send` awaited twice); on repeated resets a clean `event: error` SSE is emitted rather than a crash/502. Before this change the same `RemoteProtocolError` was uncaught and propagated to the `502` handler. - Not tested: a live 10-session concurrent-load repro against a real Anthropic h2 endpoint — reproduced deterministically at the retry boundary with an injected `RemoteProtocolError` instead. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Retrying a stream reset re-sends the (potentially large) body, but that is bounded by the existing `retry_max_attempts` + jittered backoff and only happens before the first client byte — the same contract the existing connect-error retry already relied on. This is complementary to, not a replacement for, an operator forcing HTTP/1.1; it makes the default h2 path self-heal from transient resets. Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
## Description Fixes headroomlabs-ai#1602. OpenCode Zen custom-base requests can reach Headroom through the generic passthrough path, but that route was not supplying endpoint/provider metadata for Zen chat completions. This made forwarded Zen traffic invisible in dashboard provider, usage, and token telemetry. Closes headroomlabs-ai#1602 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a narrow OpenCode Zen custom-base classifier for `POST /zen/v1/chat/completions` on `opencode.ai` and `www.opencode.ai`. - Passed `endpoint_name="chat/completions"` and `provider="zen"` into catch-all passthrough telemetry for matching Zen traffic. - Attributed normalized OpenCode transport traffic (`/v1/chat/completions` with `x-headroom-original-path: /zen/v1/chat/completions`) to `zen` for request outcomes while keeping the OpenAI parser path unchanged. - Added coverage for direct catch-all routing, normalized original-path routing, token usage outcome recording, and false-positive paths like `/mcp/v1/chat/completions`, `/npm/v1/chat/completions`, and `/context7/v1/chat/completions`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ rtk pytest tests/test_custom_base_passthrough_telemetry.py -q Pytest: 4 passed $ rtk uvx --from ruff==0.15.17 ruff check headroom/proxy/handlers/openai.py headroom/providers/proxy_routes.py tests/test_custom_base_passthrough_telemetry.py tests/test_provider_proxy_routes.py tests/test_proxy/test_openai_transport_path_prefix.py All checks passed! $ rtk uvx --from ruff==0.15.17 ruff format --check headroom/proxy/handlers/openai.py headroom/providers/proxy_routes.py tests/test_custom_base_passthrough_telemetry.py tests/test_provider_proxy_routes.py tests/test_proxy/test_openai_transport_path_prefix.py 5 files already formatted $ rtk /Library/Frameworks/Python.framework/Versions/3.13/bin/python3 -m py_compile headroom/proxy/handlers/openai.py headroom/providers/proxy_routes.py tests/test_custom_base_passthrough_telemetry.py tests/test_provider_proxy_routes.py tests/test_proxy/test_openai_transport_path_prefix.py # passed $ rtk git diff --check # passed ``` GitHub Actions also passed after the final push, including CI, Docker native/wrap/init E2E, security, lint, and PR governance. ## Real Behavior Proof - Environment: local worktree on macOS plus GitHub Actions for PR headroomlabs-ai#1648. - Exact command / steps: ran focused pytest coverage for Zen passthrough telemetry, Ruff check/format validation on touched files, Python compile validation, `git diff --check`, and waited for the full GitHub Actions rollup. - Observed result: Zen custom-base chat completions now record request outcomes as provider `zen` with endpoint `chat/completions`; false-positive OpenCode paths remain unattributed to Zen; GitHub checks are green. - Not tested: full local test suite did not collect in this worktree because the native `headroom._core` extension is not installed. `rtk npm --prefix plugins/opencode test` is also blocked locally because `vitest` is not installed in `plugins/opencode/node_modules`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes The documentation and CHANGELOG checklist items are not applicable for this narrow telemetry bug fix. No new comments were added because the code path is covered by narrowly named helper/test cases.
…roomlabs-ai#1665) ## Description Cache-mode deployments lose their primary savings metric on every proxy restart. Savings in cache mode come from provider prefix-cache reads, but those totals are tracked only in process memory (`PrefixCacheTracker` + `PrometheusMetrics` counters): `proxy_savings.json` accumulates compression savings exclusively, so a cache-mode instance's persisted lifetime stays near zero while the number the operator watches grows in RAM. Any restart (including the restart every upgrade requires) zeroes it. Observed in the field on a self-hosted cache-mode instance (1.29B lifetime input tokens over 13 days): ~400M tokens of displayed cache savings dropped to the durable-only figures after an upgrade restart, unrecoverable because they were never written to disk. This PR persists lifetime cache-read savings (tokens + USD) in the existing SavingsTracker store and points every lifetime-savings surface (dashboard cache tile, `headroom_stats` MCP summary, `headroom doctor`) at the persisted value. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `SavingsTracker` accumulates `cache_read_tokens` and `cache_savings_usd` into the persisted `lifetime` and `display_session` blocks (`record_request` already received the per-request cache counts from the outcome funnel; they were only used for cost estimation). - New `_estimate_cache_savings_usd` prices the saving as the litellm discount delta (`input_cost_per_token - cache_read_input_token_cost`), failing open to 0.0 for unpriced models while tokens still accumulate. The deliberate divergence from `proxy/cost.py`'s session-scoped provider multipliers is documented in the helper docstring. - `SCHEMA_VERSION` 3 -> 4, additive: the tolerant loader coerces missing fields to zero, so v3 files load unchanged (covered by tests, both directions). `_normalize_display_session` gains the fields so an active session reloaded from an older file cannot drop them. - `_coerce_int`/`_coerce_float` hardened against bare `Infinity`/`NaN` in a corrupted state file (uncaught `OverflowError` on startup; NaN is absorbing under `+=` and would brick an accumulator). - Dashboard: "Cache Reads (lifetime)" tile binds to `persistent_savings.lifetime`; the Prefix Cache Impact card renders after a zero-traffic restart (new `cacheSessionActive` getter), session-scoped tiles show "no activity since restart", and the dollar line gets the hero tile's three-way zero-state. - `headroom_stats` MCP summary and `headroom doctor` surface the new lifetime cache fields alongside the compression figures they already render, keeping agent/CLI parity with the dashboard. - New Playwright test pins the restart-survival card behavior; the existing savings suites gain 8 unit tests (restart survival, v3 tolerance, stateless, session-reload guard, pricing formula + fallbacks, non-finite state coercion, rollover). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text tests/test_proxy_savings_history.py tests/test_proxy_project_savings.py tests/test_ccr_mcp_server.py tests/test_cli_doctor.py ================== 94 passed, 1 skipped, 1 warning in 30.79s =================== tests/test_dashboard tests/test_proxy_dashboard_stats_cache.py ================== 10 passed, 2 skipped, 1 warning in 11.00s =================== ruff check: All checks passed! | ruff format --check: already formatted mypy headroom/proxy/savings_tracker.py headroom/ccr/mcp_server.py headroom/cli/doctor.py: Success: no issues found in 3 source files pre-commit (ruff, ruff-format, mypy): Passed Fails-before (new tests on unpatched code): 6 failed -- KeyError: 'cache_read_tokens' -- 19 passed ``` ## Real Behavior Proof - Environment: macOS, Python 3.13 venv, proxy from this branch on 127.0.0.1:8788, `--mode cache --backend anthropic`, mock Anthropic upstream on 127.0.0.1:8791 returning `usage.cache_read_input_tokens=800000`, `HEADROOM_SAVINGS_PATH` pointed at a scratch file, `HF_HUB_OFFLINE=1 LITELLM_LOCAL_MODEL_COST_MAP=true`. - Exact command / steps: started the proxy, sent two simulated `POST /v1/messages` requests with a `cache_control` block via curl, read `/stats`, stopped the proxy process, started it again with the same env, read `/stats` again with zero new traffic. - Observed result: before restart `persistent_savings.lifetime` showed `"cache_read_tokens": 1600000, "cache_savings_usd": 7.2`; after restart the same values were retained while the in-memory session totals (`prefix_cache.totals.cache_read_tokens`) correctly read 0 -- previously the lifetime figure reset to zero with the process. - Not tested: live Anthropic upstream (mock returns the usage shape verbatim); the Playwright card tests skip locally (no browser install) and run in CI; multi-process writers (out of scope -- the store is single-writer by design). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - The card's session-scoped "Net savings" header (provider-economics pricing in `proxy/cost.py`) and the new lifetime dollar figure (litellm per-model delta) use different pricing paths by design; operators may notice a $ discontinuity at cutover. Documented in the helper docstring. - A pre-existing `isinstance(x, (int, float))` in `cli/doctor.py` was switched to the union form because the repo's pre-commit UP038 rule blocks committing the file otherwise. - Pushed with `--no-verify`: the pre-push `ci-precheck` fails on the known machine-load-sensitive Rust latency benchmark; this is a Python/template -only change. - Screenshots: N/A (card behavior asserted by the new Playwright test). Co-authored-by: Omar Gerardo <ogerardo@MacBook-Air.local>
…eadroomlabs-ai#1668) ## Description Fixes four real bugs that made CODE_AWARE (AST-based) compression silently non-functional for Go, plus the product-behavior change to make CODE_AWARE the default for code (previously in headroomlabs-ai#1670, now consolidated here per review). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `code_compressor.py`: unwrap tree-sitter-go's single `statement_list` wrapper node when building `body_stmts` — its row range was swallowing the block's own closing-brace row, producing a duplicated `}` in compressed Go output. - `code_compressor.py`: match opening-brace lines by `endswith("{")` instead of `startswith("{")`, so multi-line Go signatures (e.g. `) error {`) aren't silently dropped from the compressed output. - `content_router.py`: normalize CODE_AWARE's `compressed_tokens` to `len(compressed.split())`, matching the word-split convention every other strategy (search/log/tabular/diff) already uses for `original_tokens`. Previously the mismatched scales made genuinely-good compressions look like "no savings" and get discarded for the Kompress fallback. - `content_router.py`: default `prefer_code_aware_for_code` to `True` (was `False`) — CODE_AWARE gives higher, syntax-safe compression than Kompress for code, so now that the bugs above are fixed it should be the default path. (Consolidated from headroomlabs-ai#1670, now closed.) - `server.py`: add `HEADROOM_PREFER_CODE_AWARE_FOR_CODE` env override for `ContentRouterConfig.prefer_code_aware_for_code`, mirroring the existing `HEADROOM_CODE_AWARE_ENABLED` pattern, defaulting to `True`. - Formatting: ran `ruff format` on `server.py` and `content_router.py` (CI was failing on this). - `tests/test_code_aware_regressions.py` (new): 5 regression tests — - Go `statement_list` unwrap: no duplicated closing brace after truncation. - Multi-line Go signature: `) error {` line survives truncation. - ContentRouter CODE_AWARE token accounting: `compressed_tokens` matches `len(compressed.split())`, and a real compression doesn't trigger a needless Kompress fallback. - `prefer_code_aware_for_code` defaults to `True` on the `ContentRouterConfig` dataclass. - `prefer_code_aware_for_code` defaults to `True` via the `HEADROOM_PREFER_CODE_AWARE_FOR_CODE` env var (through a real `HeadroomProxy` construction). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m ruff check headroom/proxy/server.py headroom/transforms/code_compressor.py headroom/transforms/content_router.py tests/test_code_aware_regressions.py All checks passed! $ python -m ruff format --check headroom/proxy/server.py headroom/transforms/code_compressor.py headroom/transforms/content_router.py tests/test_code_aware_regressions.py 4 files already formatted $ python -m mypy ... Not run — mypy not installed in this environment. $ python -m pytest tests/test_code_compressor_thread_safety.py tests/test_content_router_exclude_tools.py \ tests/test_content_router_tool_role_reversibility.py tests/test_compression_units.py \ tests/test_compression_determinism.py tests/test_compression_safety_rails.py tests/test_netcost_gate.py \ tests/test_code_aware_regressions.py -q 15 failed, 66 passed, 1 warning in 7.17s # The 15 failures are the same pre-existing/environment-specific ones from # before (reproduced identically on a clean upstream/main checkout with no # code changes — missing torch/trafilatura/playwright, stale Rust _core # build in this checkout), not caused by this change. All 5 new regression # tests in test_code_aware_regressions.py pass. ``` ## Real Behavior Proof - Environment: Windows, Python 3.11.9, headroom-ai pipx install (0.28.0) with the same fixes applied, plus this fork's checkout for lint/test verification. - Exact command / steps: ran `CodeAwareCompressor.compress()` directly against real `.go` files from an external ~100-file Go codebase, and separately routed the same files through the full `ContentRouter` with `HEADROOM_PREFER_CODE_AWARE_FOR_CODE=1`. - Observed result: 72/97 files routed to `code_aware` and compressed with syntactically valid Go output (parsed via tree-sitter re-check), 0 invalid-syntax fallbacks, 0 "routed but unchanged" cases, 14641 total tokens saved. Before the fix: 0 tokens saved via this path (all bugs combined made it a no-op). - Not tested: `mypy`, and the full repo test suite (blocked by unrelated pre-existing environment issues — see Test Output). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Per @JerrettDavis's review: consolidated headroomlabs-ai#1670 (the `prefer_code_aware_for_code` default flip) into this PR and closed headroomlabs-ai#1670 as the duplicate; fixed the `ruff format` CI failure; added the 4 requested regression tests (Go statement_list dedup, multiline-signature brace preservation, content-router token-accounting parity, and the config-default pin). --------- Co-authored-by: shekharcharles <shekhar.aegis@gmail.com>
…roomlabs-ai#1676) ## Description `headroom install apply` regenerates the deployment manifest on every run, and that regeneration silently drops any manually-added `--no-http2` override. The HTTP/2 workaround itself is already real and already supported by `headroom proxy`, but persistent installs had no first-class way to keep it. This PR adds `--no-http2` to `install apply`, threads it into `build_manifest()`, and persists the flag in `manifest.proxy_args` so it survives reapply. Closes headroomlabs-ai#1615 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `--no-http2` to `headroom install apply`, and forwarded the flag into `build_manifest()`. - Extended `headroom/install/planner.py` so `build_manifest(..., no_http2=True)` persists `--no-http2` into `manifest.proxy_args`. - Added planner-level regression coverage for both the override path and the default-preservation path. - Added CLI-level regression coverage that proves `install apply --no-http2` forwards correctly and that the help surface advertises the flag. - `CHANGELOG.md` intentionally not touched: repo policy generates changelog entries from conventional commits rather than manual PR edits. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_install/test_planner.py` and `uv run pytest tests/test_cli/test_install_cli.py`) - [x] Linting passes (`uv run ruff check .` and `uv run ruff format . --check`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text > rtk uv run pytest tests/test_install/test_planner.py -k no_http2 -q collected 7 items / 5 deselected / 2 selected tests\test_install\test_planner.py .. [100%] 2 passed, 5 deselected in 0.18s > rtk uv run pytest tests/test_cli/test_install_cli.py -k no_http2 -q collected 19 items / 17 deselected / 2 selected tests\test_cli\test_install_cli.py .. [100%] 2 passed, 17 deselected in 0.23s > rtk uv run pytest tests/test_install/test_runtime.py -q collected 19 items tests\test_install\test_runtime.py ..........F........ [100%] FAILED tests/test_install/test_runtime.py::test_runtime_start_lock_blocks_another_process 1 failed, 18 passed in 0.44s (Confirmed pre-existing on unmodified origin/main via `git stash` in this worktree, identical failure with none of this PR's changes applied. Environment-specific lock-file flakiness in this sandbox, unrelated to install-manifest persistence; runtime.py was not touched by this change.) > rtk uv run ruff check headroom/cli/install.py headroom/install/planner.py tests/test_install/test_planner.py tests/test_cli/test_install_cli.py All checks passed! > rtk uv run ruff format --check headroom/cli/install.py headroom/install/planner.py tests/test_install/test_planner.py tests/test_cli/test_install_cli.py 4 files already formatted ``` ## Real Behavior Proof - Environment: local source checkout with `uv` dev environment, using the existing install CLI and manifest builder, in worktree `D:\Repos\headroom-pr-1615-persist-install-http2-override`. - Exact command / steps: ran `headroom install apply --help` through `CliRunner`, ran a direct `build_manifest(..., no_http2=True)` proof, and ran the focused planner, CLI, runtime, and lint checks. - Observed result: on `origin/main`, `install apply --help` lacked `--no-http2` and `build_manifest(..., no_http2=True)` raised `TypeError: build_manifest() got an unexpected keyword argument 'no_http2'`; on this branch, `install apply --help` lists `--no-http2`, `build_manifest(..., no_http2=True)` returns a manifest whose `proxy_args` contains exactly one `--no-http2` entry (`['--host', '127.0.0.1', '--port', '8787', '--mode', 'token', '--backend', 'anthropic', '--telemetry', '--no-http2']`), persistent installs now preserve the existing HTTP/2 disable flag across `install apply` regeneration, and runtime behavior still comes entirely from replaying manifest `proxy_args` (`runtime.py` was not modified). - Not tested: a full persistent-service supervisor round-trip or full CI suite locally. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (not applicable, changelog entries are generated from conventional commits per repo policy) ## Additional Notes This stays scoped to the install-manifest persistence seam only; it does not revisit HTTP/2 default policy, retry behavior, or proxy transport construction. Attribution: the implementation shape follows the persistence pattern already established by headroomlabs-ai#1365, and the remaining install-layer gap was confirmed by `sarkarsital1959` in the 2026-07-01 comment on headroomlabs-ai#1615.
…ver blank on a truthy-but-empty provider payload
…eadroomlabs-ai#2071) ## Description `parse_tool_call` (`headroom/ccr/tool_injection.py`) extracts the retrieval hash from a CCR tool call. For the OpenAI and `openai_responses` shapes it decodes the `arguments` string with `json.loads` and catches only `JSONDecodeError`: ```python args_str = function.get("arguments", "{}") try: input_data = json.loads(args_str) except json.JSONDecodeError: input_data = {} ... hash_key = input_data.get("hash") # assumes input_data is a dict ``` If a (confused) model emits `arguments='[]'` / `'"abc"'` / `'123'`, `json.loads` succeeds and returns a **list / str / number**, so `input_data.get("hash")` raises `AttributeError`. A null value (`arguments: null` → `json.loads(None)`) raises an uncaught `TypeError`. The Anthropic branch has the same hazard if `tool_call["input"]` is present but not a dict. `parse_tool_call` is called from `parse_ccr_tool_calls` (`ccr/tool_calls.py`) and the server CCR path with no guard for this, so a malformed CCR-named tool call **crashes CCR response processing** instead of being ignored. Closes: no issue filed — found while auditing the CCR tool-call parsing. ## Fix - Catch `TypeError` as well as `JSONDecodeError` around `json.loads` (covers `arguments: null`). - Return `None` when `input_data` is not a `dict` — a non-object tool call simply isn't a valid CCR call. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/ccr/tool_injection.py`: widen the decode `except` to `(json.JSONDecodeError, TypeError)`; return `None` for non-dict `input_data`. - `tests/test_ccr_tool_injection.py`: add tests for non-object OpenAI arguments (`[]`/`"abc"`/`123`), null arguments, and a non-dict Anthropic `input`. ## Testing - [x] New regression tests added (`tests/test_ccr_tool_injection.py`) - [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17` - [ ] Full `pytest` deferred to CI (local-OOM reason below). ```text $ uvx ruff@0.15.17 check headroom/ccr/tool_injection.py tests/test_ccr_tool_injection.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.10, headroom from this branch. Importing `headroom` pulls in the torch/transformers stack and a full `pytest` gets OOM-killed on this box, so I verified the parse logic with a dependency-free script and left the full pytest to CI. - Exact command / steps: ran the four crash vectors (openai `[]`, `"abc"`, `null`; anthropic non-dict `input`) plus a valid CCR call and a non-CCR call through the old and new logic. - Observed result: the old parser crashes on every malformed case; the new one returns `None` and still parses a valid call: ```text OK [openai] '[]': old CRASHED -> new None OK [openai] '"abc"': old CRASHED -> new None OK [openai] None: old CRASHED -> new None OK [anthropic] ['not', 'a', 'dict']: old CRASHED -> new None PARSE_TOOL_CALL NON-DICT FIX VERIFIED (old crashes; new returns None; valid still parses) ``` - Not tested: a full CCR response round-trip with a malformed tool call (needs the heavy stack). The fix is confined to `parse_tool_call` and the new tests drive it directly. Full local `pytest` deferred to CI (OOM, per above). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes — ran lint + a standalone logic check; full pytest deferred to CI (local OOM, disclosed above) - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Two-line hardening plus tests; no new dependencies. - @JerrettDavis tagging you — a malformed CCR-named tool call currently crashes CCR response processing; quick one. Thanks!
## Description Extracts the pure SSE byte-buffer parser from `helpers.py` into `headroom.proxy.sse_byte_buffer_policy`. Existing helper imports remain as delegates, while the protocol parser now has its own module and direct tests. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `sse_byte_buffer_policy.py` for SSE terminator detection and complete-event parsing. - Kept `helpers.parse_sse_events_from_byte_buffer` and `_find_sse_event_terminator` delegating to the extracted policy. - Added direct policy tests for LF/CRLF terminators, buffer draining, split UTF-8 preservation, and invalid complete UTF-8 events. - Carried forward the LiteLLM callback compatibility shim needed for current mypy on `main`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests\test_sse_byte_buffer_policy.py tests\test_sse_utf8_split.py 8 passed in 0.23s python -m ruff check . All checks passed! python -m ruff format --check . 1095 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 409 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, branch `jd/architecture-slice-29`. - Exact command / steps: ran new SSE byte-buffer policy tests, existing SSE UTF-8 split tests, ruff, ruff format check, mypy, and staged gitleaks scan. - Observed result: SSE parser behavior remains covered and local lint/type/security checks pass. - Not tested: live streaming proxy runtime; existing helper imports remain intact. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and changelog updates are N/A for this internal architecture-only refactor. The push reported existing default-branch Dependabot alerts; no staged secret leaks were found for this PR.
## Description Extracts the sticky memory tool session tracker from `headroom.proxy.helpers` into a focused state module. `helpers.SessionToolTracker` remains as an env-aware compatibility wrapper so existing injection and singleton call sites keep the same API. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `headroom.proxy.tool_injection_tracker.SessionToolTracker` as the pure bounded LRU state holder. - Replaced the large in-helper tracker implementation with a small env-aware wrapper. - Added direct tracker tests for unknown sessions, ordered golden bytes, first-write wins, provider isolation, LRU eviction, and input validation. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_tool_injection_tracker.py tests/test_memory_tool_session_sticky.py tests/test_corrupt_golden_bytes_recovery.py 44 passed in 0.54s python -m ruff check . All checks passed! python -m ruff format --check . 1069 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 410 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13 - Exact command / steps: Ran direct tracker tests, sticky memory tool tests, corrupt golden byte recovery tests, full ruff, format check, mypy, and staged gitleaks scan. - Observed result: Existing sticky injection behavior and recovery behavior remain green while the tracker state domain is directly covered. - Not tested: Full repository pytest suite locally; CI covers the broader matrix. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and changelog updates are not applicable for this internal refactor. The default-branch Dependabot alerts reported during push are pre-existing and unrelated to this PR.
## Description Extracts CCR golden tool replay and fresh-definition canonicalization from `headroom.proxy.helpers.apply_session_sticky_ccr_tool` into a focused policy module. This keeps sticky CCR orchestration in helpers while making the byte replay/regeneration behavior independently testable. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `headroom.proxy.ccr_golden_policy` for replaying stored CCR golden bytes and creating canonical fresh CCR tool definitions. - Updated `apply_session_sticky_ccr_tool` to delegate CCR golden replay/fresh definition policy while preserving tracker coordination and logging decisions. - Added direct tests for golden-byte replay, invalid/corrupt bytes, non-UTF-8 bytes, and fresh canonical definition generation. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_ccr_golden_policy.py tests/test_ccr_tool_always_on.py tests/test_corrupt_golden_bytes_recovery.py tests/test_proxy/test_ccr_frozen_prefix_coupling.py 30 passed in 0.34s python -m ruff check . All checks passed! python -m ruff format --check . 1069 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 410 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, clean worktree from `headroomlabs/main` at `d2170b19`. - Exact command / steps: Ran targeted CCR golden replay/sticky injection/corrupt-byte regression tests plus ruff, ruff-format, mypy, and staged gitleaks scan. - Observed result: All targeted tests and local gates passed; staged secret scan found no leaks. - Not tested: Full Docker/native wrapper CI locally; covered by repository CI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes Documentation and changelog updates are not applicable for this internal refactor. The push reported existing default-branch Dependabot vulnerabilities; this PR's staged gitleaks scan passed and CI security checks are expected to validate the branch.
## Description Extracts proxy tool-definition name parsing from `headroom.proxy.helpers` into a focused policy module. The existing private helper remains as a compatibility wrapper while memory and CCR injection skip logic share the tested parser. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `headroom.proxy.tool_name_policy.extract_tool_name` for Anthropic custom tools, OpenAI function tools, and Anthropic native memory tools. - Updated `helpers._extract_tool_name` to delegate to the policy module while keeping its existing import path intact. - Added direct tests for name precedence, function-tool parsing, native-tool fallback, invalid values, and wrapper compatibility. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_tool_name_policy.py tests/test_memory_tool_session_sticky.py tests/test_ccr_tool_always_on.py tests/test_issue_728_empty_tools_injection.py tests/test_proxy/test_ccr_frozen_prefix_coupling.py 60 passed in 0.90s python -m ruff check . All checks passed! python -m ruff format --check . 1069 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 410 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, clean worktree from `headroomlabs/main` at `d2170b19`. - Exact command / steps: Ran targeted tool-name policy tests plus memory/CCR injection regression tests, ruff, ruff-format, mypy, and staged gitleaks scan. - Observed result: All targeted tests and local gates passed; staged secret scan found no leaks. - Not tested: Full Docker/native wrapper CI locally; covered by repository CI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes Documentation and changelog updates are not applicable for this internal refactor. The push reported existing default-branch Dependabot vulnerabilities; this PR's staged gitleaks scan passed and CI security checks are expected to validate the branch. --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## Description Extracts memory tool injection stickiness configuration parsing from `headroom.proxy.helpers` into a focused policy module. The existing helper functions remain in place for the session tool tracker and sticky injection helpers. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `headroom.proxy.tool_injection_policy` for tool sticky mode and tracker session limit resolution. - Kept `get_tool_injection_sticky_mode()` and `get_tool_tracker_max_sessions()` as compatibility wrappers in `helpers.py`. - Added direct unit tests for defaults, accepted values, and loud rejection of invalid operator configuration. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_tool_injection_policy.py tests/test_memory_tool_session_sticky.py 38 passed in 0.44s python -m ruff check . All checks passed! python -m ruff format --check . 1069 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 410 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13 - Exact command / steps: Ran focused tool injection policy tests, existing memory tool sticky suite, full ruff, format check, mypy, and staged gitleaks scan. - Observed result: Existing sticky memory-tool behavior remains green while extracted policy parsing is covered directly. - Not tested: Full repository pytest suite locally; CI covers the broader matrix. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and changelog updates are not applicable for this internal refactor. The default-branch Dependabot alerts reported during push are pre-existing and unrelated to this PR. Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## Description Extracts proxy tool-injection decision logging from `headroom.proxy.helpers` into a focused logging policy module. The public helper function remains in place and delegates to the new module, so existing injection call sites keep their current API while the logging format has direct tests. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `headroom.proxy.tool_injection_logging` with the shared `ToolInjectionDecision` type and structured logging helper. - Updated `helpers.log_tool_injection_decision` to delegate to the logging policy module while preserving the existing helper API. - Added tests that assert the emitted structured fields and verify tool names/contents are not logged. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_tool_injection_logging.py tests/test_memory_tool_session_sticky.py tests/test_ccr_tool_always_on.py tests/test_corrupt_golden_bytes_recovery.py tests/test_issue_728_empty_tools_injection.py 60 passed in 0.95s python -m ruff check . All checks passed! python -m ruff format --check . 1069 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 410 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, clean worktree from `headroomlabs/main` at `d2170b19`. - Exact command / steps: Ran targeted logging, memory injection, CCR injection, corrupt-byte, and empty-tool regression tests plus ruff, ruff-format, mypy, and staged gitleaks scan. - Observed result: All targeted tests and local gates passed; staged secret scan found no leaks. - Not tested: Full Docker/native wrapper CI locally; covered by repository CI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes Documentation and changelog updates are not applicable for this internal refactor. The push reported existing default-branch Dependabot vulnerabilities; this PR's staged gitleaks scan passed and CI security checks are expected to validate the branch. Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## Description Extracts CCR marker freshness and retrieval-tool injection decision policy from `headroom.proxy.helpers` into a focused pure module. Existing helper functions remain as compatibility wrappers for current Anthropic/OpenAI handler imports. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `headroom.proxy.ccr_marker_policy` for new-marker detection and frozen-prefix tool injection decisions. - Kept `helpers.has_new_ccr_markers()` and `helpers.should_inject_ccr_tool()` as compatibility wrappers. - Added direct policy tests for replayed markers, genuinely new markers, missing prior forwards, empty current hashes, and frozen-prefix override behavior. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_ccr_marker_policy.py tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_proxy_handler_helpers.py::TestHasNewCcrMarkers 16 passed in 0.91s python -m ruff check . All checks passed! python -m ruff format --check . 1069 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 410 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13 - Exact command / steps: Ran direct CCR marker policy tests, frozen-prefix coupling tests, existing helper marker freshness tests, full ruff, format check, mypy, and staged gitleaks scan. - Observed result: Existing frozen-prefix CCR behavior remains green while the marker freshness and injection decision policy is directly covered. - Not tested: Full repository pytest suite locally; CI covers the broader matrix. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and changelog updates are not applicable for this internal refactor. The default-branch Dependabot alerts reported during push are pre-existing and unrelated to this PR. Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## Description Extracts opt-in Codex wire-debug formatting from `helpers.py` into `headroom.proxy.wire_debug_format_policy`. The existing helper functions now delegate to the pure policy so filename-safe event names and proxy-log previews are directly testable. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `wire_debug_format_policy.py` for safe wire-debug name fragments and compact log previews. - Kept `_safe_event_name` and `_wire_debug_preview` in `helpers.py` as compatibility delegates. - Added direct tests for unsafe-name replacement, length capping, JSON preview compaction, byte decoding/truncation, and `None` handling. - Carried forward the LiteLLM callback compatibility shim needed for current mypy on `main`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests\test_wire_debug_format_policy.py 5 passed in 0.19s python -m ruff check . All checks passed! python -m ruff format --check . 1095 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 409 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, branch `jd/architecture-slice-28`. - Exact command / steps: ran focused wire-debug format policy tests, ruff, ruff format check, mypy, and staged gitleaks scan. - Observed result: formatting policy behavior is directly covered and local lint/type/security checks pass. - Not tested: live wire-debug capture writing; this slice preserves the existing helper entry points. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and changelog updates are N/A for this internal architecture-only refactor. The push reported existing default-branch Dependabot alerts; no staged secret leaks were found for this PR. --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## Description Extracts lossy diagnostic byte decoding from `helpers.py` into `headroom.proxy.diagnostic_decode_policy`. Protocol parsers stay strict while the diagnostic/logging path has a dedicated, directly tested policy. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `diagnostic_decode_policy.py` for UTF-8 diagnostic decoding with replacement characters. - Kept `helpers.safe_decode_for_logging` delegating to the extracted policy for existing callers. - Added direct tests for valid UTF-8, invalid byte replacement, max-byte truncation, and helper delegation. - Carried forward the LiteLLM callback compatibility shim needed for current mypy on `main`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests\test_diagnostic_decode_policy.py 4 passed in 0.18s python -m ruff check . All checks passed! python -m ruff format --check . 1095 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 409 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, branch `jd/architecture-slice-30`. - Exact command / steps: ran focused diagnostic decode policy tests, ruff, ruff format check, mypy, and staged gitleaks scan. - Observed result: diagnostic decode behavior is directly covered and local lint/type/security checks pass. - Not tested: live upstream error responses; existing helper import path remains intact. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and changelog updates are N/A for this internal architecture-only refactor. The push reported existing default-branch Dependabot alerts; no staged secret leaks were found for this PR. --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## Description Extracts the privacy-preserving memory-query log hash from `helpers.py` into `headroom.proxy.query_log_policy`. The helper import path remains intact, while the log identifier formula is now directly testable as a pure policy. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `query_log_policy.py` with the BLAKE2b-based short query hash formula. - Kept `helpers.hash_query_for_log` delegating to the extracted policy for existing callers. - Added direct tests for stability, short hex shape, content sensitivity, unpaired surrogate handling, and helper delegation. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests\test_query_log_policy.py 4 passed in 0.18s python -m ruff check . All checks passed! python -m ruff format --check . 1069 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 410 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, branch `jd/architecture-slice-32`. - Exact command / steps: ran focused query-log policy tests, ruff, ruff format check, mypy, and staged gitleaks scan. - Observed result: query log hash behavior is directly covered and local lint/type/security checks pass. - Not tested: live memory injection logging; existing helper entry point remains intact. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and changelog updates are N/A for this internal architecture-only refactor. The push reported existing default-branch Dependabot alerts; no staged secret leaks were found for this PR. --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## Description Extracts memory-injection mode resolution from `helpers.py` into `headroom.proxy.memory_injection_mode_policy`. The proxy still reads `HEADROOM_MEMORY_INJECTION_MODE` at request time, while the allowed values/default/error contract is now pure and directly tested. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `memory_injection_mode_policy.py` with the allowed mode type, env name/default, and resolver. - Kept `helpers.get_memory_injection_mode` as the request-time env reader and compatibility entry point. - Added direct policy tests for defaults, accepted values, normalization, and invalid mode rejection. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests\test_memory_injection_mode_policy.py tests\test_proxy_system_prompt_immutable.py 10 passed in 14.60s python -m ruff check . All checks passed! python -m ruff format --check . 1069 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 410 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, branch `jd/architecture-slice-33`. - Exact command / steps: ran new memory injection mode policy tests, existing system-prompt immutability tests, ruff, ruff format check, mypy, and staged gitleaks scan. - Observed result: memory injection mode behavior remains covered and local lint/type/security checks pass. - Not tested: live proxy request; existing helper entry point remains intact. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and changelog updates are N/A for this internal architecture-only refactor. The push reported existing default-branch Dependabot alerts; no staged secret leaks were found for this PR. --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## Description Extracts beta-header stickiness configuration parsing from `headroom.proxy.helpers` into a focused policy module. This keeps the environment-driven mode and LRU bound validation independently testable while preserving the helper functions used by the session beta tracker. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `headroom.proxy.beta_header_policy` for beta sticky mode and tracker session limit resolution. - Kept `get_beta_header_sticky_mode()` and `get_beta_tracker_max_sessions()` as compatibility wrappers in `helpers.py`. - Added direct unit tests for defaults, accepted values, and loud rejection of invalid operator configuration. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_beta_header_policy.py tests/test_anthropic_beta_session_sticky.py tests/test_openai_beta_session_sticky.py 52 passed in 0.41s python -m ruff check . All checks passed! python -m ruff format --check . 1069 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 410 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13 - Exact command / steps: Ran focused beta policy tests, existing Anthropic/OpenAI beta sticky suites, full ruff, format check, mypy, and staged gitleaks scan. - Observed result: Existing beta sticky behavior remains green while extracted policy parsing is covered directly. - Not tested: Full repository pytest suite locally; CI covers the broader matrix. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and changelog updates are not applicable for this internal refactor. The default-branch Dependabot alerts reported during push are pre-existing and unrelated to this PR. --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
…#2007) ## Description Extracts memory-tool golden byte replay and canonicalization from `headroom.proxy.helpers.apply_session_sticky_memory_tools` into a focused policy module. The session tracker, skip/deduplication decisions, and logging remain in the existing helper; the byte-level replay policy now has direct coverage. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added `headroom.proxy.memory_golden_policy` for replaying stored memory golden bytes and canonicalizing fresh memory tool definitions. - Updated `apply_session_sticky_memory_tools` to delegate golden-byte decode/canonicalization while preserving tracker and logging behavior. - Added direct tests for golden replay, invalid/corrupt bytes, non-UTF-8 bytes, and serializer parity with the existing helper. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_memory_golden_policy.py tests/test_memory_tool_session_sticky.py tests/test_corrupt_golden_bytes_recovery.py tests/test_issue_728_empty_tools_injection.py 50 passed in 0.87s python -m ruff check . All checks passed! python -m ruff format --check . 1069 files already formatted python -m mypy headroom --ignore-missing-imports Success: no issues found in 410 source files gitleaks protect --staged --no-banner --redact no leaks found ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, clean worktree from `headroomlabs/main` at `d2170b19`. - Exact command / steps: Ran targeted memory golden replay/sticky injection/corrupt-byte regression tests plus ruff, ruff-format, mypy, and staged gitleaks scan. - Observed result: All targeted tests and local gates passed; staged secret scan found no leaks. - Not tested: Full Docker/native wrapper CI locally; covered by repository CI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes Documentation and changelog updates are not applicable for this internal refactor. The push reported existing default-branch Dependabot vulnerabilities; this PR's staged gitleaks scan passed and CI security checks are expected to validate the branch. Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
… re-entrancy, misc correctness and a11y fixes
…ents # Conflicts: # headroom/cli/doctor.py # headroom/dashboard/templates/dashboard.html # headroom/proxy/output_savings.py # headroom/proxy/server.py # tests/test_proxy_dashboard_stats_cache.py # uv.lock
| try: | ||
| user_config.update(body) | ||
| except ValueError as exc: | ||
| return JSONResponse(status_code=400, content={"error": str(exc)}) |
…op dead server.py imports
…usage in user_config
Owner
Author
|
Closing to reopen clean: main now has upstream/main merged in, so this PR's diff no longer drags in unrelated pre-existing commits. Opening a fresh PR against the updated main. |
khaosdoctor
pushed a commit
that referenced
this pull request
Jul 13, 2026
…ound-trip (headroomlabs-ai#2079) ## Description Two related content-loss bugs in the Gemini `contents[]` <-> `messages[]` compression round-trip. Both drop or misplace real user content that entries with **non-text** parts should carry through untouched. They share the same theme (non-text preservation), so they're bundled here as two commits. ### 1. Google batch handler restores preserved entries by the wrong index (`handlers/batch.py`) The `batchGenerateContent` handler restored preserved (non-text) entries with the raw-index loop that commit headroomlabs-ai#836 (`_rebuild_gemini_contents`) replaced in the three non-batch Gemini handlers: ```python for orig_idx, original_content in preserved_contents.items(): if orig_idx < len(optimized_contents): optimized_contents[orig_idx] = original_content ``` `preserved_indices` are indices into the **original** `contents[]`, but `optimized_contents` is a **shorter** list (text-less entries produce no message). Indexing `optimized_contents` by `orig_idx` overwrites the wrong entry and drops any preserved entry whose original index is past the optimized length. For: ```python [user text, model functionCall, user functionResponse, model text] ``` the batch was forwarded to Google as **two** entries: the model's answer overwritten by the functionCall, and the functionResponse dropped. Unlike `gemini.py` there is no `if optimized_messages != messages` gate, so it runs on every mixed batch item. **Fix:** use the shared `_rebuild_gemini_contents` interleaving helper. ### 2. Code-execution parts not detected as non-text (`handlers/gemini.py`) `_has_non_text_parts` only recognized `inlineData`/`fileData`/`functionCall`/`functionResponse`. Gemini's code-execution feature emits `executableCode` and `codeExecutionResult` parts, echoed back in `contents[]` on later turns. Because they weren't detected: - a mixed `text`+`executableCode` entry lost its code payload (only the text survived the round-trip); - a text-less `executableCode`+`codeExecutionResult` entry was treated as a phantom in `_rebuild_gemini_contents` — it consumed the next optimized message, dropping the whole code turn and shifting a following user turn into the model's role slot (corrupting role alternation). **Fix:** add both keys to the non-text detection so those entries are preserved verbatim. Closes: no issue filed — both found while auditing the Gemini contents<->messages round-trip. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/handlers/batch.py`: use `_rebuild_gemini_contents` instead of the raw-index restore loop. - `headroom/proxy/handlers/gemini.py`: recognize `executableCode` / `codeExecutionResult` in `_has_non_text_parts`. - `tests/test_proxy_handlers_batch.py`: add `test_handle_google_batch_create_preserves_functioncall_response_order`, driving the handler with the **real** Gemini converters (the existing batch tests stub them, which hid the bug); mix `GeminiHandlerMixin` into the shared `DummyBatchHandler` so `_rebuild_gemini_contents` is available. - `tests/test_google_multimodal.py`: extend the parametrized `test_each_non_text_key_detected` to the two new keys, and add `test_code_execution_entry_survives`. ## Testing - [x] New regression tests added (`tests/test_proxy_handlers_batch.py`, `tests/test_google_multimodal.py`) - [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17` - [ ] Full `pytest` deferred to CI (local-OOM reason below). ```text $ uvx ruff@0.15.17 check headroom/proxy/handlers/batch.py headroom/proxy/handlers/gemini.py \ tests/test_proxy_handlers_batch.py tests/test_google_multimodal.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.10, headroom from this branch. Importing `headroom` pulls in the torch/transformers stack and a full `pytest` gets OOM-killed on this box, so I verified the interleaving/detection with dependency-free scripts (replicating the Gemini converters, the old loop, and `_rebuild_gemini_contents`) and left the full pytest to CI. - Exact command / steps: ran two standalone scripts. Script 1 rebuilds a Gemini batch request with `preserved_indices` holding a `functionCall`/`functionResponse` pair and compares the old raw-index loop against `_rebuild_gemini_contents`. Script 2 feeds a `codeExecutionResult` entry through `_has_non_text_parts` and the preserve path with and without the two new allowlist keys. Also ran `uvx ruff@0.15.17 check` on the changed files and tests. - Observed result: the old batch loop drops the `functionResponse` and overwrites the answer (4 parts collapse to 2); `_rebuild_gemini_contents` keeps all 4. Without the new keys the code-execution entry is dropped/shifted (2 parts, code absent); with them it survives intact (3 parts, code present). Lint clean. See the two blocks below. Batch fix (bug #1): ```text preserved_indices: [1, 2] OLD result parts: ['text', 'functionCall'] len 2 NEW result parts: ['text', 'functionCall', 'functionResponse', 'text'] len 4 GEMINI BATCH REBUILD FIX VERIFIED (old drops response + overwrites answer; new keeps all 4) ``` Code-execution fix (bug #2): ```text (b) OLD len=2 NEW len=3 (a) OLD has code=False NEW has code=True GEMINI CODE-EXECUTION PRESERVE FIX VERIFIED (old drops/shifts; new keeps intact) ``` - Not tested: a live Google/Gemini round-trip (handlers stubbed, as the existing tests do). Full local `pytest` deferred to CI (OOM, per above). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes — ran lint + standalone logic checks; full pytest deferred to CI (local OOM, disclosed above) - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Two small behavioral changes (one loop -> shared helper, two keys added to an allowlist) plus regression tests; no new dependencies. Both complete/extend the non-text preservation the non-batch handlers already do (the headroomlabs-ai#836 line). - @JerrettDavis tagging you since you reviewed the recent Gemini fixes. Both of these drop content (functionResponse/images on batch; code-execution on the normal round-trip), so they seemed worth surfacing together. Thanks. --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
khaosdoctor
pushed a commit
that referenced
this pull request
Jul 15, 2026
…eadroomlabs-ai#2123) ## Description `verbosity._ordered_events` and `_parse_session` disagree about empty assistant turns, which desyncs the response list and produces spurious fast-skips. `_parse_session` only creates a `_Response` when an assistant message actually said something: ```python if words > 0 or out_tok > 0: responses.append(_Response(...)) ``` But `_ordered_events` consumes one `responses[ri]` for **every** assistant line, with no matching filter: ```python if ltype == "assistant" and ri < len(responses): out.append((responses[ri].ts, "assistant", responses[ri])) ri += 1 ``` So an assistant turn with no text and no output tokens — for example a pure `tool_use` turn where `usage` is absent — creates no `_Response` at parse time, yet still consumes a slot in `_ordered_events`. That slot actually belongs to a *later* real response, so the two lists drift by one. A human reply that follows the real answer is then paired with the next answer's (future) timestamp, `ts - last_resp.ts` goes negative, and since a negative gap is always below the read-fraction threshold, a spurious `fast_skip` is recorded. That inflates `fast_skip_rate`, which feeds `pressure`, which lowers the recommended verbosity level. The user side of `_ordered_events` already replicates its parse-site filter (`_human_text(...) is None -> continue`); only the assistant side was missing the equivalent guard. That asymmetry is the bug. ## Fix In `_ordered_events`, compute `words`/`out_tok` for the assistant line the same way `_parse_session` does and only consume a response when `words > 0 or out_tok > 0`, keeping the two functions in lockstep. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/learn/verbosity.py`: `_ordered_events` applies the `words > 0 or out_tok > 0` guard on the assistant branch before consuming a response, with a comment explaining the desync. - `tests/test_verbosity_learn.py`: add `_empty_assistant` helper and `test_empty_assistant_message_does_not_desync_fast_skip` (an empty assistant turn before a real answer + a slow reply must not record a fast skip). - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [x] Unit tests pass (`uv run --extra dev pytest tests/test_verbosity_learn.py::TestSignalExtraction::test_empty_assistant_message_does_not_desync_fast_skip -q`) - [x] Linting passes (`uvx ruff@0.15.17 check headroom/learn/verbosity.py tests/test_verbosity_learn.py headroom/memory/factory.py`) - [x] Type checking passes (`uvx mypy==1.20.2 headroom/memory/factory.py`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/learn/verbosity.py tests/test_verbosity_learn.py All checks passed! $ python -m py_compile headroom/learn/verbosity.py tests/test_verbosity_learn.py OK ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing `headroom` pulls in the torch/transformers stack and a full `pytest` gets OOM-killed on this box, so I verified the alignment with a dependency-free script that models the parse-site filter, the old vs new `_ordered_events` consume, and the resulting human-to-response pairing, and left the full pytest to CI. - Exact command / steps: built an event stream `[empty assistant, real answer #1, fast human reply, real answer #2, reply]`, computed the response list from the parse filter, then walked the old (unfiltered) and new (filtered) consume to find the gap between the first human and the response paired before it. - Observed result: old consume pairs the reply with answer #2 (a future timestamp) -> gap `-8` (spurious fast_skip); new consume keeps alignment and pairs it with answer #1 -> gap `+1`. The new test builds a session with an empty assistant turn and a genuinely slow reply and asserts `fast_skips == 0`. - Not tested: a real Claude Code transcript end to end; full local `pytest` deferred to CI (OOM, per above). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes Merged current `main` to pick up the repository-wide mypy cache-key annotation fix, then verified the focused regression locally. the change adds the existing parse-site filter to one branch of a pure file-parsing function, verified by the standalone alignment proof and the new regression test for CI. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
I know this is a big PR, but it revamps the dashboard end to end: new panes for Active Agents, Diagnostics, Waste & Trends, and CCR feedback, a Settings panel for budget/pricing/savings-profile overrides that persist across restarts, a floating help widget, sortable tables, and a methodology modal explaining how savings are calculated.
It's a big diff mostly because of what had to happen structurally to fit it in:
dashboard.htmlwas one 2700-line file and the dashboard routes were inlined inserver.py. I split the HTML into partials, split the Alpine component into core + feature mixins, and moved the routes into their own router. That refactor accounts for most of the line count, the new feature surface itself is smaller than it looks.This branch also carries three older commits (
7f466f7d,d917b649,169aa4a8, all from 2026-06-22) that predate this UI work by two weeks and are already onmain, fixing an unrelated output-savings baseline reload bug. They're along for the ride because they haven't been upstreamed yet, not part of this PR's actual work.Closes #
Type of Change
Changes Made
dashboard_config.jsonand applied on the next proxy launchdashboard.htmlinto partials, split the Alpine component into core + feature mixins, and extracted dashboard routes fromserver.pyintoheadroom/proxy/routers/dashboard.pywiki/metrics.mddocumenting the panes and the Settings panel, and a CHANGELOG.md entryTesting
pytest)ruff check .)mypy headroom)Test Output
The 8 remaining mypy errors are all in
headroom/memory/backends/direct_mem0.py, a file this branch never touches (empty diff against upstream/main). Every mypy error in code this PR actually added or changed is fixed.Almost every pytest failure and collection error traces back to one thing:
headroom/_core.abi3.so(the compiled Rust extension) won't load in my local dev environment, a macOS linker symbol issue (___isPlatformVersionAtLeast) coming out of onnxruntime's CoreML backend. I confirmed this is a real toolchain problem and not something masking a code issue: a cleancargo build --release --manifest-path crates/headroom-py/Cargo.toml --features extension-modulereproduces the identical undefined symbol from scratch. This branch doesn't touch any.rsfile,Cargo.toml, orCargo.lock(confirmed empty diff against upstream/main), so it's a pre-existing environment/SDK problem, not something this PR broke, and CI builds the extension fresh so it shouldn't hit this. A handful of the rest (an embedding-model download test, one OpenAI Codex routing test) are also pre-existing and unrelated to anything this branch touches.One real regression did turn up while I was resolving the merge: it briefly reintroduced this branch's older
headlineSavingsPercentlogic (an active-vs-attempted ratio with a fallback) over upstream's newer, simpler fix from headroomlabs-ai#1653 (stats.tokens?.savings_percent ?? proxy_savings_percent).tests/test_dashboard_token_savings.pycaught it immediately, so it's fixed in this PR, upstream's version wins.Real Behavior Proof
ruff check .,mypy headroom, and the fullpytestsuite. Also attempted a clean rebuild of the native extension (cargo build --release) to unblock manual testing.headroom proxycan't start in this environment right now because the native extension won't load (see above), and rebuilding it from source hits the same onnxruntime/CoreML linker error, so this isn't a quick fix on my end. I need either a working native build (a different machine, or a toolchain fix on this one) or your go-ahead to treat this as a known local limitation, before I can actually drive the dashboard and grab real screenshots.Review Readiness
Checklist
Screenshots (if applicable)
Additional Notes
Manual testing is the one item I couldn't complete myself, it's blocked by a native-extension build issue on this machine, not by anything in this diff. Let me know how you want to handle it (fix the toolchain here, test on another machine, or proceed without it) and I'll close that out.