Skip to content

fix(install): guard install_agent_ensure against duplicate runtime spawns#1301

Merged
JerrettDavis merged 1 commit into
headroomlabs-ai:mainfrom
ipriyaaanshu:fix/install-agent-duplicate-runtimes
Jun 24, 2026
Merged

fix(install): guard install_agent_ensure against duplicate runtime spawns#1301
JerrettDavis merged 1 commit into
headroomlabs-ai:mainfrom
ipriyaaanshu:fix/install-agent-duplicate-runtimes

Conversation

@ipriyaaanshu

@ipriyaaanshu ipriyaaanshu commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

Description

install_agent_ensure in cli/install.py only checked probe_ready(health_url). If the proxy was alive-but-not-ready (e.g. during cold start while tokenizers load — ~38s on Windows), probe_ready returned false and it unconditionally called _start_deploymentstart_detached_agent, spawning a second runtime without:

  1. acquiring acquire_runtime_start_lock
  2. checking runtime_status
  3. stopping the existing instance

Two proxies then contend for 127.0.0.1:<port>; only one can bind, and the deployment ends up wedged (never ready). Every subsequent ensure spawns yet another runtime → restart storm.

By contrast, the hook path cli/init.py:_ensure_profile_running does it correctly: it acquires the start-lock, checks runtime_status, and stop_runtimes a wedged instance before starting a fresh one.

Closes #1151.

Changes Made

  • Added acquire_runtime_start_lock to the imports from install.runtime in headroom/cli/install.py
  • Rewrote install_agent_ensure to mirror the guarded pattern from _ensure_profile_running in cli/init.py:
    • Fast-path probe: if proxy is already ready, return immediately (preserves existing behavior)
    • Lock acquisition: acquire acquire_runtime_start_lock — if another ensure holds it, return without spawning (prevents duplicate)
    • Double-checked locking: re-probe probe_ready after acquiring the lock (race window handled)
    • Wedged instance detection: if runtime_status says "running" but proxy isn't ready within 15s grace period, call stop_runtime before starting fresh
    • Fall through to _start_deployment only when truly needed
  • Added _STARTUP_READY_TIMEOUT_SECONDS = 15 constant (matching the value used in _ensure_profile_running)
  • Failure propagation (addresses @JerrettDavis's review feedback): removed the try/except Exception wrapper around the guarded block. install agent ensure is an automation-facing CLI command and must exit non-zero on failure so callers can distinguish a successful ensure from a failed one. The init.py hook path retains its try/except because silent retry is intentional there. The control flow is shared; the error contract is intentionally different because the call sites have different needs.
  • Added 5 regression tests in tests/test_cli/test_install_cli.py:
    • test_install_agent_ensure_no_spawn_when_lock_not_acquired — verifies no runtime spawned when lock is contended (the core bug)
    • test_install_agent_ensure_stops_wedged_runtime_before_restart — verifies stop_runtime is called BEFORE _start_deployment when instance is wedged (ordering assertion: calls.index("stop") < calls.index("start_deployment"))
    • test_install_agent_ensure_starts_when_stopped_and_lock_acquired — verifies the normal start path including the real _start_deploymentstart_detached_agent wiring
    • test_install_agent_ensure_no_duplicate_spawn_after_lock_recheck — verifies double-checked locking prevents duplicate when proxy becomes ready between initial probe and lock acquisition
    • test_install_agent_ensure_propagates_start_deployment_failurenew regression test for the failure-propagation fix: monkeypatches _start_deployment to raise click.ClickException("simulated start failure") and asserts both exit_code != 0 and that the error message survives in output

Testing

  • Unit tests pass (pytest)
  • Linting passes (ruff check .)
  • Type checking passes (mypy headroom)
  • New tests added for new functionality
  • Manual testing performed

Test Output

$ uv run python -m pytest tests/test_cli/test_install_cli.py -v --tb=short

tests/test_cli/test_install_cli.py::test_install_apply_starts_service_supervisor PASSED [  6%]
tests/test_cli/test_install_cli.py::test_install_status_includes_backend_from_health_probe PASSED [ 12%]
tests/test_cli/test_install_cli.py::test_install_restart_uses_internal_helpers PASSED [ 18%]
tests/test_cli/test_install_cli.py::test_install_apply_rejects_invalid_profile PASSED [ 25%]
tests/test_cli/test_install_cli.py::test_install_apply_rejects_provider_scope_targets_without_support PASSED [ 31%]
tests/test_cli/test_install_cli.py::test_install_apply_restores_previous_deployment_after_failed_update PASSED [ 37%]
tests/test_cli/test_install_cli.py::test_install_start_rejects_task_lifecycle PASSED [ 43%]
tests/test_cli/test_install_cli.py::test_install_apply_uses_docker_runtime_for_persistent_docker PASSED [ 50%]
tests/test_cli/test_install_cli.py::test_install_remove_continues_when_runtime_teardown_errors PASSED [ 56%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_reports_already_healthy PASSED [ 62%]
tests/test_cli/test_install_cli.py::test_install_agent_run_exits_with_foreground_status PASSED [ 68%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_no_spawn_when_lock_not_acquired PASSED [ 75%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_stops_wedged_runtime_before_restart PASSED [ 81%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_starts_when_stopped_and_lock_acquired PASSED [ 87%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_no_duplicate_spawn_after_lock_recheck PASSED [ 93%]
tests/test_cli/test_install_cli.py::test_install_agent_ensure_propagates_start_deployment_failure PASSED [100%]

============================== 16 passed in 0.29s ==============================
$ uv run ruff check headroom/cli/install.py tests/test_cli/test_install_cli.py
All checks passed!

$ uv run ruff format --check headroom/cli/install.py tests/test_cli/test_install_cli.py
2 files already formatted

$ uv run mypy headroom/cli/install.py --ignore-missing-imports
Success: no issues found in 1 source file

Real Behavior Proof

  • Environment: Python 3.11.14, Linux 6.17.0, headroom dev environment (uv-synced), rebased onto upstream/main at 3be2526b
  • Exact command / steps: uv run python -m pytest tests/test_cli/test_install_cli.py -v --tb=short (and the ruff + mypy commands above)
  • Observed result: All 16 tests pass (11 existing + 5 new regression tests). The 5 new tests verify: (1) no-spawn when the lock is contended, (2) stop_runtime ordering before _start_deployment on a wedged instance, (3) normal start path, (4) double-checked locking after the lock is acquired, (5) failure propagation when _start_deployment raises — this last test is the regression for @JerrettDavis's review feedback. ruff check, ruff format --check, and mypy all pass clean.
  • Not tested: Live deployment with concurrent install agent ensure invocations on Windows (only unit tests with monkeypatched runtime functions). The fix mirrors the proven pattern from _ensure_profile_running which is already battle-tested in the init hook path.

Review Readiness

  • I have performed a self-review of my own code
  • This PR is ready for human review

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have updated the CHANGELOG.md if applicable

Screenshots (if applicable)

N/A

Additional Notes

Triage of labels on this PR:

  • status: needs author actionstale. The 4 Real Behavior Proof fields (Environment, Exact command / steps, Observed result, Not tested) are all present in this body. The bot snapshot was taken before the body was filled in. Requesting the label be dropped on the next bot run.
  • status: ci failingCI env, not caused by this PR. install-native (macos-latest) and wrap-native (macos-latest) fail during the editable Rust/Python extension build with ld: library 'clang_rt.osx' not found, which is before this command path runs. @JerrettDavis confirmed this is not caused by the PR. All Linux jobs, all unit/integration/E2E jobs, lint, commitlint, template check, and Docker E2E jobs are green. mergeable: MERGEABLE is the actual gate.

CHANGELOG: not updated — this is a single bug fix in an unreleased section, and the maintainers have not requested CHANGELOG entries for individual PRs in past PRs in this repo. Happy to add an entry under ## Unreleased if requested.

@github-actions

github-actions Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

PR governance

This PR does not yet satisfy the required template fields:

  • Fill in Real Behavior ProofEnvironment.
  • Fill in Real Behavior ProofExact command / steps.
  • Fill in Real Behavior ProofObserved result.
  • Fill in Real Behavior ProofNot tested.
  • Check I have performed a self-review before requesting human review.

Please update the PR body, or move the PR back to draft while it is still in progress.

@github-actions github-actions Bot added status: needs author action Pull request body or readiness checklist still needs author updates status: ci failing Required or reported CI checks are failing labels Jun 23, 2026
@ipriyaaanshu ipriyaaanshu force-pushed the fix/install-agent-duplicate-runtimes branch from be8b62c to 5d04e96 Compare June 23, 2026 05:36
@github-actions github-actions Bot added status: ci failing Required or reported CI checks are failing and removed status: ci failing Required or reported CI checks are failing labels Jun 23, 2026
@ipriyaaanshu

Copy link
Copy Markdown
Contributor Author

^^ correction: the 'Real Behavior Proof' sections were filled in before the template check ran. All PR governance requirements are met. Ready for review.

@JerrettDavis JerrettDavis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tightening this path; the lock + double-check structure is the right direction. One behavior regression should be fixed before merge.

install_agent_ensure now wraps the whole guarded start/restart block in except Exception, prints Deployment '...' ensure failed: ..., and then returns normally. That means failures from runtime_status, stop_runtime, _start_deployment, or the final readiness wait become exit code 0. This command is automation-facing, so a failed ensure must remain a failed command; otherwise callers can treat a deployment as healthy when it never started.

Please let click.ClickException / startup failures propagate, or convert them to a non-zero Click failure. A focused regression should cover _start_deployment raising and assert runner.invoke(...).exit_code != 0 rather than success with a message.

The two red macOS native jobs I checked fail while building the editable Rust/Python extension (ld: library 'clang_rt.osx' not found) before this command path runs, so I’m not treating those as caused by this PR. The code-level failure semantics above are the blocker.

…awns

install_agent_ensure in cli/install.py only checked probe_ready(health_url).
If the proxy was alive-but-not-ready (e.g. during cold start while tokenizers
load — ~38s on Windows), probe_ready returned false and it unconditionally
called _start_deployment → start_detached_agent, spawning a second runtime
without acquiring acquire_runtime_start_lock, checking runtime_status, or
stopping the existing instance. Two proxies then contended for
127.0.0.1:<port>; only one could bind, and the deployment ended up wedged
(never ready). Every subsequent ensure spawned yet another runtime → restart
storm.

By contrast, the hook path cli/init.py:_ensure_profile_running does it
correctly: it acquires the start-lock, checks runtime_status, and stop_runtime
s a wedged instance before starting a fresh one.

Closes headroomlabs-ai#1151.

## Changes

- Added acquire_runtime_start_lock to the imports from install.runtime in
  headroom/cli/install.py
- Rewrote install_agent_ensure to mirror the guarded pattern from
  _ensure_profile_running in cli/init.py:
  - Fast-path probe: if proxy is already ready, return immediately
  - Lock acquisition: acquire acquire_runtime_start_lock — if another
    ensure holds it, return without spawning
  - Double-checked locking: re-probe probe_ready after acquiring the lock
  - Wedged instance detection: if runtime_status says 'running' but proxy
    isn't ready within 15s, call stop_runtime before starting fresh
  - Fall through to _start_deployment only when truly needed
- Added _STARTUP_READY_TIMEOUT_SECONDS = 15 constant
- **Failure propagation (review feedback):** removed the try/except Exception
  wrapper around the guarded block. install agent ensure is an automation-
  facing CLI command and must exit non-zero on failure so callers can
  distinguish a successful ensure from a failed one. The init.py hook path
  retains its try/except because silent retry is intentional there. The
  control flow is shared; the error contract is intentionally different
  because the call sites have different needs.

## Tests

- test_install_agent_ensure_no_spawn_when_lock_not_acquired — verifies no
  runtime spawned when the lock is contended (core bug)
- test_install_agent_ensure_stops_wedged_runtime_before_restart — verifies
  stop_runtime is called BEFORE _start_deployment when instance is wedged
  (ordering assertion: calls.index('stop') < calls.index('start_deployment'))
- test_install_agent_ensure_starts_when_stopped_and_lock_acquired — verifies
  the normal start path including the real _start_deployment →
  start_detached_agent wiring
- test_install_agent_ensure_no_duplicate_spawn_after_lock_recheck —
  verifies double-checked locking prevents duplicate when proxy becomes
  ready between initial probe and lock acquisition
- test_install_agent_ensure_propagates_start_deployment_failure — new
  regression test for the failure-propagation fix: monkeypatches
  _start_deployment to raise click.ClickException, asserts exit_code != 0
  and that the error message survives in output
@ipriyaaanshu ipriyaaanshu force-pushed the fix/install-agent-duplicate-runtimes branch from 5d04e96 to 36a0aba Compare June 24, 2026 04:48
@ipriyaaanshu

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @JerrettDavis — you're right, the silent except Exception was a real contract bug for an automation-facing CLI. Fixed in this push.

What changed

headroom/cli/install.py — removed the try: ... except Exception: echo + return wrapper around the guarded block. install agent ensure now lets click.ClickException and any startup failure propagate naturally as a non-zero exit, matching the contract a script or supervisor wrapper would expect.

The control flow still mirrors _ensure_profile_running from init.py (lock acquisition → double-checked probe → runtime_status → 15s grace → stop_runtime_start_deployment), but intentionally diverges on the error contract since the call sites have different needs: the hook path is a silent retry, the CLI path is a hard-fail command.

tests/test_cli/test_install_cli.py — three changes:

  1. New regression test test_install_agent_ensure_propagates_start_deployment_failure: monkeypatches _start_deployment to raise click.ClickException("simulated start failure") and asserts both exit_code != 0 AND that the error message survives in output. This is the test the review asked for and would catch any future regression that re-introduces the swallow.

  2. Updated test_install_agent_ensure_stops_wedged_runtime_before_restart: now mocks _start_deployment directly (since the silent-swallow is gone, the real function would block on a 45s wait), and the assertions check the actual bug guard — calls.index("stop") < calls.index("start_deployment") — proving stop_runtime is called before the fresh start.

  3. The existing test_install_agent_ensure_starts_when_stopped_and_lock_acquired covers the wiring from _start_deploymentstart_detached_agent, so coverage of the normal start path is preserved.

Triage of the other items on this PR

Item Bucket Status
try/except swallowing Real code blocker Fixed
status: needs author action label Stale governance The 4 Real Behavior Proof fields (Environment, Exact command / steps, Observed result, Not tested) are all present in the PR body — the bot snapshot was taken before the body was filled in. Requesting the label be dropped.
status: ci failing (macOS native) CI env install-native (macos-latest) and wrap-native (macos-latest) fail with ld: library 'clang_rt.osx' not found during the editable Rust/Python extension build, before this command path runs. As you noted, not caused by this PR. All Linux jobs, all unit/integration/E2E jobs, lint, commitlint, and template checks are green.

Verification

$ uv run python -m pytest tests/test_cli/test_install_cli.py -v --tb=short
============================== 16 passed in 0.24s ==============================

$ uv run ruff check headroom/cli/install.py tests/test_cli/test_install_cli.py
All checks passed!

$ uv run ruff format --check headroom/cli/install.py tests/test_cli/test_install_cli.py
2 files already formatted

Rebased onto latest upstream/main (3be2526b) — branch was 16 commits behind, including the uv.lock --check workflow that runs against the merged state — and force-pushed as a single clean commit (1 file, headroom/cli/install.py; 1 file, tests/test_cli/test_install_cli.py; no uv.lock churn). Ready for re-review.

@github-actions github-actions Bot removed the status: ci failing Required or reported CI checks are failing label Jun 24, 2026

@JerrettDavis JerrettDavis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed after the failure-propagation fix. The guarded install agent ensure flow keeps the lock/double-check/wedged-runtime restart behavior, and startup failures now propagate as non-zero command failures instead of being swallowed. The new regression covers _start_deployment raising, and CI is green on the updated commit. No remaining blocker found.

@JerrettDavis JerrettDavis merged commit 8da0b4e into headroomlabs-ai:main Jun 24, 2026
25 checks passed
@github-actions github-actions Bot mentioned this pull request Jun 25, 2026
chopratejas pushed a commit that referenced this pull request Jun 29, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>0.28.0</summary>

##
[0.28.0](v0.27.0...v0.28.0)
(2026-06-29)


### Features

* add --disable-kompress-fallback to restore legacy PASSTHROUGH fallback
([#1185](#1185))
([f309244](f309244))
* add first-class OpenCode support (wrap, learn, mcp install)
([#559](#559))
([91cd210](91cd210))
* add HEADROOM_KEEPALIVE_EXPIRY to keep upstream connections warm
([#1124](#1124))
([85786b3](85786b3))
* **azure-foundry:** derive upstream URL from ANTHROPIC_FOUNDRY_RESOURCE
([#1138](#1138))
([e5031b0](e5031b0))
* **cache:** attribute prompt-cache misses to TTL lapse vs prefix change
([#1313](#1313))
([#1343](#1343))
([4658721](4658721))
* **code:** add Perl support to code-aware compressor
([#1125](#1125))
([f39858c](f39858c))
* headroom wrap opencode / unwrap opencode CLI
([#1105](#1105))
([b4571cc](b4571cc))
* **learn:** weight loops in Headroom Learn + RTK-loop eval
([#1160](#1160))
([14e8dc4](14e8dc4))
* **learn:** write per-project learnings to CLAUDE.local.md by default
([#1115](#1115))
([ced75e4](ced75e4))
* **proxy:** add request timeout config
([#738](#738))
([c0745d4](c0745d4))
* **proxy:** pilot hardening — inbound auth, security headers, audit
log, air-gap switch
([#1537](#1537))
([546ab55](546ab55))
* **proxy:** support glob patterns in exclude_tools
([#870](#870))
([#1259](#1259))
([a2159c0](a2159c0))
* **read-maturation:** activity-based hold-back Read maturation
(Mechanism B)
([#1068](#1068))
([723b80c](723b80c))
* **savings:** durable savings ledger + headroom savings command
([#1127](#1127))
([978ffa0](978ffa0))
* **wrap:** add --1m to preserve the 1M context window on wrap claude
([#1158](#1158))
([#1351](#1351))
([b50d9c1](b50d9c1))
* **wrap:** make tokensave the primary coding-task compressor, Serena
the backup
([#1230](#1230))
([dca9853](dca9853))


### Bug Fixes

* **agent-evals:** Phase 0 — coding-agent accuracy A/B framework
([#1037](#1037))
([84f9871](84f9871))
* **agno:** tolerate streaming tool-call SDK objects in parser
([#1312](#1312))
([#1336](#1336))
([5986c22](5986c22))
* **bedrock:** add boto3 1.41 + CRT for aws login credentials
([#1486](#1486))
([4db3bc9](4db3bc9))
* bump codebase-memory-mcp to v0.8.1
([#1284](#1284))
([530318b](530318b))
* **ccr:** make headroom_retrieve a hash-only full-content lookup
([#1532](#1532))
([c2fc4d3](c2fc4d3))
* **ccr:** propagate --no-ccr-marker flag to all compressors
([#1022](#1022))
([#1197](#1197))
([0c9b42a](0c9b42a))
* **ccr:** skip Anthropic marker emission when tool injection is
deferred
([#1273](#1273))
([2cae13d](2cae13d))
* **ci:** extend gitleaks allowlist to cover test fixtures + verified
examples
([#1539](#1539))
([d2565a6](d2565a6))
* **ci:** guarantee model present in test shards to end cache-miss
flakiness
([#1399](#1399))
([2e29c72](2e29c72))
* **ci:** normalize Windows CRLF line endings in PR governance script
([#1012](#1012))
([5194388](5194388))
* **cli:** add explicit UTF-8 encoding to file I/O in wrap commands
([#1126](#1126))
([#1164](#1164))
([a0cb798](a0cb798))
* **cli:** fall back gracefully when embedding-server sidecar is absent
([#1206](#1206))
([38f1404](38f1404))
* **cli:** harden all CLI surfaces + fix docs accuracy
([#1491](#1491))
([bd76235](bd76235))
* **cli:** wire --http2/--no-http2 (HEADROOM_HTTP2) into proxy command
([#1373](#1373))
([e06b616](e06b616))
* **cli:** wire --rpm/--tpm and HEADROOM_RPM/HEADROOM_TPM to the Click
proxy command
([#1375](#1375))
([8aab8f2](8aab8f2))
* **code:** slice tree-sitter byte offsets as UTF-8
([#1332](#1332))
([8238402](8238402))
* **code:** validate Python compressed syntax
([#1302](#1302))
([cbd361d](cbd361d))
* **code:** verify a real parse in tree-sitter availability check
([#1231](#1231))
([#1299](#1299))
([5e0bb69](5e0bb69))
* **codex:** retag threads on init so Codex Desktop history stays
visible ([#961](#961))
([#1349](#1349))
([e6bbc40](e6bbc40))
* **codex:** stop pinning Codex memory MCP to one project db
([#1269](#1269))
([ad7993b](ad7993b))
* **dashboard:** include RTK stats in the historical tab
([#1324](#1324))
([35939c3](35939c3))
* **deps:** remediate dependency CVEs and publish SBOM
([#1509](#1509))
([5771a80](5771a80))
* **docker:** persist session history across container revisions
([#1118](#1118))
([5912d65](5912d65))
* **gemini:** offload compression to the executor
([#1382](#1382))
([615848e](615848e))
* **gemini:** resolve Google model capabilities through ModelRegistry
([#1276](#1276))
([17ecad9](17ecad9))
* **install:** guard install_agent_ensure against duplicate runtime
spawns
([#1301](#1301))
([8da0b4e](8da0b4e))
* **install:** repair macOS launchd restart/start lifecycle
([#1290](#1290))
([da1a397](da1a397))
* **install:** stop duplicating ENTRYPOINT in persistent-docker runtime
command ([#833](#833))
([#1348](#1348))
([feedead](feedead))
* **io:** use UTF-8 with locale fallback and preserve line endings on
config/text I/O
([#1498](#1498))
([1baa04e](1baa04e))
* **kompress:** hard override keeps must-keep tokens regardless of model
score ([#1400](#1400))
([42612c8](42612c8))
* **langchain:** disable streaming on wrapped model during ainvoke()
([#1287](#1287))
([3590046](3590046))
* **mcp:** register managed installs with a resolvable headroom command
([#1386](#1386))
([22def93](22def93))
* **mcp:** report correct savings_percent in headroom_compress
([#1106](#1106))
([f216e43](f216e43))
* **opencode:** write local MCP config
([#1381](#1381))
([6c83790](6c83790))
* **packaging:** move hnswlib to optional [vector] extra so [all] needs
no C++ toolchain
([#1499](#1499))
([80fa086](80fa086))
* patch rtk hook script to use absolute path after register_claude_hooks
([#571](#571))
([b618d2d](b618d2d))
* **perf:** surface RTK/CLI context-tool savings in perf and the session
card ([#1433](#1433))
([9362747](9362747))
* **proxy:** add --protect-tool-results to prevent lossy compression of
exact-output Bash results
([#1374](#1374))
([51d4bcf](51d4bcf))
* **proxy:** add an Anthropic buffered read-timeout override
([#1331](#1331))
([3be2526](3be2526))
* **proxy:** add versionless Vertex AI routes for Claude Code
compatibility
([#1321](#1321))
([bb3e040](bb3e040))
* **proxy:** bind before eager preload so a hung compressor load can't
block startup
([#1500](#1500))
([d5ac07f](d5ac07f))
* **proxy:** build SSL contexts for custom CA bundles
([#1134](#1134))
([561ba17](561ba17))
* **proxy:** forward request-id headers on the streaming path
([#1100](#1100))
([#1258](#1258))
([3d59df7](3d59df7))
* **proxy:** gate CCR retrieve/compress endpoints to loopback
([#1338](#1338))
([acafb2d](acafb2d))
* **proxy:** honor force_kompress routing profile
([#996](#996))
([b4682d6](b4682d6))
* **proxy:** keep large compression results on the critical path
([#296](#296))
([#1352](#1352))
([90734b6](90734b6))
* **proxy:** offload /v1/compress to the compression executor to stop
blocking the loop
([#1501](#1501))
([27e010e](27e010e))
* **proxy:** preserve Responses memory continuations with store=false
([#1103](#1103))
([cdfeeac](cdfeeac))
* **proxy:** queue mid-turn user messages on non-Bedrock streaming path
([#1377](#1377))
([b09f027](b09f027))
* **proxy:** register interceptor in explicit transforms list when
HEADROOM_INTERCEPT_ENABLED
([#1376](#1376))
([55c700c](55c700c))
* **proxy:** report real input tokens on streaming message_start
([#1132](#1132))
([#1305](#1305))
([70cc96a](70cc96a))
* **proxy:** retry upstream 429 with Retry-After on both forwarders
([#1329](#1329))
([90bee89](90bee89))
* **proxy:** retry upstream 529 overloaded like 429 on both forwarders
([#1495](#1495))
([547b15d](547b15d))
* **proxy:** stop re-compressing headroom_retrieve output and emitting
unredeemable markers
([#1323](#1323))
([43494ff](43494ff))
* **proxy:** strip Codex lite header from OpenAI WebSockets
([#1543](#1543))
([5d3803a](5d3803a))
* **read-lifecycle:** persist STALE Read originals in the CCR store
([#1488](#1488))
([9157173](9157173))
* recover persistent proxy feature checks and reject non-Copilot
exchange URL
([#1465](#1465))
([16c638b](16c638b))
* remove agents.md
([#1540](#1540))
([a7d3360](a7d3360))
* respect COPILOT_PROVIDER_TYPE env var when provider_type is auto
([#549](#549))
([24cf256](24cf256))
* restore token-mode compression on frozen prefixes
([#1489](#1489))
([8e0dadf](8e0dadf))
* **router:** degrade to pure-Python detection on native panic
([#1123](#1123))
([#1260](#1260))
([a00fb67](a00fb67))
* **rtk:** stop hook registration timing out on a forked daemon
([#1314](#1314))
([9758817](9758817))
* **smart-crusher:** honor enable_ccr_marker on the opaque-blob path
([#1130](#1130))
([27d6f8e](27d6f8e))
* **subscription:** only reset 5h contribution on real rollover, not API
jitter
([#1255](#1255))
([8d6c175](8d6c175))
* **subscription:** run transcript token scan off the event loop
([#1263](#1263))
([f03021f](f03021f))
* surface output reduction without a restart, and explain $0.00 savings
on Python 3.14
([#1296](#1296))
([c30ec4c](c30ec4c))
* **tests:** reset whole headroom logger subtree so caplog stays
deterministic
([#1117](#1117))
([fda4670](fda4670))
* **tls:** add HEADROOM_TLS_STRICT=0 toggle for corporate SSL inspection
([#1308](#1308))
([#1341](#1341))
([52068dd](52068dd))
* **tokenizers:** price CJK/Kana/Hangul at ~1 token per char in
EstimatingTokenCounter
([#1093](#1093))
([a35fe86](a35fe86))
* **transforms:** gate tool string output from lossy compression
([#1307](#1307))
([#1387](#1387))
([c6c921a](c6c921a))
* **websocket:** harden responses websocket origin handling
([#1481](#1481))
([c632023](c632023))
* **windows:** pin UTF-8 encoding on text-mode subprocess calls
([#1311](#1311))
([d633e81](d633e81))
* **wrap:** add Copilot unwrap command
([#1251](#1251))
([b4fde0c](b4fde0c))
* **wrap:** isolate proxy stdio from proxy.log on Windows
([#1191](#1191))
([959ab0d](959ab0d))
* **wrap:** keep agent savings opt-in
([#1294](#1294))
([b829ceb](b829ceb))
* **wrap:** show the dashboard URL when the proxy is already running
([#1313](#1313))
([b0146c4](b0146c4))


### Performance Improvements

* **compression:** take large cold-start contexts off the synchronous
kompress path
([#1171](#1171))
([#1298](#1298))
([6c68ff4](6c68ff4))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
JerrettDavis pushed a commit to peterlodri-sec/headroom that referenced this pull request Jul 14, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>0.28.0</summary>

##
[0.28.0](headroomlabs-ai/headroom@v0.27.0...v0.28.0)
(2026-06-29)


### Features

* add --disable-kompress-fallback to restore legacy PASSTHROUGH fallback
([headroomlabs-ai#1185](headroomlabs-ai#1185))
([f309244](headroomlabs-ai@f309244))
* add first-class OpenCode support (wrap, learn, mcp install)
([headroomlabs-ai#559](headroomlabs-ai#559))
([91cd210](headroomlabs-ai@91cd210))
* add HEADROOM_KEEPALIVE_EXPIRY to keep upstream connections warm
([headroomlabs-ai#1124](headroomlabs-ai#1124))
([85786b3](headroomlabs-ai@85786b3))
* **azure-foundry:** derive upstream URL from ANTHROPIC_FOUNDRY_RESOURCE
([headroomlabs-ai#1138](headroomlabs-ai#1138))
([e5031b0](headroomlabs-ai@e5031b0))
* **cache:** attribute prompt-cache misses to TTL lapse vs prefix change
([headroomlabs-ai#1313](headroomlabs-ai#1313))
([headroomlabs-ai#1343](headroomlabs-ai#1343))
([4658721](headroomlabs-ai@4658721))
* **code:** add Perl support to code-aware compressor
([headroomlabs-ai#1125](headroomlabs-ai#1125))
([f39858c](headroomlabs-ai@f39858c))
* headroom wrap opencode / unwrap opencode CLI
([headroomlabs-ai#1105](headroomlabs-ai#1105))
([b4571cc](headroomlabs-ai@b4571cc))
* **learn:** weight loops in Headroom Learn + RTK-loop eval
([headroomlabs-ai#1160](headroomlabs-ai#1160))
([14e8dc4](headroomlabs-ai@14e8dc4))
* **learn:** write per-project learnings to CLAUDE.local.md by default
([headroomlabs-ai#1115](headroomlabs-ai#1115))
([ced75e4](headroomlabs-ai@ced75e4))
* **proxy:** add request timeout config
([headroomlabs-ai#738](headroomlabs-ai#738))
([c0745d4](headroomlabs-ai@c0745d4))
* **proxy:** pilot hardening — inbound auth, security headers, audit
log, air-gap switch
([headroomlabs-ai#1537](headroomlabs-ai#1537))
([546ab55](headroomlabs-ai@546ab55))
* **proxy:** support glob patterns in exclude_tools
([headroomlabs-ai#870](headroomlabs-ai#870))
([headroomlabs-ai#1259](headroomlabs-ai#1259))
([a2159c0](headroomlabs-ai@a2159c0))
* **read-maturation:** activity-based hold-back Read maturation
(Mechanism B)
([headroomlabs-ai#1068](headroomlabs-ai#1068))
([723b80c](headroomlabs-ai@723b80c))
* **savings:** durable savings ledger + headroom savings command
([headroomlabs-ai#1127](headroomlabs-ai#1127))
([978ffa0](headroomlabs-ai@978ffa0))
* **wrap:** add --1m to preserve the 1M context window on wrap claude
([headroomlabs-ai#1158](headroomlabs-ai#1158))
([headroomlabs-ai#1351](headroomlabs-ai#1351))
([b50d9c1](headroomlabs-ai@b50d9c1))
* **wrap:** make tokensave the primary coding-task compressor, Serena
the backup
([headroomlabs-ai#1230](headroomlabs-ai#1230))
([dca9853](headroomlabs-ai@dca9853))


### Bug Fixes

* **agent-evals:** Phase 0 — coding-agent accuracy A/B framework
([headroomlabs-ai#1037](headroomlabs-ai#1037))
([84f9871](headroomlabs-ai@84f9871))
* **agno:** tolerate streaming tool-call SDK objects in parser
([headroomlabs-ai#1312](headroomlabs-ai#1312))
([headroomlabs-ai#1336](headroomlabs-ai#1336))
([5986c22](headroomlabs-ai@5986c22))
* **bedrock:** add boto3 1.41 + CRT for aws login credentials
([headroomlabs-ai#1486](headroomlabs-ai#1486))
([4db3bc9](headroomlabs-ai@4db3bc9))
* bump codebase-memory-mcp to v0.8.1
([headroomlabs-ai#1284](headroomlabs-ai#1284))
([530318b](headroomlabs-ai@530318b))
* **ccr:** make headroom_retrieve a hash-only full-content lookup
([headroomlabs-ai#1532](headroomlabs-ai#1532))
([c2fc4d3](headroomlabs-ai@c2fc4d3))
* **ccr:** propagate --no-ccr-marker flag to all compressors
([headroomlabs-ai#1022](headroomlabs-ai#1022))
([headroomlabs-ai#1197](headroomlabs-ai#1197))
([0c9b42a](headroomlabs-ai@0c9b42a))
* **ccr:** skip Anthropic marker emission when tool injection is
deferred
([headroomlabs-ai#1273](headroomlabs-ai#1273))
([2cae13d](headroomlabs-ai@2cae13d))
* **ci:** extend gitleaks allowlist to cover test fixtures + verified
examples
([headroomlabs-ai#1539](headroomlabs-ai#1539))
([d2565a6](headroomlabs-ai@d2565a6))
* **ci:** guarantee model present in test shards to end cache-miss
flakiness
([headroomlabs-ai#1399](headroomlabs-ai#1399))
([2e29c72](headroomlabs-ai@2e29c72))
* **ci:** normalize Windows CRLF line endings in PR governance script
([headroomlabs-ai#1012](headroomlabs-ai#1012))
([5194388](headroomlabs-ai@5194388))
* **cli:** add explicit UTF-8 encoding to file I/O in wrap commands
([headroomlabs-ai#1126](headroomlabs-ai#1126))
([headroomlabs-ai#1164](headroomlabs-ai#1164))
([a0cb798](headroomlabs-ai@a0cb798))
* **cli:** fall back gracefully when embedding-server sidecar is absent
([headroomlabs-ai#1206](headroomlabs-ai#1206))
([38f1404](headroomlabs-ai@38f1404))
* **cli:** harden all CLI surfaces + fix docs accuracy
([headroomlabs-ai#1491](headroomlabs-ai#1491))
([bd76235](headroomlabs-ai@bd76235))
* **cli:** wire --http2/--no-http2 (HEADROOM_HTTP2) into proxy command
([headroomlabs-ai#1373](headroomlabs-ai#1373))
([e06b616](headroomlabs-ai@e06b616))
* **cli:** wire --rpm/--tpm and HEADROOM_RPM/HEADROOM_TPM to the Click
proxy command
([headroomlabs-ai#1375](headroomlabs-ai#1375))
([8aab8f2](headroomlabs-ai@8aab8f2))
* **code:** slice tree-sitter byte offsets as UTF-8
([headroomlabs-ai#1332](headroomlabs-ai#1332))
([8238402](headroomlabs-ai@8238402))
* **code:** validate Python compressed syntax
([headroomlabs-ai#1302](headroomlabs-ai#1302))
([cbd361d](headroomlabs-ai@cbd361d))
* **code:** verify a real parse in tree-sitter availability check
([headroomlabs-ai#1231](headroomlabs-ai#1231))
([headroomlabs-ai#1299](headroomlabs-ai#1299))
([5e0bb69](headroomlabs-ai@5e0bb69))
* **codex:** retag threads on init so Codex Desktop history stays
visible ([headroomlabs-ai#961](headroomlabs-ai#961))
([headroomlabs-ai#1349](headroomlabs-ai#1349))
([e6bbc40](headroomlabs-ai@e6bbc40))
* **codex:** stop pinning Codex memory MCP to one project db
([headroomlabs-ai#1269](headroomlabs-ai#1269))
([ad7993b](headroomlabs-ai@ad7993b))
* **dashboard:** include RTK stats in the historical tab
([headroomlabs-ai#1324](headroomlabs-ai#1324))
([35939c3](headroomlabs-ai@35939c3))
* **deps:** remediate dependency CVEs and publish SBOM
([headroomlabs-ai#1509](headroomlabs-ai#1509))
([5771a80](headroomlabs-ai@5771a80))
* **docker:** persist session history across container revisions
([headroomlabs-ai#1118](headroomlabs-ai#1118))
([5912d65](headroomlabs-ai@5912d65))
* **gemini:** offload compression to the executor
([headroomlabs-ai#1382](headroomlabs-ai#1382))
([615848e](headroomlabs-ai@615848e))
* **gemini:** resolve Google model capabilities through ModelRegistry
([headroomlabs-ai#1276](headroomlabs-ai#1276))
([17ecad9](headroomlabs-ai@17ecad9))
* **install:** guard install_agent_ensure against duplicate runtime
spawns
([headroomlabs-ai#1301](headroomlabs-ai#1301))
([8da0b4e](headroomlabs-ai@8da0b4e))
* **install:** repair macOS launchd restart/start lifecycle
([headroomlabs-ai#1290](headroomlabs-ai#1290))
([da1a397](headroomlabs-ai@da1a397))
* **install:** stop duplicating ENTRYPOINT in persistent-docker runtime
command ([headroomlabs-ai#833](headroomlabs-ai#833))
([headroomlabs-ai#1348](headroomlabs-ai#1348))
([feedead](headroomlabs-ai@feedead))
* **io:** use UTF-8 with locale fallback and preserve line endings on
config/text I/O
([headroomlabs-ai#1498](headroomlabs-ai#1498))
([1baa04e](headroomlabs-ai@1baa04e))
* **kompress:** hard override keeps must-keep tokens regardless of model
score ([headroomlabs-ai#1400](headroomlabs-ai#1400))
([42612c8](headroomlabs-ai@42612c8))
* **langchain:** disable streaming on wrapped model during ainvoke()
([headroomlabs-ai#1287](headroomlabs-ai#1287))
([3590046](headroomlabs-ai@3590046))
* **mcp:** register managed installs with a resolvable headroom command
([headroomlabs-ai#1386](headroomlabs-ai#1386))
([22def93](headroomlabs-ai@22def93))
* **mcp:** report correct savings_percent in headroom_compress
([headroomlabs-ai#1106](headroomlabs-ai#1106))
([f216e43](headroomlabs-ai@f216e43))
* **opencode:** write local MCP config
([headroomlabs-ai#1381](headroomlabs-ai#1381))
([6c83790](headroomlabs-ai@6c83790))
* **packaging:** move hnswlib to optional [vector] extra so [all] needs
no C++ toolchain
([headroomlabs-ai#1499](headroomlabs-ai#1499))
([80fa086](headroomlabs-ai@80fa086))
* patch rtk hook script to use absolute path after register_claude_hooks
([headroomlabs-ai#571](headroomlabs-ai#571))
([b618d2d](headroomlabs-ai@b618d2d))
* **perf:** surface RTK/CLI context-tool savings in perf and the session
card ([headroomlabs-ai#1433](headroomlabs-ai#1433))
([9362747](headroomlabs-ai@9362747))
* **proxy:** add --protect-tool-results to prevent lossy compression of
exact-output Bash results
([headroomlabs-ai#1374](headroomlabs-ai#1374))
([51d4bcf](headroomlabs-ai@51d4bcf))
* **proxy:** add an Anthropic buffered read-timeout override
([headroomlabs-ai#1331](headroomlabs-ai#1331))
([3be2526](headroomlabs-ai@3be2526))
* **proxy:** add versionless Vertex AI routes for Claude Code
compatibility
([headroomlabs-ai#1321](headroomlabs-ai#1321))
([bb3e040](headroomlabs-ai@bb3e040))
* **proxy:** bind before eager preload so a hung compressor load can't
block startup
([headroomlabs-ai#1500](headroomlabs-ai#1500))
([d5ac07f](headroomlabs-ai@d5ac07f))
* **proxy:** build SSL contexts for custom CA bundles
([headroomlabs-ai#1134](headroomlabs-ai#1134))
([561ba17](headroomlabs-ai@561ba17))
* **proxy:** forward request-id headers on the streaming path
([headroomlabs-ai#1100](headroomlabs-ai#1100))
([headroomlabs-ai#1258](headroomlabs-ai#1258))
([3d59df7](headroomlabs-ai@3d59df7))
* **proxy:** gate CCR retrieve/compress endpoints to loopback
([headroomlabs-ai#1338](headroomlabs-ai#1338))
([acafb2d](headroomlabs-ai@acafb2d))
* **proxy:** honor force_kompress routing profile
([headroomlabs-ai#996](headroomlabs-ai#996))
([b4682d6](headroomlabs-ai@b4682d6))
* **proxy:** keep large compression results on the critical path
([headroomlabs-ai#296](headroomlabs-ai#296))
([headroomlabs-ai#1352](headroomlabs-ai#1352))
([90734b6](headroomlabs-ai@90734b6))
* **proxy:** offload /v1/compress to the compression executor to stop
blocking the loop
([headroomlabs-ai#1501](headroomlabs-ai#1501))
([27e010e](headroomlabs-ai@27e010e))
* **proxy:** preserve Responses memory continuations with store=false
([headroomlabs-ai#1103](headroomlabs-ai#1103))
([cdfeeac](headroomlabs-ai@cdfeeac))
* **proxy:** queue mid-turn user messages on non-Bedrock streaming path
([headroomlabs-ai#1377](headroomlabs-ai#1377))
([b09f027](headroomlabs-ai@b09f027))
* **proxy:** register interceptor in explicit transforms list when
HEADROOM_INTERCEPT_ENABLED
([headroomlabs-ai#1376](headroomlabs-ai#1376))
([55c700c](headroomlabs-ai@55c700c))
* **proxy:** report real input tokens on streaming message_start
([headroomlabs-ai#1132](headroomlabs-ai#1132))
([headroomlabs-ai#1305](headroomlabs-ai#1305))
([70cc96a](headroomlabs-ai@70cc96a))
* **proxy:** retry upstream 429 with Retry-After on both forwarders
([headroomlabs-ai#1329](headroomlabs-ai#1329))
([90bee89](headroomlabs-ai@90bee89))
* **proxy:** retry upstream 529 overloaded like 429 on both forwarders
([headroomlabs-ai#1495](headroomlabs-ai#1495))
([547b15d](headroomlabs-ai@547b15d))
* **proxy:** stop re-compressing headroom_retrieve output and emitting
unredeemable markers
([headroomlabs-ai#1323](headroomlabs-ai#1323))
([43494ff](headroomlabs-ai@43494ff))
* **proxy:** strip Codex lite header from OpenAI WebSockets
([headroomlabs-ai#1543](headroomlabs-ai#1543))
([5d3803a](headroomlabs-ai@5d3803a))
* **read-lifecycle:** persist STALE Read originals in the CCR store
([headroomlabs-ai#1488](headroomlabs-ai#1488))
([9157173](headroomlabs-ai@9157173))
* recover persistent proxy feature checks and reject non-Copilot
exchange URL
([headroomlabs-ai#1465](headroomlabs-ai#1465))
([16c638b](headroomlabs-ai@16c638b))
* remove agents.md
([headroomlabs-ai#1540](headroomlabs-ai#1540))
([a7d3360](headroomlabs-ai@a7d3360))
* respect COPILOT_PROVIDER_TYPE env var when provider_type is auto
([headroomlabs-ai#549](headroomlabs-ai#549))
([24cf256](headroomlabs-ai@24cf256))
* restore token-mode compression on frozen prefixes
([headroomlabs-ai#1489](headroomlabs-ai#1489))
([8e0dadf](headroomlabs-ai@8e0dadf))
* **router:** degrade to pure-Python detection on native panic
([headroomlabs-ai#1123](headroomlabs-ai#1123))
([headroomlabs-ai#1260](headroomlabs-ai#1260))
([a00fb67](headroomlabs-ai@a00fb67))
* **rtk:** stop hook registration timing out on a forked daemon
([headroomlabs-ai#1314](headroomlabs-ai#1314))
([9758817](headroomlabs-ai@9758817))
* **smart-crusher:** honor enable_ccr_marker on the opaque-blob path
([headroomlabs-ai#1130](headroomlabs-ai#1130))
([27d6f8e](headroomlabs-ai@27d6f8e))
* **subscription:** only reset 5h contribution on real rollover, not API
jitter
([headroomlabs-ai#1255](headroomlabs-ai#1255))
([8d6c175](headroomlabs-ai@8d6c175))
* **subscription:** run transcript token scan off the event loop
([headroomlabs-ai#1263](headroomlabs-ai#1263))
([f03021f](headroomlabs-ai@f03021f))
* surface output reduction without a restart, and explain $0.00 savings
on Python 3.14
([headroomlabs-ai#1296](headroomlabs-ai#1296))
([c30ec4c](headroomlabs-ai@c30ec4c))
* **tests:** reset whole headroom logger subtree so caplog stays
deterministic
([headroomlabs-ai#1117](headroomlabs-ai#1117))
([fda4670](headroomlabs-ai@fda4670))
* **tls:** add HEADROOM_TLS_STRICT=0 toggle for corporate SSL inspection
([headroomlabs-ai#1308](headroomlabs-ai#1308))
([headroomlabs-ai#1341](headroomlabs-ai#1341))
([52068dd](headroomlabs-ai@52068dd))
* **tokenizers:** price CJK/Kana/Hangul at ~1 token per char in
EstimatingTokenCounter
([headroomlabs-ai#1093](headroomlabs-ai#1093))
([a35fe86](headroomlabs-ai@a35fe86))
* **transforms:** gate tool string output from lossy compression
([headroomlabs-ai#1307](headroomlabs-ai#1307))
([headroomlabs-ai#1387](headroomlabs-ai#1387))
([c6c921a](headroomlabs-ai@c6c921a))
* **websocket:** harden responses websocket origin handling
([headroomlabs-ai#1481](headroomlabs-ai#1481))
([c632023](headroomlabs-ai@c632023))
* **windows:** pin UTF-8 encoding on text-mode subprocess calls
([headroomlabs-ai#1311](headroomlabs-ai#1311))
([d633e81](headroomlabs-ai@d633e81))
* **wrap:** add Copilot unwrap command
([headroomlabs-ai#1251](headroomlabs-ai#1251))
([b4fde0c](headroomlabs-ai@b4fde0c))
* **wrap:** isolate proxy stdio from proxy.log on Windows
([headroomlabs-ai#1191](headroomlabs-ai#1191))
([959ab0d](headroomlabs-ai@959ab0d))
* **wrap:** keep agent savings opt-in
([headroomlabs-ai#1294](headroomlabs-ai#1294))
([b829ceb](headroomlabs-ai@b829ceb))
* **wrap:** show the dashboard URL when the proxy is already running
([headroomlabs-ai#1313](headroomlabs-ai#1313))
([b0146c4](headroomlabs-ai@b0146c4))


### Performance Improvements

* **compression:** take large cold-start contexts off the synchronous
kompress path
([headroomlabs-ai#1171](headroomlabs-ai#1171))
([headroomlabs-ai#1298](headroomlabs-ai#1298))
([6c68ff4](headroomlabs-ai@6c68ff4))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status: needs author action Pull request body or readiness checklist still needs author updates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] install agent ensure can spawn duplicate runtimes (no start-lock / no running-instance check), causing port-bind contention and a restart storm

2 participants