fix(proxy): retry upstream 529 overloaded like 429 on both forwarders#1495
Merged
chopratejas merged 2 commits intoJun 28, 2026
Merged
Conversation
Contributor
PR governanceThis PR follows the template and is marked ready for human review. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR makes upstream HTTP 529 overloaded_error retry consistently across the proxy’s non-streaming (_retry_request) and streaming (_stream_response) forwarders, aligning it with the existing 429 retry behavior and ensuring Retry-After is honored where applicable.
Changes:
- Introduces
RETRYABLE_OVERLOAD_STATUSES = {429, 529}as a shared single source of truth. - Updates both forwarders to treat
529like429(retry withRetry-After, return the status verbatim on exhaustion). - Adds unit tests covering non-streaming and streaming
529retry behavior (plus changelog entry).
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_proxy_retry_429.py | Adds coverage to ensure 529 behaves like 429 in both retry paths, including Retry-After honoring and exhaustion semantics. |
| headroom/proxy/server.py | Unifies 429/529 handling into a shared overload-retry branch and returns overload status verbatim on exhaustion. |
| headroom/proxy/helpers.py | Adds RETRYABLE_OVERLOAD_STATUSES constant used by both forwarders. |
| headroom/proxy/handlers/streaming.py | Retries streaming responses for 529 using the same logic as 429, and logs the actual status code. |
| CHANGELOG.md | Adds a changelog entry for the new 529 retry behavior. |
Comments suppressed due to low confidence (1)
CHANGELOG.md:52
CHANGELOG.mdstill contains unresolved git merge-conflict markers (<<<<<<<,=======,>>>>>>>). These markers should not ship and will cause ongoing confusion when editing/releasing. Please resolve this section by removing the markers and keeping the intended entries.
* **proxy:** retry upstream `529 overloaded_error` like a 429 on both the streaming and non-streaming forwarders, honoring `Retry-After`. The streaming path previously surfaced a 529 straight to the client with no retry (interactive sessions saw "Overloaded" immediately), and `_retry_request` retried it only via the generic 5xx path — raising on exhaustion instead of returning the 529 verbatim, and ignoring `Retry-After`. A shared `RETRYABLE_OVERLOAD_STATUSES = {429, 529}` keeps the two forwarders in agreement (extends [#1221](https://github.com/headroomlabs-ai/headroom/issues/1221)).
<<<<<<< fix/gemini-offload
* **gemini:** run compression off the asyncio event loop. The Gemini handlers (`generateContent`, Cloud Code stream, `countTokens`) ran the CPU-bound compression pipeline (Magika detection plus ML compression) synchronously on the loop, stalling every concurrent request for the duration of each Gemini request's compression. They now offload it via the shared compression executor, matching the existing OpenAI and Anthropic paths.
=======
* **proxy:** queue mid-turn user messages on non-Bedrock streaming path instead of silently dropping them — closes [#902](https://github.com/headroomlabs-ai/headroom/issues/902).
529 (Anthropic overloaded_error) was retried inconsistently. The streaming
forwarder special-cased only 429 and forwarded a 529 straight to the client
with zero retries, so interactive (streaming) sessions saw "Overloaded"
immediately. _retry_request retried 529 only via the generic 5xx path —
raising on exhaustion instead of returning the 529 verbatim, and ignoring
Retry-After.
Introduce a shared RETRYABLE_OVERLOAD_STATUSES = {429, 529} and route both
statuses through the same Retry-After-honoring retry branch in the streaming
and non-streaming forwarders. Extends headroomlabs-ai#1221, which added 429 retry, to 529.
Adds 4 tests to tests/test_proxy_retry_429.py (529 retry-then-succeed,
verbatim-on-exhaustion, Retry-After honored, streaming retry); 3 of them fail
before this change.
4122735 to
f28f678
Compare
Merged
chopratejas
pushed a commit
that referenced
this pull request
Jun 29, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>0.28.0</summary> ## [0.28.0](v0.27.0...v0.28.0) (2026-06-29) ### Features * add --disable-kompress-fallback to restore legacy PASSTHROUGH fallback ([#1185](#1185)) ([f309244](f309244)) * add first-class OpenCode support (wrap, learn, mcp install) ([#559](#559)) ([91cd210](91cd210)) * add HEADROOM_KEEPALIVE_EXPIRY to keep upstream connections warm ([#1124](#1124)) ([85786b3](85786b3)) * **azure-foundry:** derive upstream URL from ANTHROPIC_FOUNDRY_RESOURCE ([#1138](#1138)) ([e5031b0](e5031b0)) * **cache:** attribute prompt-cache misses to TTL lapse vs prefix change ([#1313](#1313)) ([#1343](#1343)) ([4658721](4658721)) * **code:** add Perl support to code-aware compressor ([#1125](#1125)) ([f39858c](f39858c)) * headroom wrap opencode / unwrap opencode CLI ([#1105](#1105)) ([b4571cc](b4571cc)) * **learn:** weight loops in Headroom Learn + RTK-loop eval ([#1160](#1160)) ([14e8dc4](14e8dc4)) * **learn:** write per-project learnings to CLAUDE.local.md by default ([#1115](#1115)) ([ced75e4](ced75e4)) * **proxy:** add request timeout config ([#738](#738)) ([c0745d4](c0745d4)) * **proxy:** pilot hardening — inbound auth, security headers, audit log, air-gap switch ([#1537](#1537)) ([546ab55](546ab55)) * **proxy:** support glob patterns in exclude_tools ([#870](#870)) ([#1259](#1259)) ([a2159c0](a2159c0)) * **read-maturation:** activity-based hold-back Read maturation (Mechanism B) ([#1068](#1068)) ([723b80c](723b80c)) * **savings:** durable savings ledger + headroom savings command ([#1127](#1127)) ([978ffa0](978ffa0)) * **wrap:** add --1m to preserve the 1M context window on wrap claude ([#1158](#1158)) ([#1351](#1351)) ([b50d9c1](b50d9c1)) * **wrap:** make tokensave the primary coding-task compressor, Serena the backup ([#1230](#1230)) ([dca9853](dca9853)) ### Bug Fixes * **agent-evals:** Phase 0 — coding-agent accuracy A/B framework ([#1037](#1037)) ([84f9871](84f9871)) * **agno:** tolerate streaming tool-call SDK objects in parser ([#1312](#1312)) ([#1336](#1336)) ([5986c22](5986c22)) * **bedrock:** add boto3 1.41 + CRT for aws login credentials ([#1486](#1486)) ([4db3bc9](4db3bc9)) * bump codebase-memory-mcp to v0.8.1 ([#1284](#1284)) ([530318b](530318b)) * **ccr:** make headroom_retrieve a hash-only full-content lookup ([#1532](#1532)) ([c2fc4d3](c2fc4d3)) * **ccr:** propagate --no-ccr-marker flag to all compressors ([#1022](#1022)) ([#1197](#1197)) ([0c9b42a](0c9b42a)) * **ccr:** skip Anthropic marker emission when tool injection is deferred ([#1273](#1273)) ([2cae13d](2cae13d)) * **ci:** extend gitleaks allowlist to cover test fixtures + verified examples ([#1539](#1539)) ([d2565a6](d2565a6)) * **ci:** guarantee model present in test shards to end cache-miss flakiness ([#1399](#1399)) ([2e29c72](2e29c72)) * **ci:** normalize Windows CRLF line endings in PR governance script ([#1012](#1012)) ([5194388](5194388)) * **cli:** add explicit UTF-8 encoding to file I/O in wrap commands ([#1126](#1126)) ([#1164](#1164)) ([a0cb798](a0cb798)) * **cli:** fall back gracefully when embedding-server sidecar is absent ([#1206](#1206)) ([38f1404](38f1404)) * **cli:** harden all CLI surfaces + fix docs accuracy ([#1491](#1491)) ([bd76235](bd76235)) * **cli:** wire --http2/--no-http2 (HEADROOM_HTTP2) into proxy command ([#1373](#1373)) ([e06b616](e06b616)) * **cli:** wire --rpm/--tpm and HEADROOM_RPM/HEADROOM_TPM to the Click proxy command ([#1375](#1375)) ([8aab8f2](8aab8f2)) * **code:** slice tree-sitter byte offsets as UTF-8 ([#1332](#1332)) ([8238402](8238402)) * **code:** validate Python compressed syntax ([#1302](#1302)) ([cbd361d](cbd361d)) * **code:** verify a real parse in tree-sitter availability check ([#1231](#1231)) ([#1299](#1299)) ([5e0bb69](5e0bb69)) * **codex:** retag threads on init so Codex Desktop history stays visible ([#961](#961)) ([#1349](#1349)) ([e6bbc40](e6bbc40)) * **codex:** stop pinning Codex memory MCP to one project db ([#1269](#1269)) ([ad7993b](ad7993b)) * **dashboard:** include RTK stats in the historical tab ([#1324](#1324)) ([35939c3](35939c3)) * **deps:** remediate dependency CVEs and publish SBOM ([#1509](#1509)) ([5771a80](5771a80)) * **docker:** persist session history across container revisions ([#1118](#1118)) ([5912d65](5912d65)) * **gemini:** offload compression to the executor ([#1382](#1382)) ([615848e](615848e)) * **gemini:** resolve Google model capabilities through ModelRegistry ([#1276](#1276)) ([17ecad9](17ecad9)) * **install:** guard install_agent_ensure against duplicate runtime spawns ([#1301](#1301)) ([8da0b4e](8da0b4e)) * **install:** repair macOS launchd restart/start lifecycle ([#1290](#1290)) ([da1a397](da1a397)) * **install:** stop duplicating ENTRYPOINT in persistent-docker runtime command ([#833](#833)) ([#1348](#1348)) ([feedead](feedead)) * **io:** use UTF-8 with locale fallback and preserve line endings on config/text I/O ([#1498](#1498)) ([1baa04e](1baa04e)) * **kompress:** hard override keeps must-keep tokens regardless of model score ([#1400](#1400)) ([42612c8](42612c8)) * **langchain:** disable streaming on wrapped model during ainvoke() ([#1287](#1287)) ([3590046](3590046)) * **mcp:** register managed installs with a resolvable headroom command ([#1386](#1386)) ([22def93](22def93)) * **mcp:** report correct savings_percent in headroom_compress ([#1106](#1106)) ([f216e43](f216e43)) * **opencode:** write local MCP config ([#1381](#1381)) ([6c83790](6c83790)) * **packaging:** move hnswlib to optional [vector] extra so [all] needs no C++ toolchain ([#1499](#1499)) ([80fa086](80fa086)) * patch rtk hook script to use absolute path after register_claude_hooks ([#571](#571)) ([b618d2d](b618d2d)) * **perf:** surface RTK/CLI context-tool savings in perf and the session card ([#1433](#1433)) ([9362747](9362747)) * **proxy:** add --protect-tool-results to prevent lossy compression of exact-output Bash results ([#1374](#1374)) ([51d4bcf](51d4bcf)) * **proxy:** add an Anthropic buffered read-timeout override ([#1331](#1331)) ([3be2526](3be2526)) * **proxy:** add versionless Vertex AI routes for Claude Code compatibility ([#1321](#1321)) ([bb3e040](bb3e040)) * **proxy:** bind before eager preload so a hung compressor load can't block startup ([#1500](#1500)) ([d5ac07f](d5ac07f)) * **proxy:** build SSL contexts for custom CA bundles ([#1134](#1134)) ([561ba17](561ba17)) * **proxy:** forward request-id headers on the streaming path ([#1100](#1100)) ([#1258](#1258)) ([3d59df7](3d59df7)) * **proxy:** gate CCR retrieve/compress endpoints to loopback ([#1338](#1338)) ([acafb2d](acafb2d)) * **proxy:** honor force_kompress routing profile ([#996](#996)) ([b4682d6](b4682d6)) * **proxy:** keep large compression results on the critical path ([#296](#296)) ([#1352](#1352)) ([90734b6](90734b6)) * **proxy:** offload /v1/compress to the compression executor to stop blocking the loop ([#1501](#1501)) ([27e010e](27e010e)) * **proxy:** preserve Responses memory continuations with store=false ([#1103](#1103)) ([cdfeeac](cdfeeac)) * **proxy:** queue mid-turn user messages on non-Bedrock streaming path ([#1377](#1377)) ([b09f027](b09f027)) * **proxy:** register interceptor in explicit transforms list when HEADROOM_INTERCEPT_ENABLED ([#1376](#1376)) ([55c700c](55c700c)) * **proxy:** report real input tokens on streaming message_start ([#1132](#1132)) ([#1305](#1305)) ([70cc96a](70cc96a)) * **proxy:** retry upstream 429 with Retry-After on both forwarders ([#1329](#1329)) ([90bee89](90bee89)) * **proxy:** retry upstream 529 overloaded like 429 on both forwarders ([#1495](#1495)) ([547b15d](547b15d)) * **proxy:** stop re-compressing headroom_retrieve output and emitting unredeemable markers ([#1323](#1323)) ([43494ff](43494ff)) * **proxy:** strip Codex lite header from OpenAI WebSockets ([#1543](#1543)) ([5d3803a](5d3803a)) * **read-lifecycle:** persist STALE Read originals in the CCR store ([#1488](#1488)) ([9157173](9157173)) * recover persistent proxy feature checks and reject non-Copilot exchange URL ([#1465](#1465)) ([16c638b](16c638b)) * remove agents.md ([#1540](#1540)) ([a7d3360](a7d3360)) * respect COPILOT_PROVIDER_TYPE env var when provider_type is auto ([#549](#549)) ([24cf256](24cf256)) * restore token-mode compression on frozen prefixes ([#1489](#1489)) ([8e0dadf](8e0dadf)) * **router:** degrade to pure-Python detection on native panic ([#1123](#1123)) ([#1260](#1260)) ([a00fb67](a00fb67)) * **rtk:** stop hook registration timing out on a forked daemon ([#1314](#1314)) ([9758817](9758817)) * **smart-crusher:** honor enable_ccr_marker on the opaque-blob path ([#1130](#1130)) ([27d6f8e](27d6f8e)) * **subscription:** only reset 5h contribution on real rollover, not API jitter ([#1255](#1255)) ([8d6c175](8d6c175)) * **subscription:** run transcript token scan off the event loop ([#1263](#1263)) ([f03021f](f03021f)) * surface output reduction without a restart, and explain $0.00 savings on Python 3.14 ([#1296](#1296)) ([c30ec4c](c30ec4c)) * **tests:** reset whole headroom logger subtree so caplog stays deterministic ([#1117](#1117)) ([fda4670](fda4670)) * **tls:** add HEADROOM_TLS_STRICT=0 toggle for corporate SSL inspection ([#1308](#1308)) ([#1341](#1341)) ([52068dd](52068dd)) * **tokenizers:** price CJK/Kana/Hangul at ~1 token per char in EstimatingTokenCounter ([#1093](#1093)) ([a35fe86](a35fe86)) * **transforms:** gate tool string output from lossy compression ([#1307](#1307)) ([#1387](#1387)) ([c6c921a](c6c921a)) * **websocket:** harden responses websocket origin handling ([#1481](#1481)) ([c632023](c632023)) * **windows:** pin UTF-8 encoding on text-mode subprocess calls ([#1311](#1311)) ([d633e81](d633e81)) * **wrap:** add Copilot unwrap command ([#1251](#1251)) ([b4fde0c](b4fde0c)) * **wrap:** isolate proxy stdio from proxy.log on Windows ([#1191](#1191)) ([959ab0d](959ab0d)) * **wrap:** keep agent savings opt-in ([#1294](#1294)) ([b829ceb](b829ceb)) * **wrap:** show the dashboard URL when the proxy is already running ([#1313](#1313)) ([b0146c4](b0146c4)) ### Performance Improvements * **compression:** take large cold-start contexts off the synchronous kompress path ([#1171](#1171)) ([#1298](#1298)) ([6c68ff4](6c68ff4)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
rocker-zhang
added a commit
to rocker-zhang/headroom
that referenced
this pull request
Jun 30, 2026
When the upstream returns a retryable server error (500/502/503/504), _retry_request retried up to the cap then raised, which the caller collapsed into a generic 502. That hides the real status and makes clients give up instead of applying their own backoff. On exhaustion, return the last upstream response (preserving its status and body) when one is available; connection/timeout errors still raise. The 429/529 overload statuses are handled separately by the RETRYABLE_OVERLOAD_STATUSES branch (returned verbatim by headroomlabs-ai#1495) and never reach this exhaustion handler, so this complements that path rather than overlapping it. Add unit tests covering 500/503 preservation, 4xx no-retry, connect-error raise, and success passthrough.
rocker-zhang
added a commit
to rocker-zhang/headroom
that referenced
this pull request
Jun 30, 2026
When the upstream returns a retryable server error (500/502/503/504), _retry_request retried up to the cap then raised, which the caller collapsed into a generic 502. That hides the real status and makes clients give up instead of applying their own backoff. On exhaustion, return the last upstream response (preserving its status and body) when one is available; connection/timeout errors still raise. The 429/529 overload statuses are handled separately by the RETRYABLE_OVERLOAD_STATUSES branch (returned verbatim by headroomlabs-ai#1495) and never reach this exhaustion handler, so this complements that path rather than overlapping it. Add unit tests covering 500/503 preservation, 4xx no-retry, connect-error raise, and success passthrough.
JerrettDavis
pushed a commit
to peterlodri-sec/headroom
that referenced
this pull request
Jul 14, 2026
…headroomlabs-ai#1495) ## Description Upstream **HTTP 529** (`overloaded_error`) is not retried consistently, so it leaks to clients even though the sibling 429 path was fixed in headroomlabs-ai#1221. - **Streaming forwarder** (`_stream_response`) special-cased only `status_code == 429`. A `529` falls through to `break` and is forwarded to the client with **zero retries** — interactive (streaming) Claude Code sessions see "Overloaded" immediately on a transient Anthropic overload. - **Non-streaming forwarder** (`_retry_request`) retried `529` only via the generic `>= 500` path: it **ignores `Retry-After`** and **raises** an `HTTPStatusError` on exhaustion instead of returning the clean `529` verbatim (inconsistent with how 429 is handled right above it). `529` is documented by Anthropic as the transient "overloaded" status — semantically identical to 429 for retry purposes ("try again shortly"). This PR routes both through one shared, `Retry-After`-honoring branch. Related: headroomlabs-ai#1221 (added the 429 retry this extends). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `RETRYABLE_OVERLOAD_STATUSES = frozenset({429, 529})` to `proxy/helpers.py` as the single source of truth shared by both forwarders. - `streaming.py`: retry when `status_code in RETRYABLE_OVERLOAD_STATUSES` (was `== 429`); log line now interpolates the actual status. - `server.py` `_retry_request`: handle `429`/`529` in one `Retry-After`-honoring branch that returns the status verbatim once `retry_max_attempts` is exhausted (529 no longer goes through the 5xx raise path). Other 4xx/5xx behavior is unchanged. - No new dependencies; no config/API surface changes. Retry volume stays bounded by the existing `retry_max_attempts` / `retry_*_delay_ms` config. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output Reproduced the CI `lint` + `commitlint` jobs exactly (pinned `ruff==0.15.17`, `mypy==1.20.2`, `@commitlint/config-conventional`), plus the affected proxy test subset: ```text # New tests in tests/test_proxy_retry_429.py — 3 of 4 fail on main, all pass here # BEFORE (source reverted, new tests kept): FAILED ::test_retry_request_returns_529_verbatim_on_exhaustion - httpx.HTTPStatusError: Server error: 529 (raised, not returned verbatim) FAILED ::test_retry_request_honors_retry_after_on_529 - slept ~0.001s (jitter), ignored Retry-After: 2 FAILED ::test_stream_response_retries_529 - assert 1 == 2 (streaming 529 forwarded raw, no retry) 3 failed, 7 passed # AFTER (this branch): 10 passed in 2.53s # Adjacent proxy suites (regression check) — retry + streaming resilience + ratelimit headers + handler helpers + request logger: 79 passed in 6.61s $ ruff check . -> All checks passed! $ ruff format --check . -> 1005 files already formatted $ mypy headroom --ignore-missing-imports Success: no issues found in 400 source files $ commitlint --from <base> --to HEAD ✔ found 0 problems, 0 warnings ``` ## Real Behavior Proof - Environment: Linux, Python 3.14.0; the proxy running **from this branch** (`headroom proxy --mode token --backend anthropic --no-optimize ...`) in front of a fake Anthropic upstream that returns a real HTTP 529 (`{"error":{"type":"overloaded_error"}}`, `Retry-After: 0`) on request headroomlabs-ai#1 then a 200 SSE stream on request headroomlabs-ai#2. Real proxy process over real sockets (a synthetic upstream is used because real Anthropic 529s cannot be induced on demand). - Exact command / steps: started the fake upstream on `:9911` and the branch proxy on `:9912` with `--anthropic-api-url http://127.0.0.1:9911`, then sent a streaming request: `curl -sN -X POST http://127.0.0.1:9912/v1/messages -H 'x-api-key: …' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d '{"model":"claude-3-5-sonnet-20241022","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}'` (full scripts in the code block below). - Observed result: the client received `HTTP/1.1 200 OK` and the complete SSE stream (`message_start … "hello" … message_stop`), and the fake upstream logged **two** calls — `call headroomlabs-ai#1` returned 529, `call headroomlabs-ai#2` returned 200 — i.e. the proxy transparently retried the 529 and the overload never reached the client. On `main` the streaming path forwards the 529 on call headroomlabs-ai#1 with no retry, exactly what `test_stream_response_retries_529` pins at `calls == 1`. - Not tested: a real (non-synthetic) Anthropic 529 (cannot induce on demand); the full sharded `pytest tests scripts/tests` job (needs CI model/torch infra) — ran the proxy suite subset above instead; the Rust jobs and non-Anthropic backends (unchanged by this PR). ```bash # fake_upstream.py: 529 (Retry-After: 0) on call headroomlabs-ai#1, then 200 SSE; logs each call python fake_upstream.py & # :9911 headroom proxy --host 127.0.0.1 --port 9912 \ --anthropic-api-url http://127.0.0.1:9911 \ --mode token --backend anthropic \ --no-optimize --no-cache --no-rate-limit & # :9912 (this branch) curl -sN -D - -X POST http://127.0.0.1:9912/v1/messages \ -H 'x-api-key: sk-ant-test' -H 'anthropic-version: 2023-06-01' \ -H 'content-type: application/json' \ -d '{"model":"claude-3-5-sonnet-20241022","max_tokens":16,"stream":true, "messages":[{"role":"user","content":"hi"}]}' # -> HTTP/1.1 200 OK + full SSE; upstream log: "call headroomlabs-ai#1" (529) then "call headroomlabs-ai#2" (200) ``` ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A — no doc/config surface change) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Replicated the CI `lint` job exactly (fresh venv, pinned `ruff==0.15.17` + `mypy==1.20.2`, `ruff check .` / `ruff format --check .` / `mypy headroom --ignore-missing-imports`) and `commitlint` (`@commitlint/config-conventional`) — all clean. The full `test` shards (model/torch) and Rust jobs were not run locally (no GPU/model cache / Rust toolchain in this environment); they are unaffected by this Python-only change. - `CHANGELOG.md`'s `## Unreleased` section currently contains unresolved merge-conflict markers on `main` (`<<<<<<< … >>>>>>>`) unrelated to this PR; I added my entry to the clean `### Bug Fixes` list above that region without touching the conflicts.
JerrettDavis
pushed a commit
to peterlodri-sec/headroom
that referenced
this pull request
Jul 14, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>0.28.0</summary> ## [0.28.0](headroomlabs-ai/headroom@v0.27.0...v0.28.0) (2026-06-29) ### Features * add --disable-kompress-fallback to restore legacy PASSTHROUGH fallback ([headroomlabs-ai#1185](headroomlabs-ai#1185)) ([f309244](headroomlabs-ai@f309244)) * add first-class OpenCode support (wrap, learn, mcp install) ([headroomlabs-ai#559](headroomlabs-ai#559)) ([91cd210](headroomlabs-ai@91cd210)) * add HEADROOM_KEEPALIVE_EXPIRY to keep upstream connections warm ([headroomlabs-ai#1124](headroomlabs-ai#1124)) ([85786b3](headroomlabs-ai@85786b3)) * **azure-foundry:** derive upstream URL from ANTHROPIC_FOUNDRY_RESOURCE ([headroomlabs-ai#1138](headroomlabs-ai#1138)) ([e5031b0](headroomlabs-ai@e5031b0)) * **cache:** attribute prompt-cache misses to TTL lapse vs prefix change ([headroomlabs-ai#1313](headroomlabs-ai#1313)) ([headroomlabs-ai#1343](headroomlabs-ai#1343)) ([4658721](headroomlabs-ai@4658721)) * **code:** add Perl support to code-aware compressor ([headroomlabs-ai#1125](headroomlabs-ai#1125)) ([f39858c](headroomlabs-ai@f39858c)) * headroom wrap opencode / unwrap opencode CLI ([headroomlabs-ai#1105](headroomlabs-ai#1105)) ([b4571cc](headroomlabs-ai@b4571cc)) * **learn:** weight loops in Headroom Learn + RTK-loop eval ([headroomlabs-ai#1160](headroomlabs-ai#1160)) ([14e8dc4](headroomlabs-ai@14e8dc4)) * **learn:** write per-project learnings to CLAUDE.local.md by default ([headroomlabs-ai#1115](headroomlabs-ai#1115)) ([ced75e4](headroomlabs-ai@ced75e4)) * **proxy:** add request timeout config ([headroomlabs-ai#738](headroomlabs-ai#738)) ([c0745d4](headroomlabs-ai@c0745d4)) * **proxy:** pilot hardening — inbound auth, security headers, audit log, air-gap switch ([headroomlabs-ai#1537](headroomlabs-ai#1537)) ([546ab55](headroomlabs-ai@546ab55)) * **proxy:** support glob patterns in exclude_tools ([headroomlabs-ai#870](headroomlabs-ai#870)) ([headroomlabs-ai#1259](headroomlabs-ai#1259)) ([a2159c0](headroomlabs-ai@a2159c0)) * **read-maturation:** activity-based hold-back Read maturation (Mechanism B) ([headroomlabs-ai#1068](headroomlabs-ai#1068)) ([723b80c](headroomlabs-ai@723b80c)) * **savings:** durable savings ledger + headroom savings command ([headroomlabs-ai#1127](headroomlabs-ai#1127)) ([978ffa0](headroomlabs-ai@978ffa0)) * **wrap:** add --1m to preserve the 1M context window on wrap claude ([headroomlabs-ai#1158](headroomlabs-ai#1158)) ([headroomlabs-ai#1351](headroomlabs-ai#1351)) ([b50d9c1](headroomlabs-ai@b50d9c1)) * **wrap:** make tokensave the primary coding-task compressor, Serena the backup ([headroomlabs-ai#1230](headroomlabs-ai#1230)) ([dca9853](headroomlabs-ai@dca9853)) ### Bug Fixes * **agent-evals:** Phase 0 — coding-agent accuracy A/B framework ([headroomlabs-ai#1037](headroomlabs-ai#1037)) ([84f9871](headroomlabs-ai@84f9871)) * **agno:** tolerate streaming tool-call SDK objects in parser ([headroomlabs-ai#1312](headroomlabs-ai#1312)) ([headroomlabs-ai#1336](headroomlabs-ai#1336)) ([5986c22](headroomlabs-ai@5986c22)) * **bedrock:** add boto3 1.41 + CRT for aws login credentials ([headroomlabs-ai#1486](headroomlabs-ai#1486)) ([4db3bc9](headroomlabs-ai@4db3bc9)) * bump codebase-memory-mcp to v0.8.1 ([headroomlabs-ai#1284](headroomlabs-ai#1284)) ([530318b](headroomlabs-ai@530318b)) * **ccr:** make headroom_retrieve a hash-only full-content lookup ([headroomlabs-ai#1532](headroomlabs-ai#1532)) ([c2fc4d3](headroomlabs-ai@c2fc4d3)) * **ccr:** propagate --no-ccr-marker flag to all compressors ([headroomlabs-ai#1022](headroomlabs-ai#1022)) ([headroomlabs-ai#1197](headroomlabs-ai#1197)) ([0c9b42a](headroomlabs-ai@0c9b42a)) * **ccr:** skip Anthropic marker emission when tool injection is deferred ([headroomlabs-ai#1273](headroomlabs-ai#1273)) ([2cae13d](headroomlabs-ai@2cae13d)) * **ci:** extend gitleaks allowlist to cover test fixtures + verified examples ([headroomlabs-ai#1539](headroomlabs-ai#1539)) ([d2565a6](headroomlabs-ai@d2565a6)) * **ci:** guarantee model present in test shards to end cache-miss flakiness ([headroomlabs-ai#1399](headroomlabs-ai#1399)) ([2e29c72](headroomlabs-ai@2e29c72)) * **ci:** normalize Windows CRLF line endings in PR governance script ([headroomlabs-ai#1012](headroomlabs-ai#1012)) ([5194388](headroomlabs-ai@5194388)) * **cli:** add explicit UTF-8 encoding to file I/O in wrap commands ([headroomlabs-ai#1126](headroomlabs-ai#1126)) ([headroomlabs-ai#1164](headroomlabs-ai#1164)) ([a0cb798](headroomlabs-ai@a0cb798)) * **cli:** fall back gracefully when embedding-server sidecar is absent ([headroomlabs-ai#1206](headroomlabs-ai#1206)) ([38f1404](headroomlabs-ai@38f1404)) * **cli:** harden all CLI surfaces + fix docs accuracy ([headroomlabs-ai#1491](headroomlabs-ai#1491)) ([bd76235](headroomlabs-ai@bd76235)) * **cli:** wire --http2/--no-http2 (HEADROOM_HTTP2) into proxy command ([headroomlabs-ai#1373](headroomlabs-ai#1373)) ([e06b616](headroomlabs-ai@e06b616)) * **cli:** wire --rpm/--tpm and HEADROOM_RPM/HEADROOM_TPM to the Click proxy command ([headroomlabs-ai#1375](headroomlabs-ai#1375)) ([8aab8f2](headroomlabs-ai@8aab8f2)) * **code:** slice tree-sitter byte offsets as UTF-8 ([headroomlabs-ai#1332](headroomlabs-ai#1332)) ([8238402](headroomlabs-ai@8238402)) * **code:** validate Python compressed syntax ([headroomlabs-ai#1302](headroomlabs-ai#1302)) ([cbd361d](headroomlabs-ai@cbd361d)) * **code:** verify a real parse in tree-sitter availability check ([headroomlabs-ai#1231](headroomlabs-ai#1231)) ([headroomlabs-ai#1299](headroomlabs-ai#1299)) ([5e0bb69](headroomlabs-ai@5e0bb69)) * **codex:** retag threads on init so Codex Desktop history stays visible ([headroomlabs-ai#961](headroomlabs-ai#961)) ([headroomlabs-ai#1349](headroomlabs-ai#1349)) ([e6bbc40](headroomlabs-ai@e6bbc40)) * **codex:** stop pinning Codex memory MCP to one project db ([headroomlabs-ai#1269](headroomlabs-ai#1269)) ([ad7993b](headroomlabs-ai@ad7993b)) * **dashboard:** include RTK stats in the historical tab ([headroomlabs-ai#1324](headroomlabs-ai#1324)) ([35939c3](headroomlabs-ai@35939c3)) * **deps:** remediate dependency CVEs and publish SBOM ([headroomlabs-ai#1509](headroomlabs-ai#1509)) ([5771a80](headroomlabs-ai@5771a80)) * **docker:** persist session history across container revisions ([headroomlabs-ai#1118](headroomlabs-ai#1118)) ([5912d65](headroomlabs-ai@5912d65)) * **gemini:** offload compression to the executor ([headroomlabs-ai#1382](headroomlabs-ai#1382)) ([615848e](headroomlabs-ai@615848e)) * **gemini:** resolve Google model capabilities through ModelRegistry ([headroomlabs-ai#1276](headroomlabs-ai#1276)) ([17ecad9](headroomlabs-ai@17ecad9)) * **install:** guard install_agent_ensure against duplicate runtime spawns ([headroomlabs-ai#1301](headroomlabs-ai#1301)) ([8da0b4e](headroomlabs-ai@8da0b4e)) * **install:** repair macOS launchd restart/start lifecycle ([headroomlabs-ai#1290](headroomlabs-ai#1290)) ([da1a397](headroomlabs-ai@da1a397)) * **install:** stop duplicating ENTRYPOINT in persistent-docker runtime command ([headroomlabs-ai#833](headroomlabs-ai#833)) ([headroomlabs-ai#1348](headroomlabs-ai#1348)) ([feedead](headroomlabs-ai@feedead)) * **io:** use UTF-8 with locale fallback and preserve line endings on config/text I/O ([headroomlabs-ai#1498](headroomlabs-ai#1498)) ([1baa04e](headroomlabs-ai@1baa04e)) * **kompress:** hard override keeps must-keep tokens regardless of model score ([headroomlabs-ai#1400](headroomlabs-ai#1400)) ([42612c8](headroomlabs-ai@42612c8)) * **langchain:** disable streaming on wrapped model during ainvoke() ([headroomlabs-ai#1287](headroomlabs-ai#1287)) ([3590046](headroomlabs-ai@3590046)) * **mcp:** register managed installs with a resolvable headroom command ([headroomlabs-ai#1386](headroomlabs-ai#1386)) ([22def93](headroomlabs-ai@22def93)) * **mcp:** report correct savings_percent in headroom_compress ([headroomlabs-ai#1106](headroomlabs-ai#1106)) ([f216e43](headroomlabs-ai@f216e43)) * **opencode:** write local MCP config ([headroomlabs-ai#1381](headroomlabs-ai#1381)) ([6c83790](headroomlabs-ai@6c83790)) * **packaging:** move hnswlib to optional [vector] extra so [all] needs no C++ toolchain ([headroomlabs-ai#1499](headroomlabs-ai#1499)) ([80fa086](headroomlabs-ai@80fa086)) * patch rtk hook script to use absolute path after register_claude_hooks ([headroomlabs-ai#571](headroomlabs-ai#571)) ([b618d2d](headroomlabs-ai@b618d2d)) * **perf:** surface RTK/CLI context-tool savings in perf and the session card ([headroomlabs-ai#1433](headroomlabs-ai#1433)) ([9362747](headroomlabs-ai@9362747)) * **proxy:** add --protect-tool-results to prevent lossy compression of exact-output Bash results ([headroomlabs-ai#1374](headroomlabs-ai#1374)) ([51d4bcf](headroomlabs-ai@51d4bcf)) * **proxy:** add an Anthropic buffered read-timeout override ([headroomlabs-ai#1331](headroomlabs-ai#1331)) ([3be2526](headroomlabs-ai@3be2526)) * **proxy:** add versionless Vertex AI routes for Claude Code compatibility ([headroomlabs-ai#1321](headroomlabs-ai#1321)) ([bb3e040](headroomlabs-ai@bb3e040)) * **proxy:** bind before eager preload so a hung compressor load can't block startup ([headroomlabs-ai#1500](headroomlabs-ai#1500)) ([d5ac07f](headroomlabs-ai@d5ac07f)) * **proxy:** build SSL contexts for custom CA bundles ([headroomlabs-ai#1134](headroomlabs-ai#1134)) ([561ba17](headroomlabs-ai@561ba17)) * **proxy:** forward request-id headers on the streaming path ([headroomlabs-ai#1100](headroomlabs-ai#1100)) ([headroomlabs-ai#1258](headroomlabs-ai#1258)) ([3d59df7](headroomlabs-ai@3d59df7)) * **proxy:** gate CCR retrieve/compress endpoints to loopback ([headroomlabs-ai#1338](headroomlabs-ai#1338)) ([acafb2d](headroomlabs-ai@acafb2d)) * **proxy:** honor force_kompress routing profile ([headroomlabs-ai#996](headroomlabs-ai#996)) ([b4682d6](headroomlabs-ai@b4682d6)) * **proxy:** keep large compression results on the critical path ([headroomlabs-ai#296](headroomlabs-ai#296)) ([headroomlabs-ai#1352](headroomlabs-ai#1352)) ([90734b6](headroomlabs-ai@90734b6)) * **proxy:** offload /v1/compress to the compression executor to stop blocking the loop ([headroomlabs-ai#1501](headroomlabs-ai#1501)) ([27e010e](headroomlabs-ai@27e010e)) * **proxy:** preserve Responses memory continuations with store=false ([headroomlabs-ai#1103](headroomlabs-ai#1103)) ([cdfeeac](headroomlabs-ai@cdfeeac)) * **proxy:** queue mid-turn user messages on non-Bedrock streaming path ([headroomlabs-ai#1377](headroomlabs-ai#1377)) ([b09f027](headroomlabs-ai@b09f027)) * **proxy:** register interceptor in explicit transforms list when HEADROOM_INTERCEPT_ENABLED ([headroomlabs-ai#1376](headroomlabs-ai#1376)) ([55c700c](headroomlabs-ai@55c700c)) * **proxy:** report real input tokens on streaming message_start ([headroomlabs-ai#1132](headroomlabs-ai#1132)) ([headroomlabs-ai#1305](headroomlabs-ai#1305)) ([70cc96a](headroomlabs-ai@70cc96a)) * **proxy:** retry upstream 429 with Retry-After on both forwarders ([headroomlabs-ai#1329](headroomlabs-ai#1329)) ([90bee89](headroomlabs-ai@90bee89)) * **proxy:** retry upstream 529 overloaded like 429 on both forwarders ([headroomlabs-ai#1495](headroomlabs-ai#1495)) ([547b15d](headroomlabs-ai@547b15d)) * **proxy:** stop re-compressing headroom_retrieve output and emitting unredeemable markers ([headroomlabs-ai#1323](headroomlabs-ai#1323)) ([43494ff](headroomlabs-ai@43494ff)) * **proxy:** strip Codex lite header from OpenAI WebSockets ([headroomlabs-ai#1543](headroomlabs-ai#1543)) ([5d3803a](headroomlabs-ai@5d3803a)) * **read-lifecycle:** persist STALE Read originals in the CCR store ([headroomlabs-ai#1488](headroomlabs-ai#1488)) ([9157173](headroomlabs-ai@9157173)) * recover persistent proxy feature checks and reject non-Copilot exchange URL ([headroomlabs-ai#1465](headroomlabs-ai#1465)) ([16c638b](headroomlabs-ai@16c638b)) * remove agents.md ([headroomlabs-ai#1540](headroomlabs-ai#1540)) ([a7d3360](headroomlabs-ai@a7d3360)) * respect COPILOT_PROVIDER_TYPE env var when provider_type is auto ([headroomlabs-ai#549](headroomlabs-ai#549)) ([24cf256](headroomlabs-ai@24cf256)) * restore token-mode compression on frozen prefixes ([headroomlabs-ai#1489](headroomlabs-ai#1489)) ([8e0dadf](headroomlabs-ai@8e0dadf)) * **router:** degrade to pure-Python detection on native panic ([headroomlabs-ai#1123](headroomlabs-ai#1123)) ([headroomlabs-ai#1260](headroomlabs-ai#1260)) ([a00fb67](headroomlabs-ai@a00fb67)) * **rtk:** stop hook registration timing out on a forked daemon ([headroomlabs-ai#1314](headroomlabs-ai#1314)) ([9758817](headroomlabs-ai@9758817)) * **smart-crusher:** honor enable_ccr_marker on the opaque-blob path ([headroomlabs-ai#1130](headroomlabs-ai#1130)) ([27d6f8e](headroomlabs-ai@27d6f8e)) * **subscription:** only reset 5h contribution on real rollover, not API jitter ([headroomlabs-ai#1255](headroomlabs-ai#1255)) ([8d6c175](headroomlabs-ai@8d6c175)) * **subscription:** run transcript token scan off the event loop ([headroomlabs-ai#1263](headroomlabs-ai#1263)) ([f03021f](headroomlabs-ai@f03021f)) * surface output reduction without a restart, and explain $0.00 savings on Python 3.14 ([headroomlabs-ai#1296](headroomlabs-ai#1296)) ([c30ec4c](headroomlabs-ai@c30ec4c)) * **tests:** reset whole headroom logger subtree so caplog stays deterministic ([headroomlabs-ai#1117](headroomlabs-ai#1117)) ([fda4670](headroomlabs-ai@fda4670)) * **tls:** add HEADROOM_TLS_STRICT=0 toggle for corporate SSL inspection ([headroomlabs-ai#1308](headroomlabs-ai#1308)) ([headroomlabs-ai#1341](headroomlabs-ai#1341)) ([52068dd](headroomlabs-ai@52068dd)) * **tokenizers:** price CJK/Kana/Hangul at ~1 token per char in EstimatingTokenCounter ([headroomlabs-ai#1093](headroomlabs-ai#1093)) ([a35fe86](headroomlabs-ai@a35fe86)) * **transforms:** gate tool string output from lossy compression ([headroomlabs-ai#1307](headroomlabs-ai#1307)) ([headroomlabs-ai#1387](headroomlabs-ai#1387)) ([c6c921a](headroomlabs-ai@c6c921a)) * **websocket:** harden responses websocket origin handling ([headroomlabs-ai#1481](headroomlabs-ai#1481)) ([c632023](headroomlabs-ai@c632023)) * **windows:** pin UTF-8 encoding on text-mode subprocess calls ([headroomlabs-ai#1311](headroomlabs-ai#1311)) ([d633e81](headroomlabs-ai@d633e81)) * **wrap:** add Copilot unwrap command ([headroomlabs-ai#1251](headroomlabs-ai#1251)) ([b4fde0c](headroomlabs-ai@b4fde0c)) * **wrap:** isolate proxy stdio from proxy.log on Windows ([headroomlabs-ai#1191](headroomlabs-ai#1191)) ([959ab0d](headroomlabs-ai@959ab0d)) * **wrap:** keep agent savings opt-in ([headroomlabs-ai#1294](headroomlabs-ai#1294)) ([b829ceb](headroomlabs-ai@b829ceb)) * **wrap:** show the dashboard URL when the proxy is already running ([headroomlabs-ai#1313](headroomlabs-ai#1313)) ([b0146c4](headroomlabs-ai@b0146c4)) ### Performance Improvements * **compression:** take large cold-start contexts off the synchronous kompress path ([headroomlabs-ai#1171](headroomlabs-ai#1171)) ([headroomlabs-ai#1298](headroomlabs-ai#1298)) ([6c68ff4](headroomlabs-ai@6c68ff4)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Upstream HTTP 529 (
overloaded_error) is not retried consistently, so it leaks to clients even though the sibling 429 path was fixed in #1221._stream_response) special-cased onlystatus_code == 429. A529falls through tobreakand is forwarded to the client with zero retries — interactive (streaming) Claude Code sessions see "Overloaded" immediately on a transient Anthropic overload._retry_request) retried529only via the generic>= 500path: it ignoresRetry-Afterand raises anHTTPStatusErroron exhaustion instead of returning the clean529verbatim (inconsistent with how 429 is handled right above it).529is documented by Anthropic as the transient "overloaded" status — semantically identical to 429 for retry purposes ("try again shortly"). This PR routes both through one shared,Retry-After-honoring branch.Related: #1221 (added the 429 retry this extends).
Type of Change
Changes Made
RETRYABLE_OVERLOAD_STATUSES = frozenset({429, 529})toproxy/helpers.pyas the single source of truth shared by both forwarders.streaming.py: retry whenstatus_code in RETRYABLE_OVERLOAD_STATUSES(was== 429); log line now interpolates the actual status.server.py_retry_request: handle429/529in oneRetry-After-honoring branch that returns the status verbatim onceretry_max_attemptsis exhausted (529 no longer goes through the 5xx raise path). Other 4xx/5xx behavior is unchanged.retry_max_attempts/retry_*_delay_msconfig.Testing
pytest)ruff check .)mypy headroom)Test Output
Reproduced the CI
lint+commitlintjobs exactly (pinnedruff==0.15.17,mypy==1.20.2,@commitlint/config-conventional), plus the affected proxy test subset:Real Behavior Proof
headroom proxy --mode token --backend anthropic --no-optimize ...) in front of a fake Anthropic upstream that returns a real HTTP 529 ({"error":{"type":"overloaded_error"}},Retry-After: 0) on request Add Claude Opus 4.5 and Claude 4 model family to context limits #1 then a 200 SSE stream on request [BUG] Decompression error: ZlibError #2. Real proxy process over real sockets (a synthetic upstream is used because real Anthropic 529s cannot be induced on demand).:9911and the branch proxy on:9912with--anthropic-api-url http://127.0.0.1:9911, then sent a streaming request:curl -sN -X POST http://127.0.0.1:9912/v1/messages -H 'x-api-key: …' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d '{"model":"claude-3-5-sonnet-20241022","max_tokens":16,"stream":true,"messages":[{"role":"user","content":"hi"}]}'(full scripts in the code block below).HTTP/1.1 200 OKand the complete SSE stream (message_start … "hello" … message_stop), and the fake upstream logged two calls —call #1returned 529,call #2returned 200 — i.e. the proxy transparently retried the 529 and the overload never reached the client. Onmainthe streaming path forwards the 529 on call Add Claude Opus 4.5 and Claude 4 model family to context limits #1 with no retry, exactly whattest_stream_response_retries_529pins atcalls == 1.pytest tests scripts/testsjob (needs CI model/torch infra) — ran the proxy suite subset above instead; the Rust jobs and non-Anthropic backends (unchanged by this PR).Review Readiness
Checklist
Additional Notes
lintjob exactly (fresh venv, pinnedruff==0.15.17+mypy==1.20.2,ruff check ./ruff format --check ./mypy headroom --ignore-missing-imports) andcommitlint(@commitlint/config-conventional) — all clean. The fulltestshards (model/torch) and Rust jobs were not run locally (no GPU/model cache / Rust toolchain in this environment); they are unaffected by this Python-only change.CHANGELOG.md's## Unreleasedsection currently contains unresolved merge-conflict markers onmain(<<<<<<< … >>>>>>>) unrelated to this PR; I added my entry to the clean### Bug Fixeslist above that region without touching the conflicts.