fix(tokenizers): bound tiktoken vocab load so a stalled download cannot hang requests (#956)#994
Conversation
PR governanceThis PR follows the template and is marked ready for human review. |
Syncs the fork's main with our fixes (open upstream PRs headroomlabs-ai#990-headroomlabs-ai#995): - fix/mcp-status-config-path: mcp status checks ~/.claude.json (headroomlabs-ai#990) - fix/configurable-compression-timeout: COMPRESSION_TIMEOUT_SECONDS env (headroomlabs-ai#991) - fix/unwrap-undoes-init: unwrap strips ANTHROPIC_BASE_URL + init hooks (headroomlabs-ai#992) - fix/litellm-optional-py314: litellm optional on Python 3.14 (headroomlabs-ai#993) - fix/tiktoken-load-timeout-956: bound tiktoken vocab load (headroomlabs-ai#994) - fix/init-enable-tool-search: init sets ENABLE_TOOL_SEARCH=true (headroomlabs-ai#995)
…t hang requests (chopratejas#956) tiktoken downloads its BPE vocab via requests.get with NO timeout, and headroom loads it lazily inside the request worker (TiktokenCounter.encoding property; AnthropicProvider.__init__ fallback at providers/anthropic.py). On a firewalled/stalled network that download blocks indefinitely, so the compression worker hangs and asyncio.wait_for trips at the 30s timeout on every request with 0 transforms -- the runtime half of chopratejas#956. (The reporter's PyO3/event-loop-on-3.14 theory was disproven: compression itself runs in ~118ms on 3.14.) Run the encoding load on a worker thread bounded by HEADROOM_TIKTOKEN_LOAD_TIMEOUT_SECONDS (default 10s); on timeout raise TiktokenLoadError and fall back to estimation (registry -> EstimatingTokenCounter; AnthropicProvider -> character estimate). The first timed-out encoding is remembered so later requests fail fast instead of re-blocking every time. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…estimation fallback (headroomlabs-ai#956) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
4d19cb6 to
d3093cd
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
d3093cd to
1ff0faf
Compare
JerrettDavis
left a comment
There was a problem hiding this comment.
This PR is not merge-ready because CI is currently red: test (2)=FAILURE, test-extras=FAILURE. Please fix or rerun the failing checks after updating from current main.
|
Updated the branch from current The tiktoken-timeout fix itself is unchanged and still green locally ( |
🤖 I have created a release *beep* *boop* --- <details><summary>0.27.0</summary> ## [0.27.0](v0.26.0...v0.27.0) (2026-06-22) ### Features * **cli:** add headroom doctor setup diagnostics ([#926](#926)) ([e45cf4e](e45cf4e)) * **cli:** add headroom update command and release banner ([#1088](#1088)) ([26be2c3](26be2c3)) * compression extraction — Rust knob exposure, CCR hardening, traffic audits ([#818](#818)) ([b7be381](b7be381)) * measure and surface token throughput (tokens/sec) through the proxy ([#983](#983)) ([0d89c67](0d89c67)) * output-token reduction — verbosity shaper, per-user learning, counterfactual savings ([#965](#965)) ([a99dc61](a99dc61)) * **policy:** decay P_alive from idle time near cache TTL ([#856](#856) P3b) ([#1028](#1028)) ([fe4f9ee](fe4f9ee)) * **providers:** add Cortex Code (Snowflake CoCo) as a supported agent ([#1190](#1190)) ([d9d0bf4](d9d0bf4)) * **proxy:** cc-switch reconciler — keep Headroom in the request path alongside cc-switch ([#1030](#1030)) ([e8fc8a0](e8fc8a0)) * **proxy:** hot-reload live env knobs so a reused proxy picks them up without a restart ([#1090](#1090)) ([6904d47](6904d47)) * **proxy:** make COMPRESSION_TIMEOUT_SECONDS configurable via env ([#946](#946)) ([#991](#991)) ([addebdb](addebdb)) * **transforms:** tabular + spreadsheet (.xlsx/.xls) compression ([#1128](#1128)) ([d789a7c](d789a7c)) * **vertex:** turnkey Claude Code + Vertex compression (+ fixes from the Vertex review) ([#1113](#1113)) ([0e05915](0e05915)) ### Bug Fixes * **ccr:** accept 12-char SmartCrusher hashes in tool injection ([#1095](#1095)) ([#1141](#1141)) ([9f7f3ad](9f7f3ad)) * **ccr:** return stored content when headroom_retrieve query matches nothing ([#1213](#1213)) ([#1236](#1236)) ([08fb845](08fb845)) * **content-router:** honor target_ratio in compression cache + add proxy --target-ratio flag ([#1108](#1108)) ([8894ee0](8894ee0)) * **dashboard:** light-mode backgrounds + aligned savings tables ([#1064](#1064)) ([5eae32b](5eae32b)) * **deps:** make litellm optional on Python 3.14 ([#956](#956)) ([#993](#993)) ([b2f04e4](b2f04e4)) * **e2e:** align Codex wrap e2e with global-only RTK guidance ([#1240](#1240)) ([#1254](#1254)) ([bc12ace](bc12ace)) * **init:** set ENABLE_TOOL_SEARCH=true so Claude Code keeps deferring tools ([#746](#746)) ([#995](#995)) ([500ec2b](500ec2b)) * **kompress:** never block the request path on the cold-cache model download ([#1161](#1161)) ([3fc2a78](3fc2a78)) * **memory:** use ONNX embedder for `wrap --memory` sync ([#1092](#1092)) ([#1262](#1262)) ([4f9feda](4f9feda)) * **openclaw:** wrap plugin export as {register} object for OpenClaw 2026.x compatibility ([#1218](#1218)) ([2e6c442](2e6c442)) * **providers:** update DeepSeek V3 context limit from 128K to 1M ([#1038](#1038)) ([#1137](#1137)) ([bcabc5c](bcabc5c)) * **proxy:** allow disabling periodic TOIN stats logging ([#1265](#1265)) ([b5f63d8](b5f63d8)) * **proxy:** honor HEADROOM_EXCLUDE_TOOLS for Codex /v1/responses tool outputs ([#940](#940)) ([#1053](#1053)) ([f03e77b](f03e77b)) * **proxy:** preserve byte-faithful Anthropic tool forwarding ([#1222](#1222)) ([1f18d59](1f18d59)) * **proxy:** route Codex OAuth image requests ([#1215](#1215)) ([381d771](381d771)) * **proxy:** scope CORS to loopback + gate operator/content endpoints ([#1226](#1226)) ([bd55a42](bd55a42)) * **proxy:** stamp X-Client: codex on Responses endpoint for unidentified callers ([#1036](#1036)) ([b0cd032](b0cd032)) * **proxy:** treat NODE_EXTRA_CA_CERTS as additive, not replacement ([#998](#998)) ([#1031](#1031)) ([c987283](c987283)) * **telemetry:** switch anonymous telemetry to opt-in (off by default) ([#1223](#1223)) ([b998697](b998697)) * **tokenizers:** bound tiktoken vocab load so a stalled download cannot hang requests ([#956](#956)) ([#994](#994)) ([7e86baf](7e86baf)) * **unwrap:** remove ANTHROPIC_BASE_URL + ENABLE_TOOL_SEARCH and init hooks on unwrap ([#992](#992)) ([5b84691](5b84691)) * **wrap:** keep Codex RTK guidance global ([#1240](#1240)) ([7c26a54](7c26a54)) * **wrap:** percent-encode non-ASCII cwd names in X-Headroom-Project header ([#1071](#1071)) ([9f712cc](9f712cc)) * **wrap:** write env.ANTHROPIC_BASE_URL to settings.json so daemon-spawned conversations inherit proxy ([#951](#951)) ([#1078](#1078)) ([a554c3a](a554c3a)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…ot hang requests (headroomlabs-ai#956) (headroomlabs-ai#994) ## Description The headroomlabs-ai#956 symptom — `compression_first_stage` always timing out at ~30s with 0 tokens removed — is not a PyO3/event-loop issue (compression itself runs ~120ms on Python 3.14). Root cause: `tiktoken` downloads its BPE vocab via `requests.get(...)` with no timeout, loaded lazily inside the compression worker (`TiktokenCounter.encoding`, `AnthropicProvider.__init__`). On a firewalled network that blocks indefinitely, so the worker hangs and `asyncio.wait_for` trips at 30s on every request (the hung download never caches, so it repeats). Refs headroomlabs-ai#956 (runtime half). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Load the encoding on a worker thread bounded by `HEADROOM_TIKTOKEN_LOAD_TIMEOUT_SECONDS` (default 10s); on timeout raise `TiktokenLoadError` and fall back to estimation (registry -> EstimatingTokenCounter; Anthropic provider -> character estimate). - Remember the first timed-out encoding so later requests fail fast instead of re-blocking. ## Testing - [x] Unit tests pass (`pytest`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_tokenizers/test_tiktoken_load_timeout.py -q 4 passed in 0.69s ``` ## Real Behavior Proof - Environment: Linux, Python 3.14 and 3.13 - Exact command / steps: timed the Rust compression call sync / via run_in_executor / 2x concurrent on both interpreters; ran the bounded-loader tests against a simulated stalled get_encoding - Observed result: compression ~118-120ms on both 3.14 and 3.13 (no event-loop block); the bounded loader raises/falls back within the timeout instead of hanging - Not tested: the real firewalled-network stall (could not reproduce on an unfirewalled host); the no-timeout requests.get is confirmed in tiktoken's source ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
🤖 I have created a release *beep* *boop* --- <details><summary>0.27.0</summary> ## [0.27.0](headroomlabs-ai/headroom@v0.26.0...v0.27.0) (2026-06-22) ### Features * **cli:** add headroom doctor setup diagnostics ([headroomlabs-ai#926](headroomlabs-ai#926)) ([e45cf4e](headroomlabs-ai@e45cf4e)) * **cli:** add headroom update command and release banner ([headroomlabs-ai#1088](headroomlabs-ai#1088)) ([26be2c3](headroomlabs-ai@26be2c3)) * compression extraction — Rust knob exposure, CCR hardening, traffic audits ([headroomlabs-ai#818](headroomlabs-ai#818)) ([b7be381](headroomlabs-ai@b7be381)) * measure and surface token throughput (tokens/sec) through the proxy ([headroomlabs-ai#983](headroomlabs-ai#983)) ([0d89c67](headroomlabs-ai@0d89c67)) * output-token reduction — verbosity shaper, per-user learning, counterfactual savings ([headroomlabs-ai#965](headroomlabs-ai#965)) ([a99dc61](headroomlabs-ai@a99dc61)) * **policy:** decay P_alive from idle time near cache TTL ([headroomlabs-ai#856](headroomlabs-ai#856) P3b) ([headroomlabs-ai#1028](headroomlabs-ai#1028)) ([fe4f9ee](headroomlabs-ai@fe4f9ee)) * **providers:** add Cortex Code (Snowflake CoCo) as a supported agent ([headroomlabs-ai#1190](headroomlabs-ai#1190)) ([d9d0bf4](headroomlabs-ai@d9d0bf4)) * **proxy:** cc-switch reconciler — keep Headroom in the request path alongside cc-switch ([headroomlabs-ai#1030](headroomlabs-ai#1030)) ([e8fc8a0](headroomlabs-ai@e8fc8a0)) * **proxy:** hot-reload live env knobs so a reused proxy picks them up without a restart ([headroomlabs-ai#1090](headroomlabs-ai#1090)) ([6904d47](headroomlabs-ai@6904d47)) * **proxy:** make COMPRESSION_TIMEOUT_SECONDS configurable via env ([headroomlabs-ai#946](headroomlabs-ai#946)) ([headroomlabs-ai#991](headroomlabs-ai#991)) ([addebdb](headroomlabs-ai@addebdb)) * **transforms:** tabular + spreadsheet (.xlsx/.xls) compression ([headroomlabs-ai#1128](headroomlabs-ai#1128)) ([d789a7c](headroomlabs-ai@d789a7c)) * **vertex:** turnkey Claude Code + Vertex compression (+ fixes from the Vertex review) ([headroomlabs-ai#1113](headroomlabs-ai#1113)) ([0e05915](headroomlabs-ai@0e05915)) ### Bug Fixes * **ccr:** accept 12-char SmartCrusher hashes in tool injection ([headroomlabs-ai#1095](headroomlabs-ai#1095)) ([headroomlabs-ai#1141](headroomlabs-ai#1141)) ([9f7f3ad](headroomlabs-ai@9f7f3ad)) * **ccr:** return stored content when headroom_retrieve query matches nothing ([headroomlabs-ai#1213](headroomlabs-ai#1213)) ([headroomlabs-ai#1236](headroomlabs-ai#1236)) ([08fb845](headroomlabs-ai@08fb845)) * **content-router:** honor target_ratio in compression cache + add proxy --target-ratio flag ([headroomlabs-ai#1108](headroomlabs-ai#1108)) ([8894ee0](headroomlabs-ai@8894ee0)) * **dashboard:** light-mode backgrounds + aligned savings tables ([headroomlabs-ai#1064](headroomlabs-ai#1064)) ([5eae32b](headroomlabs-ai@5eae32b)) * **deps:** make litellm optional on Python 3.14 ([headroomlabs-ai#956](headroomlabs-ai#956)) ([headroomlabs-ai#993](headroomlabs-ai#993)) ([b2f04e4](headroomlabs-ai@b2f04e4)) * **e2e:** align Codex wrap e2e with global-only RTK guidance ([headroomlabs-ai#1240](headroomlabs-ai#1240)) ([headroomlabs-ai#1254](headroomlabs-ai#1254)) ([bc12ace](headroomlabs-ai@bc12ace)) * **init:** set ENABLE_TOOL_SEARCH=true so Claude Code keeps deferring tools ([headroomlabs-ai#746](headroomlabs-ai#746)) ([headroomlabs-ai#995](headroomlabs-ai#995)) ([500ec2b](headroomlabs-ai@500ec2b)) * **kompress:** never block the request path on the cold-cache model download ([headroomlabs-ai#1161](headroomlabs-ai#1161)) ([3fc2a78](headroomlabs-ai@3fc2a78)) * **memory:** use ONNX embedder for `wrap --memory` sync ([headroomlabs-ai#1092](headroomlabs-ai#1092)) ([headroomlabs-ai#1262](headroomlabs-ai#1262)) ([4f9feda](headroomlabs-ai@4f9feda)) * **openclaw:** wrap plugin export as {register} object for OpenClaw 2026.x compatibility ([headroomlabs-ai#1218](headroomlabs-ai#1218)) ([2e6c442](headroomlabs-ai@2e6c442)) * **providers:** update DeepSeek V3 context limit from 128K to 1M ([headroomlabs-ai#1038](headroomlabs-ai#1038)) ([headroomlabs-ai#1137](headroomlabs-ai#1137)) ([bcabc5c](headroomlabs-ai@bcabc5c)) * **proxy:** allow disabling periodic TOIN stats logging ([headroomlabs-ai#1265](headroomlabs-ai#1265)) ([b5f63d8](headroomlabs-ai@b5f63d8)) * **proxy:** honor HEADROOM_EXCLUDE_TOOLS for Codex /v1/responses tool outputs ([headroomlabs-ai#940](headroomlabs-ai#940)) ([headroomlabs-ai#1053](headroomlabs-ai#1053)) ([f03e77b](headroomlabs-ai@f03e77b)) * **proxy:** preserve byte-faithful Anthropic tool forwarding ([headroomlabs-ai#1222](headroomlabs-ai#1222)) ([1f18d59](headroomlabs-ai@1f18d59)) * **proxy:** route Codex OAuth image requests ([headroomlabs-ai#1215](headroomlabs-ai#1215)) ([381d771](headroomlabs-ai@381d771)) * **proxy:** scope CORS to loopback + gate operator/content endpoints ([headroomlabs-ai#1226](headroomlabs-ai#1226)) ([bd55a42](headroomlabs-ai@bd55a42)) * **proxy:** stamp X-Client: codex on Responses endpoint for unidentified callers ([headroomlabs-ai#1036](headroomlabs-ai#1036)) ([b0cd032](headroomlabs-ai@b0cd032)) * **proxy:** treat NODE_EXTRA_CA_CERTS as additive, not replacement ([headroomlabs-ai#998](headroomlabs-ai#998)) ([headroomlabs-ai#1031](headroomlabs-ai#1031)) ([c987283](headroomlabs-ai@c987283)) * **telemetry:** switch anonymous telemetry to opt-in (off by default) ([headroomlabs-ai#1223](headroomlabs-ai#1223)) ([b998697](headroomlabs-ai@b998697)) * **tokenizers:** bound tiktoken vocab load so a stalled download cannot hang requests ([headroomlabs-ai#956](headroomlabs-ai#956)) ([headroomlabs-ai#994](headroomlabs-ai#994)) ([7e86baf](headroomlabs-ai@7e86baf)) * **unwrap:** remove ANTHROPIC_BASE_URL + ENABLE_TOOL_SEARCH and init hooks on unwrap ([headroomlabs-ai#992](headroomlabs-ai#992)) ([5b84691](headroomlabs-ai@5b84691)) * **wrap:** keep Codex RTK guidance global ([headroomlabs-ai#1240](headroomlabs-ai#1240)) ([7c26a54](headroomlabs-ai@7c26a54)) * **wrap:** percent-encode non-ASCII cwd names in X-Headroom-Project header ([headroomlabs-ai#1071](headroomlabs-ai#1071)) ([9f712cc](headroomlabs-ai@9f712cc)) * **wrap:** write env.ANTHROPIC_BASE_URL to settings.json so daemon-spawned conversations inherit proxy ([headroomlabs-ai#951](headroomlabs-ai#951)) ([headroomlabs-ai#1078](headroomlabs-ai#1078)) ([a554c3a](headroomlabs-ai@a554c3a)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Description
The #956 symptom —
compression_first_stagealways timing out at ~30s with 0 tokens removed — is not a PyO3/event-loop issue (compression itself runs ~120ms on Python 3.14). Root cause:tiktokendownloads its BPE vocab viarequests.get(...)with no timeout, loaded lazily inside the compression worker (TiktokenCounter.encoding,AnthropicProvider.__init__). On a firewalled network that blocks indefinitely, so the worker hangs andasyncio.wait_fortrips at 30s on every request (the hung download never caches, so it repeats). Refs #956 (runtime half).Type of Change
Changes Made
HEADROOM_TIKTOKEN_LOAD_TIMEOUT_SECONDS(default 10s); on timeout raiseTiktokenLoadErrorand fall back to estimation (registry -> EstimatingTokenCounter; Anthropic provider -> character estimate).Testing
pytest)Test Output
Real Behavior Proof
Review Readiness