[Storage] HF: pin click<8.3.0 when installing huggingface_hub on the runtime (fixes ray status / stuck INIT)#9763
Conversation
…anch 'master' of github.com:skypilot-org/skypilot
The COPY-mode HF download installs `huggingface_hub>=1.5` into the remote runtime venv. huggingface_hub>=1.5 declares `click>=8.4` (core dep + typer), which upgrades click past the runtime's `ray==2.9.3` ceiling. ray 2.9.3's CLI crashes at import under click>=8.3 (deepcopy of click Sentinel values, ray-project/ray#56747), so `ray status` exits non-zero, SkyPilot marks the cluster "abnormal", and it gets stuck in INIT (CLI can no longer tail logs / read the queue even though the job ran). Pass a uv `--overrides` file pinning `click<8.3.0` when installing huggingface_hub so click stays below the ray-breaking version while still pulling huggingface_hub's other deps. The bucket API works fine with the older click. A plain `click<8.3.0` constraint would be unsatisfiable against the declared `>=8.4`, hence an override rather than a constraint. Tested: - Repro in a py3.10 venv: `ray[default]==2.9.3` + `huggingface_hub>=1.5` upgrades click to 8.4.1 and `ray status` crashes with the Sentinel error. - With the override: click resolves to 8.2.1, `ray status` no longer crashes (reaches the connection step), and `huggingface_hub` + `HfApi().sync_bucket` import cleanly. - Added a regression unit test; full HF suite: 84 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address review on skypilot-org#9763: - Write ~/.sky/hf_uv_override.txt atomically (mktemp + mv) and only if absent, so parallel COPY syncs running this install concurrently can't truncate the file while another's `uv pip install` reads it. - Remove test_hf_hub_install_pins_click_for_ray: it only mirrored the command string (change-detector), not behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| f'[ -f {_HF_UV_OVERRIDE_FILE} ] || ' | ||
| f'(tmpfile=$(mktemp {_HF_UV_OVERRIDE_FILE}.XXXXXX) && ' | ||
| f'printf "click<8.3.0\\n" > "$tmpfile" && ' | ||
| f'mv -f "$tmpfile" {_HF_UV_OVERRIDE_FILE})', | ||
| f'{constants.SKY_UV_PIP_CMD} install ' | ||
| f'--overrides {_HF_UV_OVERRIDE_FILE} "huggingface_hub>=1.5"', |
There was a problem hiding this comment.
why do we need an override file, instead of directly specifying in the uv pip install click<8.3.0?
There was a problem hiding this comment.
uv pip install --overrides accepts a file path only — there is no --override "click<8.3.0" form (tested: error: File not found: 'click<8.3.0'). And a plain click<8.3.0 argument isn't an override at all — it's a constraint, which silently downgrades huggingface_hub to 1.16.1. So to override the declared click>=8.4, uv requires a file.
There was a problem hiding this comment.
But that's a good point for simplicity. Updated to uv pip install huggingface_hub>=1.5 first, and then uv pip install click<8.3.0 to avoid the file issue.
There was a problem hiding this comment.
Separating into two install commands introduce a 1~2 seconds gap where click is 8.4.1 on disk.
click 8.3.0 -> 8.4.1 -> 8.3.0
Might need to revert to avoid race.
Replace the `uv pip install --overrides <file>` approach with a simpler second `uv pip install "click<8.3.0"` after installing huggingface_hub, mirroring the runtime's ray install line. This keeps huggingface_hub at the latest version and downgrades only click (→ 8.2.1), with no override file, no atomic-write, and no race to handle. Addresses review on skypilot-org#9763 (why an override file): uv has no inline override (--overrides takes a file only), and putting `click<8.3.0` on the same install line silently downgrades huggingface_hub to an older release instead. A standalone `uv pip install "click<8.3.0"` does a targeted in-place downgrade. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cb9ef55 to
f3b8efe
Compare
Expand the comment on _GET_HF_HUB to explain why we cap click with a uv --overrides file instead of (1) a same-line click<8.3.0 constraint (silently downgrades huggingface_hub) or (2) a separate 'uv pip install click<8.3.0' after huggingface_hub (briefly installs click 8.4.x, racing a concurrent ray status -> cluster flips to INIT). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ebb52b5 to
7e31e06
Compare
Problem
PR #9698 added Hugging Face Buckets/repos as a storage backend. For COPY-mode
HF sources, the cluster downloads via the
huggingface_hubPython SDK, so theruntime installs it on the box:
huggingface_hub>=1.5declaresclick>=8.4(core dep, plustyper). Butthe SkyPilot runtime pins
ray==2.9.3, whose CLI crashes at import underclick>=8.3 —
copy.deepcopyof a clickSentinelvalue(ray-project/ray#56747):
So installing
huggingface_hubupgrades click to 8.4.x and breaksray status.The status-refresh daemon then sees
ray statusexit non-zero, marks thecluster "abnormal", and forces it to
INIT— even though the node ishealthy and the job ran.
sky launchthen fails when it tries to tail logs.The runtime already pins
click<8.3.0when installing ray, but the laterhuggingface_hubinstall overrides it.Fix
Install
huggingface_hubwith a uv--overridesfile that pinsclick<8.3.0,so click is capped in a single resolution — it never gets installed at the
ray-breaking
>=8.3even transiently — while huggingface_hub stays at thelatest version (1.17.0) and its other deps install normally. Result on the
cluster: huggingface_hub 1.17.0 + click 8.2.1.
Why a uv
--overridesfile rather than the obvious alternatives:click<8.3.0on the same install line is a co-requirement (constraint),not an override: it conflicts with huggingface_hub's declared
click>=8.4,so uv silently downgrades huggingface_hub to an older release (1.16.1).
uv pip install "click<8.3.0"afterwards keeps the latesthuggingface_hub, but click is briefly installed at
8.4.xbetween the twocommands — a window in which a concurrent
ray status(status-refreshdaemon) would crash and transiently flip the cluster to INIT.
click never reaches
8.3+at all. uv's--overridesonly accepts a file(no inline form), and the file is written atomically so parallel COPY syncs
don't race on it.
Why capping
clickis safeSkyPilot only uses huggingface_hub's Python SDK (
HfApi,sync_bucket,download_bucket_files,snapshot_download,hf_hub_download) — it neverinvokes the
hfCLI. In huggingface_hub,clickis used only by the CLI(
cli/*); the bucket API (_buckets.py) and the SDK entry points importneither
clicknortyper. Verified on the installed 1.17.0:The
click>=8.4floor was added as an explicit CLI dependency(huggingface/huggingface_hub#4270) and is unrelated to the SDK. Importing the
SDK and exercising the bucket methods works with
click==8.2.1(and with8.1.8locally) — below ray's 8.3 break and above the>=8.2.1thattyperneeds. So the downgrade can't affect the code paths we actually use.
Tested
Isolated repro (py3.10 venv):
ray[default]==2.9.3+huggingface_hub>=1.5upgrades click to 8.4.1 and
ray statuscrashes with theSentinelerror;with the
--overrides click<8.3.0, click resolves to 8.2.1,ray statusnolonger crashes, and
huggingface_hub/HfApi().sync_bucketimport cleanly.End-to-end
sky launchon AWS with a task exercising every HF mode (bucketCOPY / COPY-list-of-files / MOUNT / MOUNT_CACHED, plus HF model & dataset repo
mounts). Same YAML and cluster name, master vs this branch:
Job submittedwithClusterNotUpError: ... status: INIT; cluster stuck in INIT, logs neverstreamed.
pass (read-only repo write rejected, circular symlink excluded,
exit 0),ending with
✓ Job finished (status: SUCCEEDED)and the normal footer — noINIT/ClusterNotUpError.Test task YAML
Before —
sky launchon master (fails, stuck INIT)After —
sky launchon this branch (succeeds, logs stream to completion)Related
huggingface_hub>=1.5runtime install.runtime click pin originates from Pin click version for correct argument parsing #5597.
Follow-up (not in this PR)
SkyPilot flips a healthy
UPnode toINITpurely because theray statusCLI crashes at import (the cloud query returned
ClusterStatus.UP). Worthhardening the health check to distinguish a CLI import failure from a genuinely
unhealthy ray cluster, but that's separate from this dependency fix.