Skip to content

[Storage] HF: pin click<8.3.0 when installing huggingface_hub on the runtime (fixes ray status / stuck INIT)#9763

Merged
zpoint merged 36 commits into
skypilot-org:masterfrom
zpoint:fix/hf-click-pin-runtime
Jun 1, 2026
Merged

[Storage] HF: pin click<8.3.0 when installing huggingface_hub on the runtime (fixes ray status / stuck INIT)#9763
zpoint merged 36 commits into
skypilot-org:masterfrom
zpoint:fix/hf-click-pin-runtime

Conversation

@zpoint

@zpoint zpoint commented May 30, 2026

Copy link
Copy Markdown
Collaborator

Problem

PR #9698 added Hugging Face Buckets/repos as a storage backend. For COPY-mode
HF sources, the cluster downloads via the huggingface_hub Python SDK, so the
runtime installs it on the box:

# sky/cloud_stores.py (before, from #9698)
_GET_HF_HUB = [
    f'{constants.SKY_UV_PIP_CMD} install "huggingface_hub>=1.5"',
]

huggingface_hub>=1.5 declares click>=8.4 (core dep, plus typer). But
the SkyPilot runtime pins ray==2.9.3, whose CLI crashes at import under
click>=8.3copy.deepcopy of a click Sentinel value
(ray-project/ray#56747):

File ".../ray/scripts/scripts.py", line 2440, in add_command_alias
    new_command = copy.deepcopy(command)
  ...
ValueError: <object object at 0x...> is not a valid Sentinel

So installing huggingface_hub upgrades click to 8.4.x and breaks ray status.
The status-refresh daemon then sees ray status exit non-zero, marks the
cluster "abnormal", and forces it to INIT — even though the node is
healthy and the job ran. sky launch then fails when it tries to tail logs.

The runtime already pins click<8.3.0 when installing ray, but the later
huggingface_hub install overrides it.

Fix

Install huggingface_hub with a uv --overrides file that pins click<8.3.0,
so click is capped in a single resolution — it never gets installed at the
ray-breaking >=8.3 even transiently — while huggingface_hub stays at the
latest version (1.17.0) and its other deps install normally. Result on the
cluster: huggingface_hub 1.17.0 + click 8.2.1.

# sky/cloud_stores.py
_HF_UV_OVERRIDE_FILE = '~/.sky/hf_uv_override.txt'
_GET_HF_HUB = [
    'mkdir -p ~/.sky',
    # Write the override file atomically (tmp file + mv) and only if absent, so
    # parallel COPY syncs can't truncate it while another's uv install reads it.
    f'[ -f {_HF_UV_OVERRIDE_FILE} ] || '
    f'(tmpfile=$(mktemp {_HF_UV_OVERRIDE_FILE}.XXXXXX) && '
    f'printf "click<8.3.0\\n" > "$tmpfile" && '
    f'mv -f "$tmpfile" {_HF_UV_OVERRIDE_FILE})',
    f'{constants.SKY_UV_PIP_CMD} install '
    f'--overrides {_HF_UV_OVERRIDE_FILE} "huggingface_hub>=1.5"',
]

Why a uv --overrides file rather than the obvious alternatives:

  • click<8.3.0 on the same install line is a co-requirement (constraint),
    not an override: it conflicts with huggingface_hub's declared click>=8.4,
    so uv silently downgrades huggingface_hub to an older release (1.16.1).
  • A separate uv pip install "click<8.3.0" afterwards keeps the latest
    huggingface_hub, but click is briefly installed at 8.4.x between the two
    commands — a window in which a concurrent ray status (status-refresh
    daemon) would crash and transiently flip the cluster to INIT.
  • The override caps click within the single huggingface_hub resolution, so
    click never reaches 8.3+ at all. uv's --overrides only accepts a file
    (no inline form), and the file is written atomically so parallel COPY syncs
    don't race on it.

Why capping click is safe

SkyPilot only uses huggingface_hub's Python SDK (HfApi, sync_bucket,
download_bucket_files, snapshot_download, hf_hub_download) — it never
invokes the hf CLI. In huggingface_hub, click is used only by the CLI
(cli/*); the bucket API (_buckets.py) and the SDK entry points import
neither click nor typer. Verified on the installed 1.17.0:

$ grep -rl "import click" huggingface_hub/   # only the CLI uses click
huggingface_hub/cli/_cli_utils.py
huggingface_hub/cli/_help_formatter.py
huggingface_hub/cli/skills.py

The click>=8.4 floor was added as an explicit CLI dependency
(huggingface/huggingface_hub#4270) and is unrelated to the SDK. Importing the
SDK and exercising the bucket methods works with click==8.2.1 (and with
8.1.8 locally) — below ray's 8.3 break and above the >=8.2.1 that typer
needs. So the downgrade can't affect the code paths we actually use.

Tested

Isolated repro (py3.10 venv): ray[default]==2.9.3 + huggingface_hub>=1.5
upgrades click to 8.4.1 and ray status crashes with the Sentinel error;
with the --overrides click<8.3.0, click resolves to 8.2.1, ray status no
longer crashes, and huggingface_hub / HfApi().sync_bucket import cleanly.

End-to-end sky launch on AWS with a task exercising every HF mode (bucket
COPY / COPY-list-of-files / MOUNT / MOUNT_CACHED, plus HF model & dataset repo
mounts). Same YAML and cluster name, master vs this branch:

  • Before (master): errored right after Job submitted with
    ClusterNotUpError: ... status: INIT; cluster stuck in INIT, logs never
    streamed.
  • After (this branch): cluster reaches UP, logs stream, all assertions
    pass (read-only repo write rejected, circular symlink excluded, exit 0),
    ending with ✓ Job finished (status: SUCCEEDED) and the normal footer — no
    INIT / ClusterNotUpError.
Test task YAML
file_mounts:
  # HF Bucket: COPY mode with a source directory.
  /mount_bucket_copy:
    name: <ns>/<bucket>
    source: ~/tmp-workdir
    store: hf
    mode: COPY
  # HF Bucket: COPY mode with a list of files as source.
  /mount_bucket_copy_lof:
    name: <ns>/<bucket>
    source: ['~/tmp-workdir/tmp file', '~/tmp-workdir/tmp file2']
    store: hf
    mode: COPY
  # HF Bucket: MOUNT mode (read-write via hf-mount NFS backend).
  /mount_bucket_mount:
    name: <ns>/<bucket>
    source: ~/tmp-workdir
    store: hf
    mode: MOUNT
  # HF Bucket: MOUNT_CACHED mode.
  /mount_bucket_mount_cached:
    name: <ns>/<bucket>
    source: ~/tmp-workdir
    store: hf
    mode: MOUNT_CACHED
  # HF Repo: mount a public model as a read-only filesystem.
  /mount_hf_model:
    source: hf://openai-community/gpt2
    store: hf
    mode: MOUNT
  # HF Repo: mount a public dataset with an explicit revision.
  /mount_hf_dataset:
    source: hf://datasets/open-index/hacker-news@main
    store: hf
    mode: MOUNT

run: |
  set -ex
  ls -ltr /mount_bucket_copy/foo
  ls -ltr /mount_bucket_copy/tmp\ file
  ls -ltr /mount_bucket_copy_lof/tmp\ file
  ls -ltr /mount_bucket_copy_lof/tmp\ file2
  ls -ltr /mount_bucket_mount/foo
  ls -ltr /mount_bucket_mount/tmp\ file
  # Symlinks are not copied to buckets.
  ! ls /mount_bucket_copy/circle-link
  ! ls /mount_bucket_mount/circle-link
  # Writes to a bucket in MOUNT / MOUNT_CACHED mode should succeed.
  echo "hello"  > /mount_bucket_mount/hello.txt
  echo "hello2" > /mount_bucket_mount_cached/hello2.txt
  # HF repo mounts are read-only: reads work, writes should fail.
  ls /mount_hf_model/config.json
  ls /mount_hf_dataset | head -n 1
  ! echo "should fail" > /mount_hf_model/should-fail.txt
Before — sky launch on master (fails, stuck INIT)
(sky) ➜  skypilot git:(master) ✗ sky launch -c hf-cli-test --cloud aws -y /tmp/hf_cli_run.yaml
The --cloud, --region, and --zone options are deprecated. Use --infra instead.
YAML to run: /tmp/hf_cli_run.yaml
Running on cluster: hf-cli-test
  Created HF bucket 'zp0int/sky-cli-test-1780155989'
✓ Storage synced: ~/tmp-workdir -> hf://buckets/zp0int/sky-cli-test-1780155989  View logs: sky api logs -l sky-2026-05-30-23-46-55-929482/storage_sync.log
✓ Storage synced: 2 paths -> hf://buckets/zp0int/sky-cli-test-1780155989  View logs: sky api logs -l sky-2026-05-30-23-46-58-786678/storage_sync.log
✓ Storage synced: ~/tmp-workdir -> hf://buckets/zp0int/sky-cli-test-1780155989  View logs: sky api logs -l sky-2026-05-30-23-47-00-833764/storage_sync.log
✓ Storage synced: ~/tmp-workdir -> hf://buckets/zp0int/sky-cli-test-1780155989  View logs: sky api logs -l sky-2026-05-30-23-47-01-845945/storage_sync.log
Storage type StoreType.HF already exists.
Storage type StoreType.HF already exists.
Considered resources (1 node):
------------------------------------------------------------------------------
 INFRA             INSTANCE      vCPUs   Mem(GB)   GPUS   COST ($)   CHOSEN
------------------------------------------------------------------------------
 AWS (us-east-1)   m6i.2xlarge   8       32        -      0.38          ✔
------------------------------------------------------------------------------
⚙︎ Launching on AWS us-east-1 (us-east-1-atl-2a,us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1f).
└── Instance is up.
✓ Cluster launched: hf-cli-test.  View logs: sky logs --provision hf-cli-test
⚙︎ Syncing files.
  Syncing (to 1 node): hf://buckets/zp0int/sky-cli-test-1780155989 -> /mount_bucket_copy
  Syncing (to 1 node): hf://buckets/zp0int/sky-cli-test-1780155989 -> /mount_bucket_copy_lof
✓ Synced file_mounts.  View logs: sky api logs -l sky-2026-05-30-23-46-54-090786/file_mounts.log
  Mounting (to 1 node): ~/tmp-workdir -> /mount_bucket_mount
  Mounting cached mode (to 1 node): ~/tmp-workdir -> /mount_bucket_mount_cached
  Mounting (to 1 node): hf://openai-community/gpt2 -> /mount_hf_model
  Mounting (to 1 node): hf://datasets/open-index/hacker-news@main -> /mount_hf_dataset
✓ Storage mounted.  View logs: sky api logs -l sky-2026-05-30-23-46-54-090786/storage_mounts.log
⚙︎ Job submitted, ID: 1
sky.exceptions.ClusterNotUpError: Tailing logs: skipped for cluster 'hf-cli-test' (status: INIT). It is only allowed for UP and AUTOSTOPPING clusters. Wait for a launch to finish, or use this command to try to transition the cluster to UP: sky start hf-cli-test
After — sky launch on this branch (succeeds, logs stream to completion)
(sky) ➜  skypilot git:(fix/hf-click-pin-runtime) ✗ sky launch -c hf-cli-test --infra aws -y /tmp/hf_cli_run.yaml
YAML to run: /tmp/hf_cli_run.yaml
Running on cluster: hf-cli-test
  Created HF bucket 'zp0int/sky-cli-test-1780157806'
✓ Storage synced: ~/tmp-workdir -> hf://buckets/zp0int/sky-cli-test-1780157806  View logs: sky api logs -l sky-2026-05-31-00-17-32-959531/storage_sync.log
✓ Storage synced: 2 paths -> hf://buckets/zp0int/sky-cli-test-1780157806  View logs: sky api logs -l sky-2026-05-31-00-17-35-960443/storage_sync.log
✓ Storage synced: ~/tmp-workdir -> hf://buckets/zp0int/sky-cli-test-1780157806  View logs: sky api logs -l sky-2026-05-31-00-17-38-044506/storage_sync.log
✓ Storage synced: ~/tmp-workdir -> hf://buckets/zp0int/sky-cli-test-1780157806  View logs: sky api logs -l sky-2026-05-31-00-17-39-091384/storage_sync.log
Storage type StoreType.HF already exists.
Storage type StoreType.HF already exists.
Considered resources (1 node):
------------------------------------------------------------------------------
 INFRA             INSTANCE      vCPUs   Mem(GB)   GPUS   COST ($)   CHOSEN
------------------------------------------------------------------------------
 AWS (us-east-1)   m6i.2xlarge   8       32        -      0.38          ✔
------------------------------------------------------------------------------
⚙︎ Launching on AWS us-east-1 (us-east-1-atl-2a,us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1f).
└── Instance is up.
✓ Cluster launched: hf-cli-test.  View logs: sky logs --provision hf-cli-test
⚙︎ Syncing files.
  Syncing (to 1 node): hf://buckets/zp0int/sky-cli-test-1780157806 -> /mount_bucket_copy
  Syncing (to 1 node): hf://buckets/zp0int/sky-cli-test-1780157806 -> /mount_bucket_copy_lof
✓ Synced file_mounts.  View logs: sky api logs -l sky-2026-05-31-00-17-31-174071/file_mounts.log
  Mounting (to 1 node): ~/tmp-workdir -> /mount_bucket_mount
  Mounting cached mode (to 1 node): ~/tmp-workdir -> /mount_bucket_mount_cached
  Mounting (to 1 node): hf://openai-community/gpt2 -> /mount_hf_model
  Mounting (to 1 node): hf://datasets/open-index/hacker-news@main -> /mount_hf_dataset
✓ Storage mounted.  View logs: sky api logs -l sky-2026-05-31-00-17-31-174071/storage_mounts.log
⚙︎ Job submitted, ID: 1
├── Waiting for task resources on 1 node.
└── Job started. Streaming logs... (Ctrl-C to exit log streaming; job will not be killed)
(task, pid=2747) + ls --color=auto -ltr /mount_bucket_copy/foo
(task, pid=2747) -rw-rw-r-- 1 ubuntu ubuntu 0 May 30 16:19 /mount_bucket_copy/foo
(task, pid=2747) + ls --color=auto -ltr '/mount_bucket_copy/tmp file'
(task, pid=2747) -rw-rw-r-- 1 ubuntu ubuntu 0 May 30 16:19 /mount_bucket_copy/tmp file
(task, pid=2747) + ls --color=auto -ltr '/mount_bucket_copy_lof/tmp file'
(task, pid=2747) -rw-rw-r-- 1 ubuntu ubuntu 0 May 30 16:19 /mount_bucket_copy_lof/tmp file
(task, pid=2747) + ls --color=auto -ltr '/mount_bucket_copy_lof/tmp file2'
(task, pid=2747) -rw-rw-r-- 1 ubuntu ubuntu 0 May 30 16:19 /mount_bucket_copy_lof/tmp file2
(task, pid=2747) + ls --color=auto -ltr /mount_bucket_mount/foo
(task, pid=2747) -rw-r--r-- 1 ubuntu ubuntu 0 May 30 16:16 /mount_bucket_mount/foo
(task, pid=2747) + ls --color=auto -ltr '/mount_bucket_mount/tmp file'
(task, pid=2747) -rw-r--r-- 1 ubuntu ubuntu 0 May 30 16:16 /mount_bucket_mount/tmp file
(task, pid=2747) + ls --color=auto /mount_bucket_copy/circle-link
(task, pid=2747) ls: cannot access '/mount_bucket_copy/circle-link': No such file or directory
(task, pid=2747) + ls --color=auto /mount_bucket_mount/circle-link
(task, pid=2747) ls: cannot access '/mount_bucket_mount/circle-link': No such file or directory
(task, pid=2747) + echo hello
(task, pid=2747) + echo hello2
(task, pid=2747) + ls --color=auto /mount_hf_model/config.json
(task, pid=2747) /mount_hf_model/config.json
(task, pid=2747) + head -n 1
(task, pid=2747) + ls --color=auto /mount_hf_dataset
(task, pid=2747) README.md
(task, pid=2747) + echo 'should fail'
(task, pid=2747) bash: /mount_hf_model/should-fail.txt: Read-only file system
(task, pid=2747) + __skypilot_user_exit_code=0
(task, pid=2747) ++ findmnt -t fuse.rclone --noheading
(task, pid=2747) ++ wc -l
(task, pid=2747) + '[' 0 -gt 0 ']'
(task, pid=2747) + exit 0
✓ Job finished (status: SUCCEEDED).

📋 Useful Commands
Job ID: 1
├── To cancel the job:          sky cancel hf-cli-test 1
├── To stream job logs:         sky logs hf-cli-test 1
└── To view job queue:          sky queue hf-cli-test
Cluster name: hf-cli-test
├── To log into the head VM:    ssh hf-cli-test
├── To submit a job:            sky exec hf-cli-test yaml_file
├── To stop the cluster:        sky stop hf-cli-test
└── To teardown the cluster:    sky down hf-cli-test

Related

Follow-up (not in this PR)

SkyPilot flips a healthy UP node to INIT purely because the ray status
CLI crashes at import (the cloud query returned ClusterStatus.UP). Worth
hardening the health check to distinguish a CLI import failure from a genuinely
unhealthy ray cluster, but that's separate from this dependency fix.

zpoint added 30 commits January 9, 2026 11:28
…anch 'master' of github.com:skypilot-org/skypilot
zpoint and others added 4 commits May 30, 2026 16:38
The COPY-mode HF download installs `huggingface_hub>=1.5` into the remote
runtime venv. huggingface_hub>=1.5 declares `click>=8.4` (core dep + typer),
which upgrades click past the runtime's `ray==2.9.3` ceiling. ray 2.9.3's CLI
crashes at import under click>=8.3 (deepcopy of click Sentinel values,
ray-project/ray#56747), so `ray status` exits non-zero, SkyPilot marks the
cluster "abnormal", and it gets stuck in INIT (CLI can no longer tail logs /
read the queue even though the job ran).

Pass a uv `--overrides` file pinning `click<8.3.0` when installing
huggingface_hub so click stays below the ray-breaking version while still
pulling huggingface_hub's other deps. The bucket API works fine with the
older click. A plain `click<8.3.0` constraint would be unsatisfiable against
the declared `>=8.4`, hence an override rather than a constraint.

Tested:
- Repro in a py3.10 venv: `ray[default]==2.9.3` + `huggingface_hub>=1.5`
  upgrades click to 8.4.1 and `ray status` crashes with the Sentinel error.
- With the override: click resolves to 8.2.1, `ray status` no longer crashes
  (reaches the connection step), and `huggingface_hub` + `HfApi().sync_bucket`
  import cleanly.
- Added a regression unit test; full HF suite: 84 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gemini-code-assist[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 4 additional findings.

Open in Devin Review

Address review on skypilot-org#9763:
- Write ~/.sky/hf_uv_override.txt atomically (mktemp + mv) and only if absent,
  so parallel COPY syncs running this install concurrently can't truncate the
  file while another's `uv pip install` reads it.
- Remove test_hf_hub_install_pins_click_for_ray: it only mirrored the command
  string (change-detector), not behavior.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zpoint zpoint requested a review from Michaelvll May 30, 2026 16:31
Comment thread sky/cloud_stores.py
Comment on lines +771 to +776
f'[ -f {_HF_UV_OVERRIDE_FILE} ] || '
f'(tmpfile=$(mktemp {_HF_UV_OVERRIDE_FILE}.XXXXXX) && '
f'printf "click<8.3.0\\n" > "$tmpfile" && '
f'mv -f "$tmpfile" {_HF_UV_OVERRIDE_FILE})',
f'{constants.SKY_UV_PIP_CMD} install '
f'--overrides {_HF_UV_OVERRIDE_FILE} "huggingface_hub>=1.5"',

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need an override file, instead of directly specifying in the uv pip install click<8.3.0?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uv pip install --overrides accepts a file path only — there is no --override "click<8.3.0" form (tested: error: File not found: 'click<8.3.0'). And a plain click<8.3.0 argument isn't an override at all — it's a constraint, which silently downgrades huggingface_hub to 1.16.1. So to override the declared click>=8.4, uv requires a file.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But that's a good point for simplicity. Updated to uv pip install huggingface_hub>=1.5 first, and then uv pip install click<8.3.0 to avoid the file issue.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separating into two install commands introduce a 1~2 seconds gap where click is 8.4.1 on disk.
click 8.3.0 -> 8.4.1 -> 8.3.0

Might need to revert to avoid race.

zpoint added a commit to zpoint/skypilot that referenced this pull request May 30, 2026
Replace the `uv pip install --overrides <file>` approach with a simpler second
`uv pip install "click<8.3.0"` after installing huggingface_hub, mirroring the
runtime's ray install line. This keeps huggingface_hub at the latest version
and downgrades only click (→ 8.2.1), with no override file, no atomic-write,
and no race to handle.

Addresses review on skypilot-org#9763 (why an override file): uv has no inline override
(--overrides takes a file only), and putting `click<8.3.0` on the same install
line silently downgrades huggingface_hub to an older release instead. A
standalone `uv pip install "click<8.3.0"` does a targeted in-place downgrade.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zpoint zpoint requested a review from Michaelvll May 30, 2026 16:56
devin-ai-integration[bot]

This comment was marked as resolved.

@zpoint zpoint force-pushed the fix/hf-click-pin-runtime branch 2 times, most recently from cb9ef55 to f3b8efe Compare May 31, 2026 03:54
Expand the comment on _GET_HF_HUB to explain why we cap click with a uv
--overrides file instead of (1) a same-line click<8.3.0 constraint (silently
downgrades huggingface_hub) or (2) a separate 'uv pip install click<8.3.0'
after huggingface_hub (briefly installs click 8.4.x, racing a concurrent
ray status -> cluster flips to INIT).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zpoint zpoint force-pushed the fix/hf-click-pin-runtime branch from ebb52b5 to 7e31e06 Compare May 31, 2026 14:01
@zpoint zpoint merged commit 11228f1 into skypilot-org:master Jun 1, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants