SkyPilot v0.13.0rc1: Sky Batch, Hugging Face Storage, Lifecycle Hooks, Cost Caps, Blackwell GPUs, and More

SkyPilot v0.13.0rc1 introduces Sky Batch for large-scale batch inference and data processing across worker pools, a native Hugging Face storage backend (hf://), a generalized lifecycle-hooks framework for running custom logic around cluster events, and the **sky debug-dump** command for one-shot troubleshooting. This release also adds per-instance cost caps, NVIDIA Blackwell / CUDA 13 images, and GKE Autopilot support.

Get it now with:

uv pip install "skypilot[all]==0.13.0rc1"

Or, upgrade your team SkyPilot API server:

NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.13.0rc1

helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
  --set apiService.image=berkeleyskypilot/skypilot:$VERSION \
  --version $VERSION --devel --reuse-values

Deprecations & Breaking Changes:

• **api_access renamed to api_server_access, now defaulting to True** (#9169). Tasks now automatically receive API server credentials. The legacy api_access key is still accepted for backward compatibility.
• **sky.jobs.queue(version=1) remains deprecated** and continues to emit a warning; use sky.jobs.queue(version=2), which returns richer job metadata as dictionaries (#9118).
• Removed all_includes_in_cluster from the SkyPilot config schema (#9671).

Highlights

[New] Sky Batch

**sky batch** brings large-scale batch inference and data processing to SkyPilot with a simple Python API (#8850, #9438, #9456). Given a dataset and a processing function, Sky Batch splits the data into batches, distributes them across a pool of GPU/CPU workers, and collects the results, with no manual sharding, provisioning, or failure handling. Workers are reused across jobs, so expensive setup (installing packages, downloading weights, loading models onto GPUs) happens only once.

import sky

# Point at your input data in cloud storage
ds = sky.batch.Dataset(source='s3://my-bucket/prompts/')

# Map a processing function over the dataset across a worker pool
ds.map(my_inference_fn, pool='my-gpu-pool', output='s3://my-bucket/results/')

Inputs and outputs can be read from and written to Amazon S3 (s3://) or Google Cloud Storage (gs://).

[New] Hugging Face Storage Backend

SkyPilot now supports Hugging Face as a first-class storage backend via the hf:// scheme (#9698, #9763). It wraps read-write Hugging Face Buckets (Xet-backed, S3-like object storage) and, through the same handle, read-only mounting of model, dataset, and space repos. Bucket lifecycle operations go through huggingface_hub, and MOUNT/MOUNT_CACHED modes use the FUSE-based hf-mount, consistent with SkyPilot's existing mount infrastructure.

file_mounts:
  # Read-write Hugging Face Bucket
  /data:
    source: hf://buckets/my-org/my-bucket
  # Read-only model / dataset / space repo
  /model:
    source: hf://meta-llama/Llama-3.1-8B
    mode: MOUNT

URL formats: hf://buckets/<ns>/<bucket> (bucket, read-write), hf://<ns>/<model>[@<rev>], hf://datasets/<ns>/<ds>, and hf://spaces/<ns>/<sp> (repos, read-only). Authentication reads HF_TOKEN from the environment or the Hugging Face CLI token file.

[New] Lifecycle Hooks Framework

The autostop hook introduced in v0.12 has grown into a generalized lifecycle-hooks framework via resources.hooks (#9064). Each hook declares a script and the events it fires on: autostop, preemption, and down. Use it to checkpoint state, sync experiment trackers, or send notifications around cluster lifecycle transitions.

resources:
  hooks:
    - run: |
        wandb sync
        curl -X POST $SLACK_WEBHOOK -d '{"text": "Cluster shutting down"}'
      events: [autostop, down]
      timeout: 300

The legacy autostop.hook YAML continues to work with a deprecation warning.

[New] sky debug-dump: One-Shot Troubleshooting

The new **sky debug-dump** command collects diagnostic information (logs, state, and configuration) for clusters, requests, and managed jobs into a single zip for troubleshooting and bug reports (#8868, #9752, #9753). Dumps are stored under ~/.sky/debug_dumps/ and automatically cross-link related resources (e.g. a cluster and its associated requests) so you capture complete context in one command.

sky debug-dump --cluster my-cluster

Cost Caps: Max Hourly Cost

You can now set a per-instance hourly budget cap with resources.max_hourly_cost (#9132, #9170). Only instances at or below the limit are considered during optimization. When use_spot is set, the cap applies against spot prices; otherwise against on-demand prices.

resources:
  accelerators: A100
  max_hourly_cost: 10.0

NVIDIA Blackwell & CUDA 13 Images

SkyPilot's default GPU image now ships NVIDIA driver 580 (open) + CUDA 13 for Blackwell support (#9639), with a legacy CUDA 12 image retained for pre-Turing GPUs and an arm64 build fix (#9789). RunPod adds RTX PRO Blackwell GPUs to its catalog (#9187).

GKE Autopilot Support

SkyPilot now detects GKE Autopilot clusters and trusts Node Auto-Provisioning (NAP) instead of walking existing node pools, so GPU requests provision correctly on Autopilot instead of being incorrectly rejected (#9565).

What's new: compared to v0.12.0

API Server

• RBAC viewer role for read-only dashboard/API access (#9681).
• Reconcile internal daemon rows on API server boot (#9668) and skip request env overlay for internal daemon requests (#9663).
• Bundle SSH credentials with the launch response (#9586).
• Refinements for graceful upgrade (#9317) and MIN_COMPATIBLE_API_VERSION bump for 0.12 clients (#9121).
• Reap multiprocess Prometheus files from dead/exited workers (#9689, #9690) and periodic scrape-timeout tuning (#9480, #9434).
• Return 404 for missing dashboard _next/ assets instead of index.html (#9722); don't cache error responses on /dashboard/_next paths (#9498).
• Fixed incorrect headers passed by oauth2-proxy (#9468) and oauth pod scheduling (#9288).
• Async engine improvements: hand asyncpg the libpq DSN via async_creator (#9483), put_async for the queue backend (#9346).
• Workspace controls: name-length limits (#9162), workspace permission checks for enabled_clouds and sky check (#9284), cached permission checks (#9125).
• Helm chart: devMode flag for in-cluster iteration (#9609), persist the whole sky_logs dir (#9454), optional server log mirror (#9754), configurable probes (#9477), disable session affinity on ingress (#9347), option to disable serving the server log (#9140).

Kubernetes

• GKE Autopilot support via autopilot.enabled detection and NAP trust (#9565).
• Per-workspace namespaces via context_configs (#9640).
• Container operations for clusters on Kubernetes (#9066).
• **ephemeral-storage requests/limits** on pods (#9053), allowed_nodes config for node-level filtering ([#9151](https://github.com/skypilot-org/skypilo...

SkyPilot v0.12.3.post1

Get it now with:

uv pip install "skypilot>=0.12.3.post1"

What's Changed

• Release 0.12.3.post1 by @Michaelvll in #9736

Full Changelog: v0.12.3rc1...v0.12.3.post1

SkyPilot v0.12.3

Get it now with:

uv pip install "skypilot>=0.12.3"

What's Changed

• Release 0.12.3 by @Michaelvll in #9708

Full Changelog: v0.12.3rc1...v0.12.3

SkyPilot v0.12.2.post1

Get it now with:

uv pip install "skypilot>=0.12.2.post1"

What's changed?

• Pin kubernetes<36 due to RBAC regression from the package

SkyPilot v0.12.2

Get it now with:

uv pip install "skypilot>=0.12.2"

Full Changelog: v0.12.1...v0.12.2

SkyPilot v0.12.1

Get it now with:

uv pip install "skypilot>=0.12.1"

Full Changelog: v0.12.0...v0.12.1

SkyPilot v0.12.0: Slurm Support, Job Groups for RL, Agent Skill, Recipes, Pool Autoscaling for Batch Inference, 7x Data Mounting, and More

SkyPilot v0.12.0 brings major new capabilities: Slurm integration for running SkyPilot on existing Slurm clusters, Job Groups for heterogeneous parallel workloads like RL training, an Agent Skill that teaches AI coding agents to use SkyPilot, Recipes for sharing reusable YAML templates across teams, and significant Pool enhancements including autoscaling. This release also drops Python 3.7/3.8, with Python 3.9+ now required.

Get it now with:

uv pip install "skypilot[all]>=0.12.0"

Or, upgrade your team SkyPilot API server:

NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.12.0

helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
  --set apiService.image=berkeleyskypilot/skypilot:$VERSION \
  --version $VERSION --devel --reuse-values

Deprecations & Breaking Changes:

• Dropped Python 3.7 and 3.8 support—Python 3.9+ is now required (#8489).
• sky.jobs.queue(version=1) is deprecated and will be removed in v0.13. Use sky.jobs.queue(version=2) instead. The new version returns richer job metadata as dictionaries (#9118).

Highlights

[New] Slurm Support

SkyPilot now supports connecting your Slurm clusters, bringing its unified interface to one of the most widely used job schedulers in high-performance computing (#5491, #8198, #8219, #8268, #8291, #8470, #8604, #8729, and 25+ additional PRs). Users can launch SkyPilot clusters and managed jobs on Slurm clusters with the same CLI and YAML they use for cloud and Kubernetes, enabling seamless workload portability across all AI infra.

Key capabilities include:

• Multi-node clusters with partition-level resource management
• Container support via pyxis/enroot for reproducible environments
• SSH ProxyJump for clusters behind bastion hosts
• GPU availability viewing with sky show-gpus for Slurm partitions
• Custom sbatch directives via sbatch_options in task YAML
• Configurable workdir and tmpdir for shared filesystem environments
• Admin policy support for Slurm partition routing

# Launch on a Slurm cluster
resources:
  accelerators: H100:8
  infra: slurm

# View GPU availability across Slurm clusters
sky show-gpus --infra slurm

# Launch a training job on Slurm
sky launch --infra slurm/my-cluster train.yaml

[New] Agent Skill: AI Agents Meet SkyPilot

SkyPilot now ships an official Agent Skill that teaches AI coding agents—Claude Code, Codex, and others—how to use SkyPilot (#8823, #9017, #9037). With the skill installed, your agent can launch clusters, run managed jobs, serve models, compare GPU pricing, and manage cloud resources—all through natural language.

Install the skill (docs) by telling your agent:

Fetch and follow the install guide https://github.com/skypilot-org/skypilot/blob/HEAD/agent/INSTALL.md

In our Scaling Autoresearch blog post, we gave Claude Code the SkyPilot agent skill and access to a 16-GPU Kubernetes cluster. Over 8 hours, the agent autonomously submitted ~910 experiments in parallel, achieving a 9x speedup over sequential search—and even discovered hardware-specific optimizations on its own.

Example interactions your agent can now handle:

[New] Job Groups: Heterogeneous Parallel Workloads

SkyPilot Job Groups let you define multiple tasks with different resource requirements that run together as a single managed job (#8456, #8664, #8686, #8688, #8713, #8940). This is ideal for reinforcement learning workflows where training, inference, and data servers need different hardware—H100s for policy training, cheaper GPUs for rollout inference, and high-memory CPUs for replay buffers.

SkyPilot provisions all resources together, configures networking automatically with built-in service discovery ({task_name}-{node_index}.{job_group_name}), and manages the full lifecycle as one unit.

# Multi-document YAML for a Job Group
---
name: rl-training
execution: parallel
primary_tasks: [ppo-trainer]

---
name: data-server
resources:
  cpus: 4+
run: |
  python data_server.py --port 8000

---
name: ppo-trainer
resources:
  accelerators: H100:1
run: |
  python ppo_trainer.py --data-server data-server-0.rl-training:8000

[New] Recipes: Shared YAML Registry

Recipes allow teams to store and share SkyPilot YAMLs in a centralized, team-accessible registry (#8755, #8825, #8851, #8873, #8876, #8882). Launch standardized workloads directly from the CLI or dashboard without local YAML files, reducing DevOps overhead and ensuring consistent configurations across the team.

Recipes support clusters, managed jobs, pools, and SkyServe, with built-in validation that blocks local file dependencies to ensure portability.

# Launch a recipe directly
sky launch recipes:dev-cluster

# Launch with custom overrides
sky launch recipes:gpu-cluster --cpus 16 --gpus H100:4 --env DATA_PATH=s3://my-data

Pool Autoscaling & Enhancements

SkyPilot Pools receive major upgrades with autoscaling, multiple jobs per worker, heterogeneous pools, and memory-aware scheduling (#8483, #8192, #8315, #8279, #8509, #7891).

• Autoscaling (#8483): Pools now automatically scale workers up and down (including to zero) based on queue length, maximizing GPU utilization while minimizing cost.
• Multiple jobs per worker (#8192): Workers can now run multiple concurrent jobs, improving resource utilization for smaller workloads.
• Heterogeneous pools (#8315): Pools can now contain workers with different resource configurations.
• Memory-aware scheduling (#8279): The pool scheduler now considers memory requirements when assigning jobs to workers.
• Fractional GPU improvements (#8509, #8480): Fixed fractional GPU support across multiple workers and corrected dashboard display.

# autoscaling-pool.yaml
pool:
  min_workers: 0
  max_workers: 10

resources:
  accelerators: H100

setup: |
  echo "Setup complete!"

Dashboa...

SkyPilot v0.11.2: Slurm Support, JobGroups, Enhanced Pools, External Links, Autostop Hooks, 7x data mount speed up and More

SkyPilot v0.11.2 delivers Slurm support in Beta, JobGroups for heterogeneous parallel workloads, and significantly enhanced Pools with autoscaling, multi-job scheduling and heterogeneous GPU support. This release also brings Autostop Hooks, 7x MOUNT_CACHED mode uploads speed up, automatic EFA on EKS, and numerous admin, security, and performance improvements.

Get it now with:

uv pip install "skypilot>=0.11.2"

Or, upgrade your team SkyPilot API server:

NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.11.2

helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
  --set apiService.image=berkeleyskypilot/skypilot:$VERSION \
  --version $VERSION --devel --reuse-values

Breaking Change: Python 3.9+ required — Python 3.7 and 3.8 are no longer supported (#8489). Please upgrade before installing this release.

Highlights

[Beta] Slurm Support

SkyPilot now supports Slurm as a new infrastructure backend, enabling users to orchestrate workloads on HPC clusters alongside cloud VMs and Kubernetes — all through the same unified interface (docs, #5491, #8138).

This release brings comprehensive Slurm capabilities:

• Multi-node distributed workloads — run distributed training jobs across multiple Slurm nodes with proper environment variable propagation and per-node logging (#8219)
• Containerized execution via NVIDIA pyxis/enroot — specify Docker images with --image-id for reproducible, GPU-accelerated workloads (#8604, #8609)
• Resource-scoped SSH sessions — ssh <cluster> drops you inside the Slurm job allocation, so nvidia-smi correctly reflects only your allocated GPUs (#8268)
• Interactive SSH authentication (2FA, password prompts) for clusters requiring keyboard-interactive auth (#8317)
• Partition support — Slurm partitions are mapped to SkyPilot zones, enabling partition-aware scheduling (#8198)
• Multi-cluster support — configure and use multiple Slurm clusters simultaneously
• Dashboard integration — Slurm clusters appear in the infrastructure page with status and GPU utilization

See the Slurm documentation for setup instructions.

JobGroups: Heterogeneous Parallel Workloads

JobGroups enable running multiple jobs with different resource requirements together as a managed group (blog, #8456). Define multi-task pipelines in a single multi-document YAML file:

# Header: job group metadata
name: rl-training
execution: parallel
primary_tasks: [trainer]
termination_delay: 30s

---
name: trainer
resources:
accelerators: A100:8
run: python train.py
---
name: reward-server
resources:
accelerators: A100:1
run: python reward_server.py

Key capabilities:

• Parallel cluster launch and monitoring — all jobs launched and tracked concurrently
• Inter-job networking — easy task name based hostname discovery (.) for job-to-job communication
• Dashboard UI — expandable rows for multi-task groups, task-specific log filtering
• CLI support — sky jobs logs --task-name <name> for viewing specific task logs

Example applications included: RL post-training (RLHF) pipeline and parallel train-eval pipeline.

Enhanced Pools: Multi-Job Scheduling and Heterogeneous GPUs

SkyPilot Pools receive significant upgrades in this release:

• Multiple jobs per worker — the scheduler now performs resource-aware bin-packing, tracking CPU, memory, and accelerator usage to fit multiple jobs on a single worker (#8192, #8279)
• Autoscaling — pools can now automatically scale workers up and down (including to zero) based on queue length. Specify min_workers, max_workers, and target queue length; a QueueLengthAutoscaler handles the rest while protecting running jobs from cancellation (#8483)
• Heterogeneous GPU support — specify any_of resource configurations and the scheduler dynamically resolves to available hardware (#8315):

resources:
any_of:
  - accelerators: T4:1
  - accelerators: A100:1

• 9x faster concurrent job launch — launching 100 concurrent jobs reduced from 4.5 minutes to 30 seconds using the new --num-jobs argument (#7891)
• Fractional GPU scheduling fix — fractional GPU jobs now correctly schedule across all workers (#8509)

External Links

SkyPilot dashboard now automatically detects your W&B links generated by your AI workloads. No need to dig into the job logs to figure out where your training panels are. (#8405)

Autostop Hooks

An autostop hook mechanism allow running custom scripts before a cluster is automatically stopped — for example, to save checkpoints, sync W&B runs, or send Slack notifications (#8412):

resources:
autostop:
idle_minutes:10
    hook:|
      wandb sync
      curl -X POST $SLACK_WEBHOOK -d '{"text": "Cluster shutting down"}'
hook_timeout:300

• New sky logs --autostop command to view hook execution logs
• sky exec is rejected on AUTOSTOPPING clusters; sky launch waits for autostop to complete before restarting

Automatic EFA Setup on Amazon EKS

SkyPilot now automatically configures Elastic Fabric Adapter (EFA) on EKS with a single flag (#8557):

resources:
  network_tier: best

This automates what was previously a complex manual setup, delivering ~78.8 GB/s inter-node bandwidth (vs ~4.1 GB/s without EFA), critical for distributed training performance. EFA interfaces are allocated proportionally to the requested GPU count.

7x MOUNT_CACHED Uploads Speed Up

Parallel uploads are now the default for MOUNT_CACHED file mounts, delivering a 7x speedup — flush time dropped from 151s to 21s for a ~14.6 GB test workload (#8455). A new data.mount_cached.sequential_upload config option allows reverting to sequential uploads if needed.

Exit Code-Based Job Recovery

Users can now specify exit codes that trigger automatic job recovery in managed jobs (#8324):

resources:
  job_recovery:
    recover_on_exit_codes:[29]

When a job exits with a specified code, SkyPilot automatically recovers it — useful for transient failures with known error codes.

Windows WSL Support

Automatically detect that SkyPilot is running in WSL, and seamless set up VSCode Remote-SSH for Windows users (#8669)

Admin Deployment Improvement

• External authentication proxy support — deploy behind AWS ALB with Cognito, Azure Front Door, or custom SSO proxies; supports both plaintext and JWT header formats (#8751)
• Sidecar container support — Istio, Datadog, and other sidecar injection no longer breaks K8s provisioning; SkyPilot explicitly targets the ray-node container (#8353, #8444)

What’s New

Kubernetes

• Improved Kueue integration — 24-hour provisioning timeout, workspace-level queue configuration, controller pod exclusion, and pod annotations (#8484)
• GPU detection fix — L40S no longer misidentified as L4; added Blackwell, newer Hopper, and Ada Lovelace GPUs (#8593)
• kubernetes.set_pod_resource_limits — set pod CPU/memory limits relative to requests for pod resource limit enforcement ([#8644](https://github.com/skypilot-org/skypilot/pul...

This patch release is a minor bump from v0.11.1 to to get you the latest fixes:

• Always set SSH key permission to avoid issue when jobs controller restarts by high availability setting #8316

Install the release candidate:

# Select needed clouds.
uv pip install 'skypilot[kubernetes,aws,gcp]==0.11.2rc1'

Upgrade your remote API server:

NAMESPACE=skypilot # TODO: change to your installed namespace
RELEASE_NAME=skypilot # TODO: change to your installed release name
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
  --version 0.11.2-rc.1 \
  --reset-then-reuse-values \
  --set apiService.image=null  # reset to default image

Full Changelog: v0.11.1...v0.11.2rc1

This patch release is a minor bump from v0.11.0 to to get you the latest fixes:

• #8285: Fix an issue where the API server request database is incorrectly locked causing an irresponsive API server with an error:

  sqlite3.OperationalError: database is locked
  

See the full v0.11 release notes for everything new in SkyPilot v0.11!