[Image] Bump GPU image to NVIDIA 580 open + CUDA 13 for Blackwell#9639
Conversation
The published skypilot:custom-gpu-ubuntu-2204 image installs NVIDIA driver 535 + CUDA 12.4 via this provisioner. That driver branch does not support Blackwell parts (B200 and the newly added RTX PRO 6000 Blackwell on G4), so `nvidia-smi` fails on those VMs at first boot when using the default image. Changes: - Install the 580-open driver. NVIDIA's open kernel module is the only flavor supported on Blackwell data-center GPUs (B200, RTX PRO 6000 Blackwell Server); the proprietary module is rejected. The open module also works on every previously supported GPU (Turing and later), so one image continues to cover the full matrix. - Bump CUDA toolkit to 13.0 and cuDNN to libcudnn9-cuda-13. This pairs with driver 580 per NVIDIA's compatibility matrix. - Install gcc-12 (+ matching g++-12 and linux-headers) and set it as the default compiler before driver install. The DKMS rebuild of the NVIDIA kernel module on GCP's 6.8.x kernels needs gcc >= 12 because the kernel Makefile passes `-ftrivial-auto-var-init=zero`, which gcc-11 (Ubuntu 22.04 default) does not understand. Without this the module silently fails to build and the image ships with no working driver. - Collapse the arm64 / x86 split: same driver/toolkit packages work on both architectures with the open module; only `nvidia-modprobe` is kept arm64-only (existing behavior). - Add `set -euxo pipefail` so a future silent failure in the build surfaces immediately instead of producing a broken image. This script is consumed by `sky/catalog/images/skypilot-gcp-gpu-ubuntu.pkr.hcl` (and the equivalent AWS / Azure builders). Maintainers need to rerun Packer in the `sky-dev-465` GCP project and ship the new image via a PR to `skypilot-org/skypilot-catalog` for users to see it. The script change alone has no runtime effect until that image is rebuilt. Tested: - `bash -n cuda.sh` syntax check passes. - This script (identical content) has been baked into a Blackwell image in our internal GCP project and used to launch RTX PRO 6000 (G4) workloads; `nvidia-smi` reports driver 580.xx, CUDA 13.0. Companion to skypilot-org#9634 ([GCP] Add RTX PRO 6000 Blackwell (G4) support), which adds the catalog/launch-path metadata. This PR addresses the image-content side; the two are independent and can be reviewed / merged in either order. Co-Authored-By: Arthur Vive <arthur.vive@spore.bio>
There was a problem hiding this comment.
Code Review
This pull request updates the CUDA provisioning script to support NVIDIA Blackwell GPUs by transitioning to the open-kernel-module driver (version 580) and CUDA 13.0. Key changes include adding gcc-12 for DKMS compatibility on newer kernels and unifying the installation path for x86_64 and arm64 architectures. Reviewers suggested explicitly installing nvidia-modprobe for all architectures to ensure device nodes are correctly created and performing an apt-get update before the initial package installations to prevent potential failures.
| if [ "$ARCH_PATH" = "arm64" ]; then | ||
| sudo apt install -y nvidia-driver-535 | ||
| sudo apt install -y nvidia-modprobe | ||
| sudo apt-get install -y cuda-toolkit-12-4 | ||
| sudo apt-get install libcudnn9-cuda-12 | ||
| sudo apt-get install libcudnn9-dev-cuda-12 | ||
| else | ||
| sudo apt-get install -y cuda-drivers-535 | ||
| sudo apt-get install -y cuda-toolkit-12-4 | ||
| # Install cuDNN | ||
| # https://docs.nvidia.com/deeplearning/cudnn/latest/installation/linux.html#installing-on-linux | ||
| sudo apt-get install libcudnn8 | ||
| sudo apt-get install libcudnn8-dev | ||
| sudo apt-get install -y nvidia-modprobe | ||
| fi |
There was a problem hiding this comment.
The nvidia-modprobe utility is required to ensure that NVIDIA device nodes (e.g., /dev/nvidia*) are correctly created, especially in headless environments. In the previous version, this was implicitly handled on x86_64 by the cuda-drivers metapackage. Since the script now uses the specific nvidia-driver-580-open package for both architectures, nvidia-modprobe should be installed explicitly for both x86_64 and arm64 to avoid potential 'device not found' errors when running GPU workloads.
| if [ "$ARCH_PATH" = "arm64" ]; then | |
| sudo apt install -y nvidia-driver-535 | |
| sudo apt install -y nvidia-modprobe | |
| sudo apt-get install -y cuda-toolkit-12-4 | |
| sudo apt-get install libcudnn9-cuda-12 | |
| sudo apt-get install libcudnn9-dev-cuda-12 | |
| else | |
| sudo apt-get install -y cuda-drivers-535 | |
| sudo apt-get install -y cuda-toolkit-12-4 | |
| # Install cuDNN | |
| # https://docs.nvidia.com/deeplearning/cudnn/latest/installation/linux.html#installing-on-linux | |
| sudo apt-get install libcudnn8 | |
| sudo apt-get install libcudnn8-dev | |
| sudo apt-get install -y nvidia-modprobe | |
| fi | |
| sudo apt-get install -y nvidia-modprobe |
There was a problem hiding this comment.
Done in bcba832 — moved nvidia-modprobe out of the arm64-only branch so it runs on both architectures, with a comment explaining why it's no longer pulled transitively.
| # default) does not support. Without this step the module silently fails to | ||
| # build and the resulting image has no working driver. Kernel headers must | ||
| # also be present so DKMS can build against the running kernel. | ||
| sudo apt-get install -y gcc-12 g++-12 "linux-headers-$(uname -r)" |
There was a problem hiding this comment.
The script attempts to install gcc-12 and kernel headers without first updating the package index. On a fresh system or in certain build environments, the local package cache might be empty or outdated, which can lead to installation failures. It is recommended to run sudo apt-get update before the first installation command.
| sudo apt-get install -y gcc-12 g++-12 "linux-headers-$(uname -r)" | |
| sudo apt-get update | |
| sudo apt-get install -y gcc-12 g++-12 "linux-headers-$(uname -r)" |
There was a problem hiding this comment.
Done in bcba832 — added sudo apt-get update before the gcc-12 / kernel-headers install.
DanielZhangQD
left a comment
There was a problem hiding this comment.
Thanks! @pauldonzier
| # The open module also works on every previously supported GPU (Turing and | ||
| # later), so a single image continues to cover all GPU types. | ||
| # | ||
| # Driver / toolkit pinning: NVIDIA 580 branch (open) + CUDA 13.0. |
There was a problem hiding this comment.
The Open kernel module may drop support for V100 / Pascal / Maxwell.
Do we need to keep support for those GPUs? cc @Michaelvll
There was a problem hiding this comment.
can we select the image based on the accelerators requested then?
There was a problem hiding this comment.
Confirmed and slightly worse than "may drop" — per NVIDIA's driver 580 README, the open kernel modules architecturally cannot support pre-Turing GPUs ("the open kernel modules depend on the GPU System Processor (GSP) first introduced in Turing"). So the regression covers Maxwell, Pascal (P100, P4), and Volta (V100) — V100 being the biggest practical concern since it's still widely used via GCP V100 SKUs, AWS p3.*, and Azure NCv3. The previous comment in this PR claiming "every previously supported GPU (Turing and later)" was inaccurate; corrected in e2ac2aa.
To @Michaelvll's question on selecting image by accelerator: yes, viable and there's already a precedent in the codebase — K80 routes to a separate _DEFAULT_GPU_K80_IMAGE_ID in sky/clouds/{gcp,aws,azure}.py. We can mirror that pattern with a _LEGACY_GPU_IMAGE_ID for V100/Pascal/Maxwell, backed by a second Packer-built image that keeps driver 535 + CUDA 12.4. The user-facing image_id schema doesn't need changes — the selection happens server-side when no image_id is passed.
Two questions on sequencing before I write the follow-up:
- Should I bundle everything into one coordinated PR (this driver bump + legacy
cuda.shflavor + cloud selection logic), or land this PR as-is with the understanding that maintainers will not rebuild/republishskypilot:custom-gpu-ubuntu-2204until the legacy-image PR lands? A safer middle option would be to publish the rebuilt Blackwell image under a new tag (e.g.skypilot:custom-gpu-ubuntu-2204-blackwell) and leave the existing tag pointing at the driver-535 image until the per-accelerator routing lands. - Are we OK keeping V100/Pascal on a frozen driver-535 image indefinitely (with a logged deprecation warning), or is there a preferred sunset timeline?
There was a problem hiding this comment.
For 1, we can merge the current PR first, then need to rebuild the images, and land the legacy-image PR.
For 2, I think it's OK to keep V100/Pascal on a frozen driver-535 image indefinitely.
|
Worth noting If you're open to it, retitling from |
|
Hi @pauldonzier, could you please help address the comments from Gemini? Everything else looks good to me. Thanks! |
…86_64 - Run `apt-get update` before installing gcc-12 / kernel headers so a stale or empty cache on fresh build hosts cannot fail the install. - Install `nvidia-modprobe` unconditionally. The previous x86_64 path relied on `cuda-drivers-535` pulling it transitively; the explicit `nvidia-driver-580-open` package does not depend on it, so the device nodes (`/dev/nvidia*`) would otherwise be missing on headless hosts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
| # Make sure CUDA toolkit and driver versions are compatible: | ||
| # https://docs.nvidia.com/deploy/cuda-compatibility/index.html | ||
| # Current State: Driver 580.x (open) + CUDA 13.0. | ||
| sudo apt-get install -y nvidia-driver-580-open |
There was a problem hiding this comment.
🔴 nvidia-driver-580-open breaks pre-Turing GPUs (V100, P100, P4) that still use this image
The switch from nvidia-driver-535 (proprietary) to nvidia-driver-580-open (open kernel module) will produce images with a non-functional NVIDIA driver on Volta and Pascal architecture GPUs. The PR's own comment at line 11-12 acknowledges this limitation: "The open module also works on every previously supported GPU (Turing and later)." However, V100 (Volta), P100 (Pascal), and P4 (Pascal) are all pre-Turing GPUs that are still actively listed in the SkyPilot catalog (sky/catalog/__init__.py:354, sky/catalog/gcp_catalog.py:156-175, sky/catalog/gcp_catalog.py:195-215) and are still routed to the default GPU image built by this script. Specifically, sky/clouds/gcp.py:607-609 sends all non-K80 GPUs (including V100, P100) to _DEFAULT_GPU_IMAGE_ID, and sky/clouds/aws.py:438-440 does the same for AWS (V100 on p3 instances). When images are rebuilt with this script, launching V100/P100/P4 instances will fail because the open kernel module cannot load on those architectures.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
yes, we are aware of this and discussing it in the other thread
The previous comment claimed the open kernel module "works on every previously supported GPU (Turing and later)" — this is inaccurate. Per NVIDIA's driver 580 README, the open modules depend on the GSP firmware first introduced in Turing, so Maxwell, Pascal (P100, P4), and Volta (V100) are explicitly unsupported. All three worked under the previous 535 proprietary driver, so this image rebuild drops support for them and needs a separate proprietary-driver image to keep covering those GPUs. Ref: https://download.nvidia.com/XFree86/Linux-x86_64/580.105.08/README/kernel_open.html Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sorry for the late reply, i was out of office. |
DanielZhangQD
left a comment
There was a problem hiding this comment.
LGTM! Thanks! @pauldonzier
…#9789) The default GPU image now installs the NVIDIA 580 open kernel module + CUDA 13 (PR #9639). The open module only supports Turing and later, so pre-Turing GPUs (V100, P100, P4, M60) must keep using the older proprietary-driver (CUDA 12) image. - Add sky/clouds/utils/gpu_utils.py as the single source of truth for which GPUs need the legacy image (LEGACY_DRIVER_GPUS). - AWS/GCP: point the default GPU image tag at the new cuda13 image and route pre-Turing GPUs to a new cuda12 image tag (the existing 535 image, reused unchanged). K80 keeps its own dedicated image. Existing catalog tags are untouched, so old clients are unaffected. - cuda.sh: use the SBSA CUDA repo for ARM (Graviton+GPU, GH200, GB200) instead of the Jetson/Tegra arm64 repo, which does not carry CUDA 13. New catalog rows for the cuda13 image must be shipped to skypilot-catalog before the new default tags resolve. Tested: - pytest tests/unit_tests/test_sky/clouds/{test_gpu_utils,test_gcp_cloud}.py and test_aws_cloud.py::TestGetDefaultAmi (28 new cases, all pass). - bash -n on cuda.sh; verified cuda-toolkit-13-0 exists under the sbsa CUDA repo and is absent from the arm64 repo. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary
Bump the GPU image provisioner from NVIDIA driver 535 + CUDA 12.4 to driver 580 (open kernel module) + CUDA 13, so Blackwell parts (B200 and the newly added RTX PRO 6000 Blackwell on G4) actually run on
skypilot:custom-gpu-ubuntu-2204.gcc-12(+ headers) before the driver: GCP's 6.8.x kernels need gcc ≥ 12 for the DKMS rebuild (kernel Makefile passes-ftrivial-auto-var-init=zero, gcc-11 doesn't support it). Without this, the module silently fails to build and the image ships with no working driver.nvidia-modprobestays arm64-only (existing behavior).set -euxo pipefailso future silent failures surface immediately.Companion to #9634 ([GCP] Add RTX PRO 6000 Blackwell (G4) support). #9634 adds the catalog/launch-path metadata; this PR fixes the image content. The two are independent and can be reviewed / merged in either order.
Note for maintainers: this script change has no runtime effect until someone runs
packer build skypilot-gcp-gpu-ubuntu.pkr.hclin thesky-dev-465GCP project and ships the new image via a PR toskypilot-org/skypilot-catalog(persky/catalog/images/README.md). I don't have permission to do either step.Tested:
bash -n cuda.shsyntax check passes.nvidia-smireports driver 580.x and CUDA 13.0.sky-dev-465.