Can you please clarify what you call proprietary vs OSS driver?
Also, how did you create your guest image? Are you using the script build_custom_image.sh?
I'm surprised the Blackwell family isn't supported by the proprietary driver.

Can you please clarify what you call proprietary vs OSS driver?

Whenever you install the Nvidia drivers via nvidia's .run file, there are two options present in an ncurses window:

"Proprietary" / "MIT/GPL"

Hope that makes sense?

I'm surprised the Blackwell family isn't supported by the proprietary driver.

Yes it is surprising, however Nvidia appear to be only focusing on the MIT/GPL driver from the Blackwell family and onwards.

https://developer.nvidia.com/blog/nvidia-transitions-fully-towards-open-source-gpu-kernel-modules/

Also, how did you create your guest image? Are you using the script build_custom_image.sh?

Two ways, neither work with the 5060 Ti - both work with the 3090/3060 etc.

1. Custom Kernel built with PCI support, and container support - no cruft that doesn't belong in a microVM
2. Ubuntu's generic Kernel image that comes with the cloud images the booted up with Rust firmware explained here

For QEMU (which I've explained works with both cards with their respective OSS/proprietary drivers), this is most similar to 2. an Ubuntu cloud image with its generic Kernel.

In 1. all the Kernel drivers are loading, the issue appears to be pointing towards something in cloud-hypervisor's PCI support.

The card/OS/Kernel will:

• Load all nvidia drivers (shown on lsmod and dmesg)
• Actually load xorg onto the card and show it allocated via nvidia-smi (I see HDMI output)

But the core problem is that CUDA is broken for one reason or another within cloud hypervisor on this card. It also breaks in CH when using the "OSS" driver with the 3060.

Could the OSS driver be broken with CH in general?

A small tester program that would use CUDA fails:

cat > cuda_test.cu << 'EOF'
#include <cstdio>
#include <cuda_runtime.h>

int main() {
    int count = 0;
    cudaError_t err = cudaGetDeviceCount(&count);
    printf("cudaGetDeviceCount: %s (%d devices)\n",
           cudaGetErrorString(err), count);

    for (int i = 0; i < count; ++i) {
        cudaDeviceProp prop{};
        cudaError_t e2 = cudaGetDeviceProperties(&prop, i);
        if (e2 == cudaSuccess) {
            printf("Device %d: %s\n", i, prop.name);
        } else {
            printf("Device %d: error getting properties: %s\n",
                   i, cudaGetErrorString(e2));
        }
    }
    return 0;
}
EOF

nvcc cuda_test.cu -o cuda_test
./cuda_test

@sboeuf could you perhaps try nvidia.run on your own machine and use the MIT/GPL option?

This combo doesn't work with the 3090 and could reveal an issue with the driver or CH's PCI support where you can patch and explore unhindered?

Yes that would be a good thing to try since it would allow me to do some debugging if I was able to reproduce the issue on my machine. I'll give it a try when I get a chance!

(Looks like #7539 updates the README per the suggestion.)

are you aware of any development/testing in this area by the maintainers, or other users?

no I'm not