fix(deps): preload `cusolver` if `cupy` is installed by gforsyth · Pull Request #1117 · rapidsai/cucim

gforsyth · 2026-07-09T21:41:22Z

I'm testing out smaller base images for wheel testing in #1116 and those tests flagged up that we're missing a cusolver dependency on wheel-only installs (the previous testing images have a full cuda toolkit installed beforehand).

The missing dependency turns out to be due to a cupy bug upstream with wheel-only installs (reported in cupy/cupy#10095).

This is a temporary workaround (if we want one) to enable switching to smaller test images (and also allowing end-users to use cucim on wheel-only installs)

xref rapidsai/build-planning#143
xref rapidsai/ci-imgs#400
xref rapidsai/shared-workflows#521

jameslamb

Left a blocking suggestion.

If we end up using Python code to pre-load cusolver after you consider this feedback then I think that's ok and I think where you chose is the right place.

But some other thoughts...

if some DSO in cucim needs libcusolver.so.* at runtime, why isn't auditwheel trying to vendor it? Or IS it making a copy? Worth checking, because I don't see it excluded here and would expect to:

cucim/ci/build_wheel.sh

Line 67 in 5c68f29

python -m auditwheel repair -w "${RAPIDS_WHEEL_BLD_OUTPUT_DIR}" dist/*

I don't think we need a cuda-pathfinder dependency. Could you look into just adding RPATH entries into site-packages/, like we do for RAFT's cusolver dependency?

https://github.com/NVIDIA/raft/blob/99ca82537c2263249018f6bfb53de3d31442f8ca/python/libraft/CMakeLists.txt#L40-L51

Normally when I jump into one of these packaging-codeowners reviewers where I wasn't assigned I remove the auto-assigned person, but leaving @KyleFromNVIDIA here because I think his thoughts on this topic would also be really helpful and possibly different from mine.

jameslamb · 2026-07-10T16:38:59Z

+      - output_types: [requirements, pyproject]
+        packages:
+          - cuda-toolkit[cusolver]
+          - cuda-pathfinder


As written, this would allow cucim-cu13 to be installed with cuda-toolkit==12.*. I don't think we want that.. We won't catch that in CI because we == pin cuda-toolkit as an additional constraint:

cucim/dependencies.yaml

Lines 185 to 204 in 5c68f29

- matrix:

cuda: "12.2"

use_cuda_wheels: "true"

packages:

- cuda-toolkit==12.2.*

- matrix:

cuda: "12.5"

use_cuda_wheels: "true"

packages:

- cuda-toolkit==12.5.*

- matrix:

cuda: "12.8"

use_cuda_wheels: "true"

packages:

- cuda-toolkit==12.8.*

- matrix:

cuda: "12.9"

use_cuda_wheels: "true"

packages:

- cuda-toolkit==12.9.*

... but that's a CI only thing.

We should ensure the wheel metadata provides a correct solve for users (CUDA 12 for -cu12, CUDA 13 for -cu13).

For this I think we want:

matrices by CUDA major version, with the dependency spelled cuda-toolkit[cusolver]==12.* and cuda-toolkit[cusolver]==13.*

usa_cuda_wheels guards, similar to https://github.com/rapidsai/cudf/blob/dda07069af24236c612dec623f471e946a6cdcfb/dependencies.yaml#L943-L962

The usa_cuda_wheels guards aren't strictly necessary since this project doesn't have devcontainers and isn't part of DLFW builds, but having the spelling be consistent with the rest of RAPIDS would make all-of-RAPIDS updates easier.

gforsyth · 2026-07-10T17:30:31Z

if some DSO in cucim needs libcusolver.so.* at runtime, why isn't auditwheel trying to vendor it? Or IS it making a copy? Worth checking, because I don't see it excluded here and would expect to:

This is a good question -- I took another look at the failures and the tracebacks a little more closely and this is looking less like a cucim problem and like maybe more of a cupy problem.

From this run on #1116: https://github.com/rapidsai/cucim/actions/runs/29044082057/job/86214270820?pr=1116

________ test_custom_eigvals_kernels_vs_linalg_eigvalsh[float32-shape1] ________
[gw4] linux -- Python 3.12.13 /pyenv/versions/3.12.13/bin/python
python/cucim/src/cucim/skimage/feature/tests/test_corner.py:351: in test_custom_eigvals_kernels_vs_linalg_eigvalsh
    evs1 = _reference_eigvals_computation(H)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
python/cucim/src/cucim/skimage/feature/tests/test_corner.py:338: in _reference_eigvals_computation
    eigs = cp.linalg.eigvalsh(matrices)[..., ::-1]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/pyenv/versions/3.12.13/lib/python3.12/site-packages/cupy/linalg/_eigenvalue.py:343: in eigvalsh
    import cupyx.cusolver
E   ImportError: libcusolver.so.12: cannot open shared object file: No such file or directory

gforsyth · 2026-07-10T17:32:34Z

like maybe more of a cupy problem.

or alternatively, not a problem, just a mismatch. We have all of this wheel loading logic that prefers wheel installations over system installations, but because there CAN be a system installation, we don't fail loudly when we're missing DSOs in wheels.

I wonder if all that's needed for cucim is adding cusolver to the test environment?

gforsyth · 2026-07-10T17:36:22Z

I wonder if all that's needed for cucim is adding cusolver to the test environment?

or potentially even simpler, should we be installing the cupy-cuda??x[ctk] extra?

gforsyth · 2026-07-10T18:12:39Z

or potentially even simpler, should we be installing the cupy-cuda??x[ctk] extra?

nope, we are already installing the ctk extra in the test runs

gforsyth · 2026-07-10T18:29:20Z

nope, we are already installing the ctk extra in the test runs

Ok, this is a cupy bug, reporting upstream, then we can decide what to do here

gforsyth · 2026-07-10T18:42:47Z

Reported upstream in cupy/cupy#10095, trying out temporary fixes involving cuda.pathfinder in #1116 (TLDR: I think we can assume cuda.pathfinder is installed without adding a new dependency here, because cupy ships it, so IF cupy is installed, we can attempt to preload libcusolver using pathfinder to avoid the load error)

gforsyth · 2026-07-10T18:44:06Z

Thanks for the careful review @jameslamb ! I wouldn't have thought I'd end up at cupy bug when I first saw the tracebacks in #1116

This reverts commit bf1bf22.

gforsyth · 2026-07-10T19:37:42Z

Tested these changes here (obviously) and also in #1116 to demonstrate that libcusolver gets correctly loaded from the wheel (since that's the only option in that image)

jameslamb

Thanks so much for looking into those things I linked, I support the implementation you ended up with.

I read (and subscribed to) the cupy issue, great repo! Thanks for doing that.

It's a nice side benefit of using a slimmer test image, that we can catch issues like this in CI 🙂

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

Co-authored-by: jakirkham <jakirkham@gmail.com>

jakirkham

Thanks Gil! 🙏

Have gone ahead and applied reasonable suggestions from here and the upstream issue. Hope that is ok.

LGTM

fix(deps): add explicit dependency for cusolver

bf1bf22

gforsyth requested review from a team as code owners July 9, 2026 21:41

gforsyth requested a review from KyleFromNVIDIA July 9, 2026 21:41

gforsyth mentioned this pull request Jul 9, 2026

Require only CUDA wheels (no system CTK) in pip packages rapidsai/build-planning#143

Open

gforsyth added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Jul 10, 2026

jameslamb requested changes Jul 10, 2026

View reviewed changes

gforsyth added 2 commits July 10, 2026 15:02

Revert "fix(deps): add explicit dependency for cusolver"

b266046

This reverts commit bf1bf22.

fix: preload cusolver if cupy is installed

964e076

gforsyth changed the title ~~fix(deps): add explicit dependency for cusolver~~ fix(deps): preload cusolver if cupy is installed Jul 10, 2026

chore: add more detail to comment on cuda.pathfinder handling

03a45db

bdice approved these changes Jul 10, 2026

View reviewed changes

Comment thread python/cucim/src/cucim/__init__.py Outdated

jameslamb approved these changes Jul 10, 2026

View reviewed changes

jakirkham changed the base branch from main to release/26.08 July 16, 2026 20:37

jakirkham added this to the v26.08.00 milestone Jul 16, 2026

jakirkham added this to cucim Jul 16, 2026

jakirkham and others added 2 commits July 16, 2026 13:39

Merge branch 'release/26.08' into fix_cusolver_dep

5714a89

Link upstream CuPy issue in workaround comment

c283b21

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

jakirkham reviewed Jul 16, 2026

View reviewed changes

Comment thread python/cucim/src/cucim/__init__.py Outdated

Incorporating Sebastian's simplified workaround

40bf3ec

Co-authored-by: jakirkham <jakirkham@gmail.com>

jakirkham approved these changes Jul 16, 2026

View reviewed changes

	- matrix:
	cuda: "12.2"
	use_cuda_wheels: "true"
	packages:
	- cuda-toolkit==12.2.*
	- matrix:
	cuda: "12.5"
	use_cuda_wheels: "true"
	packages:
	- cuda-toolkit==12.5.*
	- matrix:
	cuda: "12.8"
	use_cuda_wheels: "true"
	packages:
	- cuda-toolkit==12.8.*
	- matrix:
	cuda: "12.9"
	use_cuda_wheels: "true"
	packages:
	- cuda-toolkit==12.9.*

Uh oh!

Conversation

gforsyth commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jameslamb left a comment

Choose a reason for hiding this comment

Uh oh!

jameslamb Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

gforsyth commented Jul 10, 2026

Uh oh!

gforsyth commented Jul 10, 2026

Uh oh!

gforsyth commented Jul 10, 2026

Uh oh!

gforsyth commented Jul 10, 2026

Uh oh!

gforsyth commented Jul 10, 2026

Uh oh!

gforsyth commented Jul 10, 2026

Uh oh!

gforsyth commented Jul 10, 2026

Uh oh!

gforsyth commented Jul 10, 2026

Uh oh!

Uh oh!

jameslamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gforsyth commented Jul 9, 2026 •

edited

Loading