You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
cuda-core is built and run against several components that are versioned independently: its own source,
the cuda.h header, the cuda-bindings it is compiled against, the cuda-bindings installed at run time,
and the CUDA driver. Today it accepts almost any combination of them, including any cuda-bindings of the
right major, at either stage, and the cuda.h found under CUDA_PATH. Our CI pins the CUDA Toolkit to
match the bindings, but the build itself never checks that the two agree, and a source build gets whatever
the user's environment provides.
The documentation does not say which combinations we support, and neither does the code: builds succeed
for configurations we never test. Users have no way to tell a supported configuration from an unsupported
one, so some reasonably build the latest cuda-core against an older cuda-bindings and then report the
failure as a bug (#2699, #2700). Inside cuda-core, tolerating old bindings has led to layers of redundant
checks. Most feature gates (the run-time checks that turn a feature on or off) test both the driver version
and the bindings version. Some driver function pointers are optional. Some features call the slower
Python-object layer of cuda-bindings, known as lowpp, rather than the Cython layer, because a cimport of
a newer function could not be relied upon. And the C++ layer has its own compile-time guards. Version skew
is handled in three places (cuda-bindings, cuda-core's Cython, and cuda-core's C++), and none of them can
see what the others have decided. The result is worse than features being unavailable: features break. A
build against an older header silently compiles a feature out, and the run-time checks, which look at the
bindings and the driver rather than the header, never notice.
A related problem lies in how cuda-core's C++ layer calls the driver. It calls through cuda-bindings'
Cython wrappers, which were written for Cython callers. When the driver lacks a function, the wrapper
raises a Python exception that the C++ code never sees, and the exception surfaces later as a SystemError. This is not a versioning problem in itself, but it is part of how the two packages fit
together, and the changes below make a simpler design possible, so this issue addresses both.
The remedy for the versioning problem is to support only the configurations we actually build and test,
and to enforce that at build and import time. cuda-bindings already requires a cuda.h of its own
major.minor for its builds; cuda-core will adopt the same rule, so that the cuda.h it is compiled
against matches the cuda-bindings it is compiled against. It will also require a recent cuda-bindings, per
CUDA major, at both build and run time. When either requirement is not met, the build or the import fails
early with a clear message.
Those guarantees do most of the work. Once the build-time bindings are known to match the header and the
run-time bindings are at least as new, the only thing that can still be missing is a driver function, and
cuda-bindings already handles that in one place when it resolves driver symbols. Every other check becomes
redundant and can be deleted. New driver functions can be used through the fast Cython layer as soon as a
cuda-bindings release exposes them, instead of through the slower lowpp layer, so adding support for a CUDA
minor release no longer costs performance. Configurations we do not support fail at build or import, where
the failure is cheap to diagnose, instead of at the call site, where a feature quietly does the wrong
thing. And once the header is known to match the bindings, cuda-core can take its driver entry points
straight from the table that cuda-bindings resolves, which removes the wrapper problem as well.
Proposal: enforce a per-major cuda-bindings version floor, remove the code that exists only to tolerate
older bindings, and call the driver directly from C++. The changes are:
Floor. Each cuda-core release requires a minimum cuda-bindings version per CUDA major, at build
time and at run time. Older drivers stay supported.
Header rule. A source build requires a cuda.h of the same major.minor as the build-time
cuda-bindings.
Major-only C++ guards. The compile-time #if CUDA_VERSION guards in cuda/core/_cpp/ are
replaced by guards on CUDA_CORE_BUILD_MAJOR, which the build will define for the C++ compiler with -D. Today that name exists only as a Cython compile-time constant.
Direct driver calls. C++ calls the driver entry points that cuda-bindings resolved, not the Cython
wrappers. A pointer is either the driver's entry point or null because the driver does not provide the
function, and the version gates keep null pointers from ever being called.
Driver-only feature gates. Gates that test both the driver version and the bindings version test
the driver version only.
Problems in detail
The Summary describes the situation in general terms. This section records the specific failures we have
seen or can construct, since each one motivates one of the changes. One rule already exists on the
bindings side: cuda-bindings X.Y is built against cuda.h X.Y exactly, and it requires the same of anyone
who builds it. cuda-core has no comparable rule, and most of the failures below follow from that gap. The
last two follow from how cuda-core's C++ layer calls the driver.
Import fails with older cuda-bindings. Cython binds every cimported driver function when a module
initializes, so if the installed bindings lack even one of them, import cuda.core raises ImportError
before any version check can run. This was cuda-core 1.0 fails to import with cuda-bindings 13.0: cuDevSmResourceSplit not found #2063. About 200 call sites are bound this way.
Guards disable working features. In a source build, two 13.x features are compiled out when cuda.h is older than 13.1 or 13.2. Run such a build with newer bindings and a newer driver, and the
run-time checks pass while the feature fails with CUDA_ERROR_NOT_SUPPORTED.
Untested skew. CI pairs new wheels with cuda-bindings 13.0 and never with 13.1 or 13.2.
Missing pointers crash. The pointer loaders in _resource_handles.pyx are declared noexcept, so
a missing name becomes an "Exception ignored" warning, the loader stops, and the remaining pointers
stay null. The first use of one is a null-pointer crash.
One condition, two failure modes. Each C++ driver function pointer points at a cuda-bindings Cython
wrapper. A missing driver function therefore fails in two ways: the pointer is null when the bindings
are too old, or the wrapper raises FunctionNotFoundError when the driver is too old. In the second
case the wrapper sets a Python exception and returns a CUresult sentinel, and the C++ code, which does
not check for the exception, carries on. Python later reports a SystemError, or the exception surfaces
in unrelated code. Handle deleters ignore the return value entirely.
Policy
cuda-core supports two CUDA major versions at a time, currently 12 and 13, and each gets its own floor:
the newest cuda-bindings release of that major at the time of the cuda-core release, which is also the
version the wheels for that major are built against. A build belongs to one major series, determined by
the cuda-bindings and cuda.h it is compiled against, and every version check below applies within that
series only. A cu13 build is checked against the 13 floor and a cu12 build against the 12 floor. The wheel
we publish contains both builds, as cuda.core.cu12 and cuda.core.cu13 subpackages; at import,
cuda-core looks at the major of the installed cuda-bindings, selects the matching subpackage, and applies
that series' run-time check.
Build. cuda-core requires cuda-bindings >= floor and a cuda.h of the same major.minor as that
cuda-bindings. Anything else fails the build with a message that names the versions found and required.
Run.import cuda.core requires a cuda-bindings at least as new as the one the package was built
against (for our wheels, the floor), of the same major. Anything else raises ImportError with the pip command that fixes it.
Driver. Unchanged. For a given version of cuda-core, the driver alone decides whether a feature is
available.
Newer cuda-bindings than the build, same major. Supported, as today. This relies on cuda-bindings
keeping its Cython-layer names and signatures stable within a major, which its generator does today by
giving new ABI variants new names.
New cuda-bindings APIs. To use one, cuda-core bumps the floor in the same PR and records it under
"Breaking Changes" in the release notes.
support.rst currently promises that all minor versions within a major work with the same cuda-core
package. That remains true for drivers and toolkits at run time and becomes false for cuda-bindings older
than the floor. There is precedent: 0.4.0 dropped cuda-bindings below 12.6.2, and 0.6.0 raised the
source-build minimum to 12.9.0.
Implementation sketch
1. Floor. The floors live in one small module in the cuda-core source, one entry per CUDA major, and
everything else reads them from there: build_hooks.py, the import-time check in cuda/core/__init__.py,
the version helpers, and a test that keeps the pyproject.toml extras in step. The build backend enforces
the floor itself rather than trusting the pip build requirement, which conda-forge, pixi, and --no-build-isolation installs bypass. It also writes the cuda-bindings version it built against into the
package, so that at import cuda-core can require an installed cuda-bindings at least that new before it
selects the cu12 or cu13 subpackage. A failed check names the version found, the version required, and
the pip command that fixes it.
2. Header rule. The build backend reads CUDA_VERSION from the cuda.h it is about to compile
against and requires the header's major.minor to equal the cuda-bindings' major.minor. It does this even
when CUDA_CORE_BUILD_MAJOR is set in the environment, which today skips reading the header. The rule
exists because the driver-pointer keys in (4) are derived from that header's macros.
3. C++ guards. The C++ code stops testing CUDA_VERSION and tests CUDA_CORE_BUILD_MAJOR instead,
which the build defines for the C++ compiler with -D (today the name exists only as a Cython
compile-time constant). A new _cpp/versions.hpp, included first by every C++ file, checks the header
against the floor once and rejects a mismatch with #error. The #if CUDA_VERSION blocks, their void*
placeholder declarations, and the has_*() pointer-presence probes are deleted.
4. Direct driver calls. cuda-core takes its driver entry points from the table that cuda-bindings
builds with cuGetProcAddress, through _inspect_function_pointers() in cuda.bindings._internal.driver, an interface the cuda-bindings maintainers have agreed to keep stable.
cuda-core never loads the driver or resolves symbols itself. The pointer names are derived from cuda.h
macros at compile time, so cuStreamDestroy looks up __cuStreamDestroy_v2. Resolution is lazy: the
table is filled the first time cuda-core's C++ layer needs it, so import cuda.core never touches the
driver. At that point cuda-core checks once that every function older than the series' minimum driver is
present, and raises a clear error for an unsupported driver otherwise. Functions newer than that can be
null; the Cython driver-version gates guarantee such a pointer is never called, so C++ performs no
per-call checks. A symbol the bindings lack is an error, never a missing entry.
5. Gates. Double version checks (driver and bindings) become single checks (driver only). A symbol
that is missing from a whole major series of cuda-bindings is guarded with IF CUDA_CORE_BUILD_MAJOR
instead of a run-time check. Each driver threshold matches the version that cuda-bindings requests from cuGetProcAddress for that symbol. Several features call the driver through cuda.bindings.driver, the
lowpp layer, only because a cimport of a newer function could not be relied upon; they now cimport cydriver directly. That detour cost performance for no reason other than tolerating older bindings, and
it goes away.
Packaging and CI. The cu12 and cu13 extras pin cuda-bindings to the floor and the major, and the
conda-forge recipe pins the floor in its host requirement and the exact build-time version in its run
requirement. cuda-bindings stays out of the base pip dependencies because pip cannot know the user's CUDA
major. In CI, the rows that today pair a new wheel with cuda-bindings 13.0 pair it with the floor bindings
instead, keeping their older CUDA Toolkit libraries at run time. Those rows run the latest driver;
older-driver coverage comes from separate rows, and more of them are added. Two small tests cover the
failure paths: a unit test of the build backend's version check, and an import of the built wheel into an
environment with an older cuda-bindings that asserts the error message. The nightly row that runs an
older cuda-core against newer cuda-bindings stays.
User impact
Upgrading cuda-core may require upgrading cuda-bindings within the same major; pip install -U cuda-core[cu13] does both. Source builders need a toolkit and cuda-bindings of the same minor, at or
above the floor; #692, which would let isolated builds ignore the local toolkit, is the complementary
work. Several error messages change, and the release notes will list them.
Sequencing
A floor version must be published on PyPI and conda-forge before it can be set. It must also be reachable
from the cuda-bindings source roots that CI builds: today main for cu13 and the backport branch for
cu12, and after #2737 both roots on main. The structural work lands first, with the newest released
bindings as floors. Later PRs bump the floors as bindings releases ship.
Add versions.hpp and the -D macros, replace the CUDA_VERSION guards with CUDA_CORE_BUILD_MAJOR, and update _cpp/DESIGN.md.
Take the driver pointers from _inspect_function_pointers(): derive the names from cuda.h, fill
the table on first use, add the baseline check, and cover the NVRTC, NVVM, and nvJitLink pointers.
Remove the capsule loaders and the optional-pointer machinery. Reconcile the GIL contract in _cpp/DESIGN.md and AGENTS.md, which currently disagree.
Collapse the double version checks to driver-only checks, replace the lowpp fallbacks with direct cydriver calls, and update the affected tests.
Update the conda-forge cuda-core-feedstock requirements.
File cuda-bindings follow-ups: document _inspect_function_pointers() as stable, state the
within-major stability rule for the Cython layer's names and signatures, enforce the header rule in
its own build, and surface cuGetProcAddress's symbolStatus.
Summary
cuda-core is built and run against several components that are versioned independently: its own source,
the
cuda.hheader, the cuda-bindings it is compiled against, the cuda-bindings installed at run time,and the CUDA driver. Today it accepts almost any combination of them, including any cuda-bindings of the
right major, at either stage, and the
cuda.hfound underCUDA_PATH. Our CI pins the CUDA Toolkit tomatch the bindings, but the build itself never checks that the two agree, and a source build gets whatever
the user's environment provides.
The documentation does not say which combinations we support, and neither does the code: builds succeed
for configurations we never test. Users have no way to tell a supported configuration from an unsupported
one, so some reasonably build the latest cuda-core against an older cuda-bindings and then report the
failure as a bug (#2699, #2700). Inside cuda-core, tolerating old bindings has led to layers of redundant
checks. Most feature gates (the run-time checks that turn a feature on or off) test both the driver version
and the bindings version. Some driver function pointers are optional. Some features call the slower
Python-object layer of cuda-bindings, known as lowpp, rather than the Cython layer, because a
cimportofa newer function could not be relied upon. And the C++ layer has its own compile-time guards. Version skew
is handled in three places (cuda-bindings, cuda-core's Cython, and cuda-core's C++), and none of them can
see what the others have decided. The result is worse than features being unavailable: features break. A
build against an older header silently compiles a feature out, and the run-time checks, which look at the
bindings and the driver rather than the header, never notice.
A related problem lies in how cuda-core's C++ layer calls the driver. It calls through cuda-bindings'
Cython wrappers, which were written for Cython callers. When the driver lacks a function, the wrapper
raises a Python exception that the C++ code never sees, and the exception surfaces later as a
SystemError. This is not a versioning problem in itself, but it is part of how the two packages fittogether, and the changes below make a simpler design possible, so this issue addresses both.
The remedy for the versioning problem is to support only the configurations we actually build and test,
and to enforce that at build and import time. cuda-bindings already requires a
cuda.hof its ownmajor.minor for its builds; cuda-core will adopt the same rule, so that the
cuda.hit is compiledagainst matches the cuda-bindings it is compiled against. It will also require a recent cuda-bindings, per
CUDA major, at both build and run time. When either requirement is not met, the build or the import fails
early with a clear message.
Those guarantees do most of the work. Once the build-time bindings are known to match the header and the
run-time bindings are at least as new, the only thing that can still be missing is a driver function, and
cuda-bindings already handles that in one place when it resolves driver symbols. Every other check becomes
redundant and can be deleted. New driver functions can be used through the fast Cython layer as soon as a
cuda-bindings release exposes them, instead of through the slower lowpp layer, so adding support for a CUDA
minor release no longer costs performance. Configurations we do not support fail at build or import, where
the failure is cheap to diagnose, instead of at the call site, where a feature quietly does the wrong
thing. And once the header is known to match the bindings, cuda-core can take its driver entry points
straight from the table that cuda-bindings resolves, which removes the wrapper problem as well.
Proposal: enforce a per-major cuda-bindings version floor, remove the code that exists only to tolerate
older bindings, and call the driver directly from C++. The changes are:
time and at run time. Older drivers stay supported.
cuda.hof the same major.minor as the build-timecuda-bindings.
#if CUDA_VERSIONguards incuda/core/_cpp/arereplaced by guards on
CUDA_CORE_BUILD_MAJOR, which the build will define for the C++ compiler with-D. Today that name exists only as a Cython compile-time constant.wrappers. A pointer is either the driver's entry point or null because the driver does not provide the
function, and the version gates keep null pointers from ever being called.
the driver version only.
Problems in detail
The Summary describes the situation in general terms. This section records the specific failures we have
seen or can construct, since each one motivates one of the changes. One rule already exists on the
bindings side: cuda-bindings X.Y is built against
cuda.hX.Y exactly, and it requires the same of anyonewho builds it. cuda-core has no comparable rule, and most of the failures below follow from that gap. The
last two follow from how cuda-core's C++ layer calls the driver.
cimported driver function when a moduleinitializes, so if the installed bindings lack even one of them,
import cuda.coreraisesImportErrorbefore any version check can run. This was cuda-core 1.0 fails to import with cuda-bindings 13.0: cuDevSmResourceSplit not found #2063. About 200 call sites are bound this way.
support policy does not say whether the configuration is supported at all.
cuda.his older than 13.1 or 13.2. Run such a build with newer bindings and a newer driver, and therun-time checks pass while the feature fails with
CUDA_ERROR_NOT_SUPPORTED._resource_handles.pyxare declarednoexcept, soa missing name becomes an "Exception ignored" warning, the loader stops, and the remaining pointers
stay null. The first use of one is a null-pointer crash.
wrapper. A missing driver function therefore fails in two ways: the pointer is null when the bindings
are too old, or the wrapper raises
FunctionNotFoundErrorwhen the driver is too old. In the secondcase the wrapper sets a Python exception and returns a
CUresultsentinel, and the C++ code, which doesnot check for the exception, carries on. Python later reports a
SystemError, or the exception surfacesin unrelated code. Handle deleters ignore the return value entirely.
Policy
cuda-core supports two CUDA major versions at a time, currently 12 and 13, and each gets its own floor:
the newest cuda-bindings release of that major at the time of the cuda-core release, which is also the
version the wheels for that major are built against. A build belongs to one major series, determined by
the cuda-bindings and
cuda.hit is compiled against, and every version check below applies within thatseries only. A cu13 build is checked against the 13 floor and a cu12 build against the 12 floor. The wheel
we publish contains both builds, as
cuda.core.cu12andcuda.core.cu13subpackages; at import,cuda-core looks at the major of the installed cuda-bindings, selects the matching subpackage, and applies
that series' run-time check.
cuda-bindings >= floorand acuda.hof the same major.minor as thatcuda-bindings. Anything else fails the build with a message that names the versions found and required.
import cuda.corerequires a cuda-bindings at least as new as the one the package was builtagainst (for our wheels, the floor), of the same major. Anything else raises
ImportErrorwith thepipcommand that fixes it.available.
keeping its Cython-layer names and signatures stable within a major, which its generator does today by
giving new ABI variants new names.
"Breaking Changes" in the release notes.
support.rstcurrently promises that all minor versions within a major work with the same cuda-corepackage. That remains true for drivers and toolkits at run time and becomes false for cuda-bindings older
than the floor. There is precedent: 0.4.0 dropped cuda-bindings below 12.6.2, and 0.6.0 raised the
source-build minimum to 12.9.0.
Implementation sketch
1. Floor. The floors live in one small module in the cuda-core source, one entry per CUDA major, and
everything else reads them from there:
build_hooks.py, the import-time check incuda/core/__init__.py,the version helpers, and a test that keeps the
pyproject.tomlextras in step. The build backend enforcesthe floor itself rather than trusting the pip build requirement, which conda-forge, pixi, and
--no-build-isolationinstalls bypass. It also writes the cuda-bindings version it built against into thepackage, so that at import cuda-core can require an installed cuda-bindings at least that new before it
selects the cu12 or cu13 subpackage. A failed check names the version found, the version required, and
the
pipcommand that fixes it.2. Header rule. The build backend reads
CUDA_VERSIONfrom thecuda.hit is about to compileagainst and requires the header's major.minor to equal the cuda-bindings' major.minor. It does this even
when
CUDA_CORE_BUILD_MAJORis set in the environment, which today skips reading the header. The ruleexists because the driver-pointer keys in (4) are derived from that header's macros.
3. C++ guards. The C++ code stops testing
CUDA_VERSIONand testsCUDA_CORE_BUILD_MAJORinstead,which the build defines for the C++ compiler with
-D(today the name exists only as a Cythoncompile-time constant). A new
_cpp/versions.hpp, included first by every C++ file, checks the headeragainst the floor once and rejects a mismatch with
#error. The#if CUDA_VERSIONblocks, theirvoid*placeholder declarations, and the
has_*()pointer-presence probes are deleted.4. Direct driver calls. cuda-core takes its driver entry points from the table that cuda-bindings
builds with
cuGetProcAddress, through_inspect_function_pointers()incuda.bindings._internal.driver, an interface the cuda-bindings maintainers have agreed to keep stable.cuda-core never loads the driver or resolves symbols itself. The pointer names are derived from
cuda.hmacros at compile time, so
cuStreamDestroylooks up__cuStreamDestroy_v2. Resolution is lazy: thetable is filled the first time cuda-core's C++ layer needs it, so
import cuda.corenever touches thedriver. At that point cuda-core checks once that every function older than the series' minimum driver is
present, and raises a clear error for an unsupported driver otherwise. Functions newer than that can be
null; the Cython driver-version gates guarantee such a pointer is never called, so C++ performs no
per-call checks. A symbol the bindings lack is an error, never a missing entry.
5. Gates. Double version checks (driver and bindings) become single checks (driver only). A symbol
that is missing from a whole major series of cuda-bindings is guarded with
IF CUDA_CORE_BUILD_MAJORinstead of a run-time check. Each driver threshold matches the version that cuda-bindings requests from
cuGetProcAddressfor that symbol. Several features call the driver throughcuda.bindings.driver, thelowpp layer, only because a
cimportof a newer function could not be relied upon; they nowcimportcydriverdirectly. That detour cost performance for no reason other than tolerating older bindings, andit goes away.
Packaging and CI. The
cu12andcu13extras pin cuda-bindings to the floor and the major, and theconda-forge recipe pins the floor in its host requirement and the exact build-time version in its run
requirement. cuda-bindings stays out of the base pip dependencies because pip cannot know the user's CUDA
major. In CI, the rows that today pair a new wheel with cuda-bindings 13.0 pair it with the floor bindings
instead, keeping their older CUDA Toolkit libraries at run time. Those rows run the latest driver;
older-driver coverage comes from separate rows, and more of them are added. Two small tests cover the
failure paths: a unit test of the build backend's version check, and an import of the built wheel into an
environment with an older cuda-bindings that asserts the error message. The nightly row that runs an
older cuda-core against newer cuda-bindings stays.
User impact
Upgrading cuda-core may require upgrading cuda-bindings within the same major;
pip install -U cuda-core[cu13]does both. Source builders need a toolkit and cuda-bindings of the same minor, at orabove the floor; #692, which would let isolated builds ignore the local toolkit, is the complementary
work. Several error messages change, and the release notes will list them.
Sequencing
A floor version must be published on PyPI and conda-forge before it can be set. It must also be reachable
from the cuda-bindings source roots that CI builds: today
mainfor cu13 and the backport branch forcu12, and after #2737 both roots on
main. The structural work lands first, with the newest releasedbindings as floors. Later PRs bump the floors as bindings releases ship.
Tasks
the extras, the CI matrix changes, and the two small tests for the failure paths. Update
support.rst,install.rst, the per-feature minima, and the release notes. Close [BUG]: cuda.core fails to build from source against cuda-bindings 13.0.x #2699 and declinecuda.core: fix build compatibility with cuda-bindings 13.0.x #2700.
versions.hppand the-Dmacros, replace theCUDA_VERSIONguards withCUDA_CORE_BUILD_MAJOR, and update_cpp/DESIGN.md._inspect_function_pointers(): derive the names fromcuda.h, fillthe table on first use, add the baseline check, and cover the NVRTC, NVVM, and nvJitLink pointers.
Remove the capsule loaders and the optional-pointer machinery. Reconcile the GIL contract in
_cpp/DESIGN.mdandAGENTS.md, which currently disagree.cydrivercalls, and update the affected tests.cuda-core-feedstockrequirements._inspect_function_pointers()as stable, state thewithin-major stability rule for the Cython layer's names and signatures, enforce the header rule in
its own build, and surface
cuGetProcAddress'ssymbolStatus.References
_has_cuGraphNodeGetParamscheck missed the binding version check #2052, Fix missing binding version checks alongside driver version checks #2054, Standardize internal version checks in cuda.core #1825: dual version gates.cuda-bindingsbuildable against CTK wheels? #692: build isolation.cuda.bindingsABI stability #1030: ABI stability tests.std::abort#2758: error policy.main.