[BUG]: PinnedMemoryResource rejects usable HOST_NUMA pools on CUDA 12.6-12.8 drivers

[carterbox]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Related but distinct: #2486 (the original unusable-buffer bug) and #2487 (the fix that introduced this regression).

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

#2487 added a capability pre-check before allocating a pinned host memory pool. On CUDA 12.6–12.8 drivers that check produces a false negative and rejects a HOST_NUMA pool that the driver creates and uses successfully.

This is a regression introduced in cuda.core 1.2.0 (released 2026-09-03, the first release containing #2487, which merged 2026-08-07). cuda.core 1.1.0 predates the pre-check and is unaffected.

The problem is that the capability attributes are newer than the capability itself:

Attribute	Ordinal	First defined in
CU_DEVICE_ATTRIBUTE_HOST_NUMA_MEMORY_POOLS_SUPPORTED	142	CUDA 12.9
CU_DEVICE_ATTRIBUTE_HOST_MEMORY_POOLS_SUPPORTED	144	CUDA 13.0

HOST_NUMA pools have been supported since CUDA 12.6 (see the cuMemPoolCreate docs

check_numa_pool.py

, which state that a host NUMA pool requires CUmemLocation::type = CU_MEM_LOCATION_TYPE_HOST_NUMA, and that CU_MEM_LOCATION_TYPE_HOST yields CUDA_ERROR_INVALID_VALUE).

On a 12.6–12.8 driver, querying ordinal 142 returns CUDA_ERROR_INVALID_VALUE. cuda.core surfaces that as a confident False rather than "unknown", so Device(0).properties.host_numa_memory_pools_supported returns False with no exception. The #2487 pre-check reads that False and raises. The net effect is that on drivers 12.6–12.8 there is currently no way to obtain a NUMA host pool through cuda.core.PinnedMemoryResource, even though the driver supports it.

All three constructor paths fail on such a driver, but only one is a defect:

Path	Result on CUDA 12.6	Assessment
options=None	RuntimeError: Getting the current memory pool requires CUDA 13.0 or later	deliberate version gate — correct
PinnedMemoryResourceOptions()	CUDAError: CUDA_ERROR_INVALID_VALUE	driver correctly rejects a HOST-location pool — correct
PinnedMemoryResourceOptions(numa_id=0)	RuntimeError: device does not support the requested host memory pool	this bug

This was verified against driver ground truth: a standalone C reproducer compiled against 12.6 headers, using only the raw driver API with no cuda.core in the path, creates that exact HOST_NUMA pool and round-trips verified data through it on the same machine. So the rejection originates in the cuda.core capability layer, not in the driver.

Probing by raw ordinal rather than by enum name is necessary to see this: on 12.6 headers the enum names do not exist, so a name-based probe cannot distinguish "the driver rejected the query" from "my headers are too old".

How to Reproduce

On a machine with a CUDA 12.6–12.8 driver:

from cuda.core import Device, PinnedMemoryResource, PinnedMemoryResourceOptions

dev = Device(0)
dev.set_current()

# False because the driver cannot answer the query, not because the
# capability is genuinely absent:
print(dev.properties.host_numa_memory_pools_supported)  # -> False

PinnedMemoryResource(options=PinnedMemoryResourceOptions(numa_id=0))
# RuntimeError: device does not support the requested host memory pool

The equivalent raw-driver sequence — cuMemPoolCreate with CUmemLocation::type = CU_MEM_LOCATION_TYPE_HOST_NUMA and id = 0 — succeeds on the same machine, and memory allocated from the resulting pool is usable and verifiable.

Confirmed on an NVIDIA A10M, driver 560.35.05 (cuDriverGetVersion -> 12060), cuda.core 1.2.0, cuda-bindings 12.9.7, cuda-version 12.6, Python 3.14.7 (conda-forge).

As a baseline, the same probes on a CUDA 13.2 driver report all attributes queryable and both pool kinds working, with no detection gap.

Expected behavior

An attribute that cannot be queried should be treated as unknown, not as unsupported.

On drivers where ordinal 142 is unqueryable but HOST_NUMA pools are documented as supported (12.6+), PinnedMemoryResource(options=PinnedMemoryResourceOptions(numa_id=...)) should either attempt the allocation and surface a genuine driver error if it fails, or determine the capability from the driver version rather than from an unanswerable attribute query. It should not pre-emptively reject a configuration the driver demonstrably supports.

More generally, it would help consumers if cuda.core distinguished "queried and got no" from "could not ask" — code that tries to tell those apart with try/except currently sees a confident False on precisely the drivers where that answer is wrong.

Operating System

Ubuntu Linux, kernel 6.8.0-139-generic (x86_64)

nvidia-smi output

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.05              Driver Version: 560.35.05      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A10M                    Off |   00000000:61:00.0 Off |                    0 |
|  0%   42C    P0             41W /  150W |       1MiB /  19190MiB |      3%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A10M                    Off |   00000000:DB:00.0 Off |                    0 |
|  0%   50C    P0             53W /  150W |       1MiB /  19190MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

[carterbox]

check_numa_pool.py

[carterbox]

/* Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
 *
 * SPDX-License-Identifier: Apache-2.0
 *
 * Probe whether HOST_NUMA stream-ordered memory pools actually work on this
 * CUDA stack, independently of whether the capability attribute is queryable.
 *
 * Why this exists
 * ---------------
 * CUDA 12.6's cuMemPoolCreate documents support for
 * CU_MEM_LOCATION_TYPE_HOST_NUMA pools and explicitly rejects
 * CU_MEM_LOCATION_TYPE_HOST with CUDA_ERROR_INVALID_VALUE. But the capability
 * attributes arrived later:
 *
 *   ordinal 142 CU_DEVICE_ATTRIBUTE_HOST_NUMA_MEMORY_POOLS_SUPPORTED -> CUDA 12.9
 *   ordinal 144 CU_DEVICE_ATTRIBUTE_HOST_MEMORY_POOLS_SUPPORTED      -> CUDA 13.0
 *
 * So on a 12.6-12.8 driver neither probe can be answered, and code that infers
 * "no host memory pools" from that gives up a NUMA pool that in fact works.
 * This program demonstrates that empirically.
 *
 * Attributes are probed by RAW ORDINAL, never by enum name: the names do not
 * exist in a 12.6 cuda.h, so this file must compile against it unchanged.
 *
 * Build (any CUDA >= 12.6 toolkit):
 *     gcc -O2 -o check_numa_pool check_numa_pool.c \
 *         -I/usr/local/cuda/include -L/usr/local/cuda/lib64/stubs -lcuda
 * or, if libcuda.so is already on the link path:
 *     gcc -O2 -o check_numa_pool check_numa_pool.c -lcuda
 *
 * Run:
 *     ./check_numa_pool
 *
 * Exit status: 0 = detection and reality agree, 1 = they disagree (bug present).
 */

#include <cuda.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/* Raw ordinals; see file header for why these are not the enum names. */
#define ATTR_HOST_NUMA_ID 134
#define ATTR_HOST_NUMA_MEMORY_POOLS_SUPPORTED 142
#define ATTR_HOST_MEMORY_POOLS_SUPPORTED 144

/* CUmemLocationType values, stable since CUDA 12.2. */
#define LOC_DEVICE 1
#define LOC_HOST 2
#define LOC_HOST_NUMA 3

#define NBYTES 4096
#define NELEM (NBYTES / sizeof(unsigned int))

static const char *err_name(CUresult rc)
{
    const char *s = NULL;
    if (cuGetErrorName(rc, &s) != CUDA_SUCCESS || s == NULL)
        return "UNKNOWN_CUresult";
    return s;
}

#define CHECK(call)                                                                   \
    do {                                                                              \
        CUresult _rc = (call);                                                        \
        if (_rc != CUDA_SUCCESS) {                                                    \
            fprintf(stderr, "%s:%d: %s failed: %s\n", __FILE__, __LINE__, #call,      \
                    err_name(_rc));                                                   \
            exit(2);                                                                  \
        }                                                                             \
    } while (0)

/* Query one attribute by ordinal. Returns 1 if queryable, storing the value. */
static int probe_attribute(int ordinal, const char *label, CUdevice dev, int *out)
{
    int value = 0;
    CUresult rc = cuDeviceGetAttribute(&value, (CUdevice_attribute)ordinal, dev);
    if (rc == CUDA_SUCCESS) {
        printf("  [%3d] %-42s = %d\n", ordinal, label, value);
        if (out)
            *out = value;
        return 1;
    }
    printf("  [%3d] %-42s ! UNQUERYABLE (%s)\n", ordinal, label, err_name(rc));
    return 0;
}

/* Create a pool at the given location, then prove host and device can both use it. */
static int try_pool(int loc_type, int loc_id, const char *label, CUdevice dev, CUstream stream)
{
    CUmemPoolProps props;
    CUmemoryPool pool = NULL;
    CUmemAccessDesc desc;
    CUdeviceptr ptr = 0, dev_buf = 0;
    unsigned int *host_view;
    size_t i;
    int ok = 0;
    CUresult rc;

    printf("\n--- %s ---\n", label);

    memset(&props, 0, sizeof(props));
    props.allocType = CU_MEM_ALLOCATION_TYPE_PINNED;
    props.handleTypes = CU_MEM_HANDLE_TYPE_NONE;
    props.location.type = (CUmemLocationType)loc_type;
    props.location.id = loc_id;
    props.maxSize = 0;

    rc = cuMemPoolCreate(&pool, &props);
    if (rc != CUDA_SUCCESS) {
        printf("  cuMemPoolCreate            -> %s   ==> NOT USABLE\n", err_name(rc));
        return 0;
    }
    printf("  cuMemPoolCreate            -> OK\n");

    /* Make the pool visible to the device, exactly as nvmath does. */
    memset(&desc, 0, sizeof(desc));
    desc.location.type = (CUmemLocationType)LOC_DEVICE;
    desc.location.id = dev;
    desc.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
    rc = cuMemPoolSetAccess(pool, &desc, 1);
    if (rc != CUDA_SUCCESS) {
        printf("  cuMemPoolSetAccess         -> %s   ==> NOT USABLE\n", err_name(rc));
        goto done;
    }
    printf("  cuMemPoolSetAccess         -> OK\n");

    rc = cuMemAllocFromPoolAsync(&ptr, NBYTES, pool, stream);
    if (rc != CUDA_SUCCESS) {
        printf("  cuMemAllocFromPoolAsync    -> %s   ==> NOT USABLE\n", err_name(rc));
        goto done;
    }
    CHECK(cuStreamSynchronize(stream));
    printf("  cuMemAllocFromPoolAsync    -> OK (ptr = 0x%llx)\n", (unsigned long long)ptr);

    /* The allocation must be directly addressable by the host. */
    host_view = (unsigned int *)(uintptr_t)ptr;
    for (i = 0; i < NELEM; ++i)
        host_view[i] = (unsigned int)i;
    printf("  host store through pointer -> OK\n");

    /* ...and readable/writable by the device. Round-trip via a real device buffer. */
    CHECK(cuMemAlloc(&dev_buf, NBYTES));
    CHECK(cuMemcpyAsync(dev_buf, ptr, NBYTES, stream));
    CHECK(cuStreamSynchronize(stream));
    memset(host_view, 0, NBYTES);
    CHECK(cuMemcpyAsync(ptr, dev_buf, NBYTES, stream));
    CHECK(cuStreamSynchronize(stream));
    for (i = 0; i < NELEM; ++i) {
        if (host_view[i] != (unsigned int)i) {
            printf("  device round-trip          -> DATA MISMATCH at index %zu\n", i);
            goto done;
        }
    }
    printf("  device round-trip          -> OK (data verified)\n");

    {
        unsigned int mem_type = 0;
        if (cuPointerGetAttribute(&mem_type, CU_POINTER_ATTRIBUTE_MEMORY_TYPE, ptr) == CUDA_SUCCESS)
            printf("  pointer memory type        -> %u (%u = CU_MEMORYTYPE_HOST)\n",
                   mem_type, (unsigned int)CU_MEMORYTYPE_HOST);
    }

    printf("  ==> USABLE\n");
    ok = 1;

done:
    if (dev_buf)
        cuMemFree(dev_buf);
    if (ptr) {
        cuMemFreeAsync(ptr, stream);
        cuStreamSynchronize(stream);
    }
    if (pool)
        cuMemPoolDestroy(pool);
    return ok;
}

int main(void)
{
    CUdevice dev;
    CUcontext ctx;
    CUstream stream;
    char name[256];
    int driver_version = 0;
    int numa_id = -1, numa_pools = 0, host_pools = 0;
    int numa_id_ok, numa_pools_ok, host_pools_ok;
    int numa_usable, host_usable;

    CHECK(cuInit(0));
    CHECK(cuDriverGetVersion(&driver_version));
    CHECK(cuDeviceGet(&dev, 0));
    CHECK(cuDeviceGetName(name, sizeof(name), dev));

    printf("=== environment ===\n");
    printf("  driver version : %d (%d.%d)\n", driver_version, driver_version / 1000,
           (driver_version % 1000) / 10);
    printf("  compiled with  : CUDA_VERSION %d\n", CUDA_VERSION);
    printf("  device 0       : %s\n", name);

    CHECK(cuDevicePrimaryCtxRetain(&ctx, dev));
    CHECK(cuCtxSetCurrent(ctx));
    CHECK(cuStreamCreate(&stream, CU_STREAM_NON_BLOCKING));

    printf("\n=== capability attributes (probed by raw ordinal) ===\n");
    numa_id_ok = probe_attribute(ATTR_HOST_NUMA_ID, "HOST_NUMA_ID", dev, &numa_id);
    numa_pools_ok = probe_attribute(ATTR_HOST_NUMA_MEMORY_POOLS_SUPPORTED,
                                    "HOST_NUMA_MEMORY_POOLS_SUPPORTED", dev, &numa_pools);
    host_pools_ok = probe_attribute(ATTR_HOST_MEMORY_POOLS_SUPPORTED,
                                    "HOST_MEMORY_POOLS_SUPPORTED", dev, &host_pools);

    if (!numa_id_ok || numa_id < 0) {
        printf("\n  NOTE: HOST_NUMA_ID unavailable or negative; using node 0 for the test.\n");
        numa_id = 0;
    }

    printf("\n=== raw driver pool tests ===\n");
    {
        char label[64];
        snprintf(label, sizeof(label), "HOST_NUMA pool (node %d)", numa_id);
        numa_usable = try_pool(LOC_HOST_NUMA, numa_id, label, dev, stream);
    }
    host_usable = try_pool(LOC_HOST, 0, "HOST pool (OS-managed)", dev, stream);

    printf("\n=== verdict ===\n");
    printf("  HOST_NUMA pool usable         : %s\n", numa_usable ? "yes" : "no");
    printf("  HOST (OS-managed) pool usable : %s\n", host_usable ? "yes" : "no");
    printf("  attribute 142 queryable       : %s\n", numa_pools_ok ? "yes" : "no");
    printf("  attribute 144 queryable       : %s\n", host_pools_ok ? "yes" : "no");
    printf("\n");

    cuStreamDestroy(stream);
    cuDevicePrimaryCtxRelease(dev);

    if (numa_usable && !numa_pools_ok) {
        printf("  >>> CONFIRMS THE BUG: a HOST_NUMA pool works here, but attribute 142\n");
        printf("      cannot be queried, so probing it is not a sound way to decide\n");
        printf("      whether NUMA host pools are available on this driver.\n");
        return 1;
    }
    if (!numa_usable && numa_pools_ok && numa_pools) {
        printf("  >>> Attribute 142 reports supported, but pool creation FAILED.\n");
        return 1;
    }
    printf("  Attribute probing and observed behaviour agree on this machine.\n");
    return 0;
}