feat(os/nvidia): derive NVIDIA module options from the GPU topology - #1157
Conversation
The image ships one static /etc/modprobe.d/nvidia.conf that unconditionally
sets RmEnableProtectedPcie=0x1. Protected PCIe is the Hopper multi-GPU
confidential mode; Blackwell multi-GPU uses MPT CC instead, where Fabric
Manager stays on the host and the CC provisioning flow explicitly sets
ppcie-mode=off on the GPUs. Enabling PPCIe on those tenants contradicts the
mode they were provisioned in.
The opposite gap exists for a single-GPU tenant. With no fabric behind it the
NVLink fabric-probe state machine arms and blocks on a Fabric Manager that is
never going to route anything, surfacing as NV_ERR_NVLINK_FABRIC_NOT_READY
(802). That needs NVreg_NvLinkDisable=1, which must not be set when the tenant
has several GPUs because it would disable the NVLink the workload runs on.
Neither can be baked in, and neither belongs on the kernel command line: that
string is measured, so a per-shape token would fork the attestation baseline
once per topology. Generate the options at boot instead, from what is actually
visible on the guest PCI bus:
NVSwitch present -> Protected PCIe. The VMM only hands NVSwitches to
the guest in PPCIe mode (vmm-cli --ppcie attaches
all GPUs *and* NVSwitches), so an NVSwitch in the
guest is exactly that topology.
exactly one GPU, no NVSwitch -> NVreg_NvLinkDisable=1.
otherwise -> no options, which is Blackwell MPT CC.
Reading the live topology rather than a device-ID table means new GPU
generations need no change here.
nvidia-gpu-detect grows a count-gpus query alongside its existing
ExecCondition modes, and the generated file lands in /run/modprobe.d because
the rootfs is dm-verity protected. Removing the static conf keeps a single
source of `options nvidia`.
os/yocto is excluded from the repository shellcheck and formatting hooks, so
the detection was verified by hand against a synthetic PCI tree: GPU-less
host, 1 GPU, 2 and 8 GPUs without NVSwitch, 8 GPUs with 4 NVSwitches, a VGA
plus 3D-controller pair counting as two GPUs, and a non-NVIDIA bridge not
being mistaken for an NVSwitch. The sysfs root and the output directory are
overridable for that purpose only; the unit sets neither.
There was a problem hiding this comment.
🟡 Changes recommended
The new oneshot unit’s Before= ordering does not guarantee it will be started when the NVIDIA services are started directly, so module options may not be generated before modprobing in some start paths.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates the Yocto NVIDIA guest-image layer (os/yocto/layers/meta-nvidia) to generate NVIDIA kernel module options at boot based on the guest-visible PCI topology, avoiding a single baked /etc/modprobe.d/nvidia.conf that is incorrect for some GPU shapes (single-GPU vs Hopper PPCIe vs Blackwell MPT CC).
Changes:
- Replace the static
options nvidia ...config with a boot-time generator that writes/run/modprobe.d/nvidia-dstack.conf. - Add a new systemd oneshot unit to run the generator before NVIDIA userspace services that modprobe the driver.
- Extend
nvidia-gpu-detectwith acount-gpusquery mode and make its sysfs root overridable for synthetic testing.
File summaries
| File | Description |
|---|---|
| os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/nvidia-modprobe-config_1.0.bb | Switch package contents from static modprobe config to a generator script + systemd unit, and add dependency on nvidia-gpu-detect. |
| os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia.conf | Remove the static options nvidia configuration. |
| os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-module-options.service | Add a oneshot unit intended to run before NVIDIA services that modprobe modules. |
| os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-module-options | New boot-time script that derives and writes the appropriate modprobe options nvidia ... line to /run/modprobe.d. |
| os/yocto/layers/meta-nvidia/recipes-graphics/nvidia/files/nvidia-gpu-detect | Add count-gpus mode and allow overriding the PCI sysfs root for testability. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| [Install] | ||
| WantedBy=multi-user.target |
There was a problem hiding this comment.
Correct on the mechanism, and applied — thanks. Before= is ordering and not a dependency, so starting either service out of band would have ordered against a unit that was never queued. Both are now in [Install] WantedBy= alongside sysinit.target, and the test asserts the exact set so it cannot silently regress.
Two notes for the record.
The review was written against the first revision of this unit, which had WantedBy=multi-user.target. It has since moved to sysinit.target with DefaultDependencies=no, because the premise both the unit and this suggestion rested on — that the two services' ExecStartPre=modprobe nvidia are the only load sites — was wrong. nvidia.ko and nvidia-drm.ko carry PCI aliases matching every passed-through GPU and the image ships systemd's 80-drivers.rules, so udev coldplug was loading the driver during systemd-udev-trigger.service, long before either service. Adding the services to WantedBy= would not have helped there: by the time either starts, the driver is already up and the options are ignored.
So the suggestion closes a real but narrow gap (a start after the boot run failed), while the load-ordering problem was elsewhere. Both are fixed now — the udev path via a blacklist for nvidia and nvidia-drm, this one via WantedBy=.
The shapes that matter -- a Hopper box with NVSwitches passed through, a Blackwell tenant with eight GPUs and none -- are not available on a development machine, so the PCI topology is faked instead. A directory of vendor/class files is bind-mounted over /sys/bus/pci/devices inside a mount namespace and the scripts run from their installed paths with no environment overrides, which exercises the same code the image runs including the hardcoded /usr/bin/nvidia-gpu-detect and /run/modprobe.d. Where user namespaces are unavailable the matrix falls back to the overrides, covering the decision logic but not the path wiring. Beyond the topology cases the suite asserts two things the generator quietly depends on: that modprobe really reads /run/modprobe.d, which is a compile-time list in kmod rather than something the script can influence, and that every unit in this layer which modprobes nvidia is named in the generator's Before=. The second is the one that rots -- a new load site added elsewhere would race the generator and silently get the wrong options.
The generator was ordered only ahead of nvidia-persistenced.service and
nvidia-fabricmanager.service, on the assumption that their ExecStartPre
modprobe calls were the only place the image loads the driver. They are not.
nvidia.ko carries PCI aliases -- pci:v000010DEd*sv*sd*bc03sc02i00* matches
every passed-through GPU, and both H200 (10de:2335) and B300 (10de:3182)
resolve through it -- and the image ships systemd's 80-drivers.rules with
`ENV{MODALIAS}=="?*", RUN{builtin}+="kmod load"`. Coldplug therefore loads
nvidia during systemd-udev-trigger.service, in sysinit.target, long before
multi-user.target. Options written after the driver has initialised are
ignored, so the generated file never took effect on a GPU instance. The
static file it replaces did not have this problem precisely because it was
on disk from boot.
Keeping the modules out of modules-load.d is not the same as keeping them
from being autoloaded; it only stops the unconditional load on GPU-less
hosts.
Move the unit into early boot with DefaultDependencies=no and order it ahead
of systemd-udev-trigger.service. PID1 mounts /run before any unit runs and
the kernel finishes PCI enumeration before userspace starts, so both the
output directory and the sysfs topology are available there. The two NVIDIA
services stay in Before= because they still modprobe explicitly.
The load-site test now expects systemd-udev-trigger.service in Before= as
well, and asserts the unit keeps DefaultDependencies=no and
WantedBy=sysinit.target -- dropping either would make the udev ordering
unsatisfiable and silently reintroduce the race.
Correction: the first version ran too latePushed The unit was ordered only ahead of
Both H200 ( So coldplug loads Worth noting for the record: keeping the NVIDIA modules out of FixThe unit moves into early boot: PID1 mounts Re-verifiedOrdering was re-checked by loading the real unit into systemd with only (The relative order of the last three is an artefact of the user-session harness; in the real system The test suite grew two guards for this: |
…oload Three related gaps, all of which let the wrong module options reach a tenant. Stop udev autoloading the driver. `blacklist` suppresses loading by alias while leaving an explicit `modprobe nvidia` working, which is what the two NVIDIA services do from ExecStartPre, so this makes those services the only load sites and the generator's ordering sufficient rather than merely likely. nvidia-drm has to be blacklisted alongside nvidia: it carries the same three PCI aliases and depends on nvidia-modeset, which depends on nvidia, so blacklisting nvidia alone is bypassed -- `modprobe -n -v` against the built module tree loads all three from the B300 modalias with only nvidia blacklisted, and nothing with both listed. Pull the generator in, not just order against it. Before= is ordering and not a dependency, so starting either NVIDIA service out of band would previously order against a unit that was never queued. Both services now appear in WantedBy= as well. Thanks to the Copilot review for catching this. Share the payload instead of reaching across backends. The mkosi backend installed six files by absolute path out of the yocto layer, which is how it came to install nvidia.conf -- a modprobe.d `options` line -- into /etc/modules-load.d, where it never applied the PPCIe setting and only made systemd-modules-load look for a module named "options". The files both backends stage now live in os/common/nvidia, following the convention os/common/README.md already sets for the shared rootfs payload, with the yocto recipes reaching them through FILESEXTRAPATHS. mkosi stages the two new files and the blacklist, its preset enables the generator, and parity.json lists all three so the backend comparison covers them. The test suite grows a backend parity section asserting that both backends stage every file in os/common/nvidia and that the mkosi preset enables the generator, which is the check that would have caught the modules-load.d mistake.
Update: backend parity, udev blacklist, shared payload (
|
| config | result |
|---|---|
| no blacklist | loads nvidia, nvidia-modeset, nvidia-drm |
only blacklist nvidia |
still loads all three |
blacklist nvidia + blacklist nvidia-drm |
loads nothing |
both blacklisted, explicit modprobe nvidia |
loads normally |
nvidia-modeset, nvidia-uvm and nvidia-peermem carry no aliases, so they need no entry.
The early-boot ordering from 98757fd stays as a second belt: if the blacklist is ever overridden, the generator still wins the race.
Verification
Full suite, 19 checks:
topology matrix (mount namespace, installed paths, no overrides) 10 ok
modprobe integration 1 ok
module load sites 3 ok
udev autoload blacklist 3 ok
backend parity 3 ok
The parity section is new and is exactly the check that would have caught the modules-load.d mistake: it asserts both backends stage every file in os/common/nvidia, and that the mkosi preset enables the generator.
The yocto side was rebuilt after the move to confirm FILESEXTRAPATHS resolves — bitbake -c package on nvidia-gpu-detect, nvidia-persistenced, nvidia-modprobe-config and nvidia-fabricmanager all succeed, and nvidia-modprobe-config packages:
-rwxr-xr-x /usr/bin/nvidia-module-options
-rw-r--r-- /usr/lib/systemd/system/nvidia-module-options.service
-rw-r--r-- /usr/lib/modprobe.d/nvidia-blacklist.conf
-rw-r--r-- /usr/lib/systemd/system-preset/98-nvidia-modprobe-config.preset
Left alone deliberately
The fabricmanager drop-in goes to /usr/lib/systemd/system/... on yocto and /etc/systemd/system/... on mkosi. That divergence predates this PR and parity.json encodes the etc path, so changing it is a separate cleanup.
The drop-in went to /etc/systemd/system/nvidia-fabricmanager.service.d on mkosi and to /usr/lib/systemd/system/... on yocto. /usr/lib is the correct one: /etc is the administrator's layer, and systemctl(1) is explicit that `systemctl revert` removes <unit>.d/ below /etc and /run wholesale. A perfectly ordinary revert would therefore delete the ExecCondition=nvidia-gpu-detect guard, and the fabric manager would start failing rather than skipping on instances without an NVSwitch -- which is the whole reason one image can serve GPU and GPU-less hosts. Nothing is lost by moving it. Drop-ins from every directory are applied in lexical order by filename, so `systemctl edit` still wins with its override.conf; the difference is only that the image stops squatting on the namespace reserved for the operator. There is no measurement asymmetry to weigh here: dstack overlays /etc, /usr and /bin alike, with the measured content in the lower layer and writes in a tmpfs that resets each boot. The parity test now rejects any drop-in staged under /etc/systemd/system.
Moving the shared payload into os/common took it out of two exemptions it had been relying on: prek's shellcheck hook excludes ^os/yocto/.*, and REUSE.toml carries a blanket annotation for os/yocto/**. CI caught both. nvidia-gpu-detect had four pre-existing findings that the exclusion had been hiding. `[ -r a ] && [ -r b ] || continue` (SC2015) becomes two independent guards, which is what it always meant. The two `case` patterns are globs by design -- that is the whole interface of match_vendor_class -- so those get an explicit disable with the reason rather than a rewrite. For licensing, os/common/nvidia joins the annotation that already covers os/common/rootfs and os/common/scripts, matching how its sibling directories are handled rather than adding headers to files that are otherwise pure renames. `reuse lint` now reports 1212/1212, shellcheck is clean, and the topology suite still passes unchanged.
The suite asserts that both image backends stage the same files to the same
destinations, so living under os/yocto/tests misdescribed it -- and, because
prek's shellcheck hook excludes ^os/yocto/.*, meant the one test in the repo
that checks cross-backend behaviour was itself never linted.
Moving it to os/tests fixed that and shellcheck immediately found real
smells rather than style nits. The parity loop indexed two variables through
`eval "got=\$${backend}_installs"`, which is why shellcheck could not see
they were used; it is now a function taking the list as an argument, which
is both clearer and warning-free. `ls | sort` becomes a glob, and the one
remaining suppression is the grep pattern that has to match a literal
"$common/" in the mkosi script.
Both parity assertions were mutation-tested after the rewrite: adding an
unstaged file to os/common/nvidia fails for both backends, and pointing a
drop-in back at /etc/systemd/system fails the location check.
Why
The image ships one static
/etc/modprobe.d/nvidia.conf:It is installed unconditionally —
nvidiaRDEPENDS onnvidia-modprobe-config— so every NVIDIA image gets it regardless of what the tenant was actually given. Two problems:Protected PCIe is the Hopper multi-GPU confidential mode. Blackwell multi-GPU uses MPT CC instead, where Fabric Manager stays on the host and NVIDIA's CC provisioning flow explicitly sets
ppcie-mode=offon the GPUs. Enabling PPCIe on those tenants contradicts the mode they were provisioned in.A single-GPU tenant needs the opposite knob. With no fabric behind it the NVLink fabric-probe state machine arms and blocks on a Fabric Manager that is never going to route anything, surfacing as
NV_ERR_NVLINK_FABRIC_NOT_READY(802). That needsNVreg_NvLinkDisable=1— which must not be set on a multi-GPU tenant, because it would disable the NVLink the workload runs on.Neither can be baked in, and neither belongs on the kernel command line: that string is measured, so a per-shape token would fork the attestation baseline once per topology.
What changed
Generate the options at boot from what is visible on the guest PCI bus:
NVreg_RegistryDwords="RmEnableProtectedPcie=0x1"NVreg_NvLinkDisable=1NVSwitch presence is a sound proxy for PPCIe: the VMM only hands NVSwitches to the guest in that mode (
vmm-cli --ppcieattaches all GPUs and NVSwitches). Blackwell MPT CC tenants get GPUs only. Reading the live topology rather than a device-ID table means new GPU generations need no change here.nvidia-gpu-detectgrows acount-gpusquery alongside its existinggpu/nvswitchExecCondition modes.nvidia-module-optionsscript + oneshot unit orderedBefore=the two units that modprobe the driver. ThoseExecStartPre=lines are the only place this image loads the NVIDIA modules — they are deliberately kept out ofmodules-load.dso the shared image stays quiet on GPU-less hosts./run/modprobe.d/nvidia-dstack.conf; the rootfs is dm-verity protected and modprobe reads both directories.options nvidia.gpus=N nvswitch=yes|no) so a wrong topology is visible rather than silent.Verification
No Hopper PPCIe or Blackwell hardware was available, so the topology was simulated.
os/yocto/tests/test-nvidia-module-options.shbind-mounts a synthetic/sys/bus/pci/devicesinside a mount namespace and runs the scripts from their installed paths with no environment overrides, so the exercised code includes the hardcoded/usr/bin/nvidia-gpu-detectand/run/modprobe.d:Three assumptions the change rests on were checked rather than assumed:
modprobe reads
/run/modprobe.d. kmod's search path is compiled in, not configurable. Confirmed both by inspecting the binary (/etc/modprobe.d,/run/modprobe.d,/usr/lib/modprobe.d,/usr/local/lib/modprobe.d) and by writing a sentinel and seeing it come back frommodprobe --showconfig.systemd runs the generator before the modprobe sites. The real unit file was loaded by systemd with only
ExecStartredirected to a probe and the two consumers stubbed. Execution order wasgenerator, then both consumers'ExecStartPre.The yocto recipe builds and packages correctly.
bitbake -c package nvidia-modprobe-configsucceeds and produces exactly:with the preset containing
enable nvidia-module-options.service, confirming the unit is enabled at boot.Also
sh -nandshellcheck -x -P SCRIPTDIRon both scripts. The remaining shellcheck notes (SC2015, SC2254) are pre-existing idioms innvidia-gpu-detect— thecasepatterns are deliberately globs — and the new code mirrors them for consistency.The sysfs root and output directory are overridable purely so the matrix can also run where user namespaces are unavailable; the systemd unit sets neither.
Notes
ExecCondition=nvidia-gpu-detect nvswitchalready skips it cleanly on Blackwell tenants, which never see an NVSwitch.os/yoctois excluded from the shellcheck and formatting hooks inprek.tomland there is no yocto test job today. Wiringos/yocto/tests/into a workflow is worth doing separately.