Bugfix in dcgmlib: Check per-field nvmlReturn for DCGM_FI_DEV_MEMORY_TEMP_CELSIUS

[brianwolfe]

Summary

On L4 nodes, and likely other GPUs that do not support NVML_FI_DEV_MEMORY_TEMP, DCGM reports an invalid "0" value for the temperature. This was raised as a comment here, potentially impacting other cards. I don't have those cards to test.

nvmlDeviceGetFieldValues returns NVML_SUCCESS when it populates any entry in the values array. In nvml.h, it states that the caller must check each value's own nvmlReturn before reading it, because value is undefined unless that per-field status is NVML_SUCCESS.

nvmlDeviceGetFieldValues API

nvml.h states the requirement in the nvmlFieldValue_st struct:

    nvmlReturn_t nvmlReturn;    //!< Return code for retrieving this value. This must be checked before looking at value, as value is undefined if nvmlReturn != NVML_SUCCESS
    nvmlValue_t value;          //!< Value for this field. This is only valid if nvmlReturn == NVML_SUCCESS

and in the function call:

 *         - \ref NVML_SUCCESS                 if any values in \a values were populated. Note that you must
 *                                             check the nvmlReturn field of each value for each individual
 *                                             status

So NVML_SUCCESS means that "the array was populated", not that any values were populated.

The DCGM_FI_DEV_MEMORY_TEMP_CELSIUS handler in BufferOrCacheLatestGpuValue checked only the call's return and then read value.value.uiVal. On a GPU with no memory temperature sensor NVML succeeds and sets the per-field status to NVML_ERROR_NOT_SUPPORTED, leaving value at its zero-initialized state, so the handler cached a literal 0 as a real measurement rather than DCGM_INT64_NOT_SUPPORTED.

What is the Result

This results in a valid-looking temperature of 0 C. This is inside the valid domain for a temperature, so an unsupported field is then indistinguishable from a valid reading for every consumer of the field. This shows up when monitoring GPUs with dcgm-exporter on L4 GPUs on GCP.

Using such an L4 GCP node, I reproduced the underlying behavior. Calling NVML directly on an L4 (driver 580.173.02), with no DCGM in the path:

  NVML_FI_DEV_MEMORY_TEMP (id 82)
    call nvmlDeviceGetFieldValues -> NVML_SUCCESS
    value.nvmlReturn              -> NVML_ERROR_NOT_SUPPORTED
    value.uiVal                   -> 0

nvidia-smi recognizes the missing sensor on the same card:

$ nvidia-smi -q -d TEMPERATURE
        GPU Current Temp                               : 57 C
        Memory Current Temp                            : N/A
        Memory Max Operating T.Limit Temp              : N/A

Fix

Check the per-field nvmlReturn for DCGM_FI_DEV_MEMORY_TEMP_CELSIUS.

I made this fix. Afterwards, the previous test scripts using python and using dcgm-exporter showed the expected behavior (missing the time series, not 0 C).

I did not have an obvious way to test the other usages of nvmlDeviceGetFieldValues, but it looks like DCGM_FI_DEV_BOARD_POWER_RAW_WATTS could have the same error. I did not verify or fix. I didn't have hardware to test that failure on so left it as-is.