update-module: KeyError crash in 95cleanup_images when PREV_IMAGE_URL is unset

[danb35]

Steps to reproduce

• Have a module instance whose state/environment has never had a PREV_IMAGE_URL (and friends) set — e.g. its first-ever update-module run since install.
• Run update-module <image_url> <module_id> targeting that instance.

Expected behavior

The update completes cleanly: image pulled, container recreated, old images cleaned up, admin UI refreshed.

Actual behavior

The container itself updates and starts fine, but the update-module task aborts partway through with a KeyError, and the cluster-level update-module action then reports the whole instance as failed — which also means the admin UI extraction step for that instance never runs (it's gated behind a successful per-instance result), so the UI silently stays stale even though the module's own code is current.

Root cause is in core/imageroot/usr/local/agent/actions/update-module/95cleanup_images:

image_old = 'PREV_IMAGE_URL'
image_new = 'IMAGE_URL'
if os.environ.get(image_new) != os.environ.get(image_old):
    print(f"Removing {os.environ.get(image_old)} (differs from {os.environ.get(image_new)})", file=sys.stderr)
    agent.run_helper('podman', 'image', 'rm', os.environ[image_old])   # <-- os.environ[...], not .get(...)

Every other access in this same file (both this pair's own print statement, and the earlier loop over PREV_*_IMAGE keys) reads via os.environ.get(...), which is None-safe. This one line indexes directly, so when PREV_IMAGE_URL isn't present at all, the != comparison is trivially true (None != "<image_url>") and the very next line raises KeyError: 'PREV_IMAGE_URL' instead of just skipping the removal (there's nothing to remove — there was no previous image URL).

Real-node journal excerpt (module id automx1):

task/module/automx1/f3ca9040-b007-4b63-89c2-31944811759d: update-module/95cleanup_images is starting
Removing None (differs from ghcr.io/danb35/automx:latest)
Traceback (most recent call last):
  File "/usr/local/agent/actions/update-module/95cleanup_images", line 32, in <module>
    agent.run_helper('podman', 'image', 'rm', os.environ[image_old])
                                                ~~~~~~~~~~^^^^^^^^^^^
KeyError: 'PREV_IMAGE_URL'
task/module/automx1/f3ca9040-b007-4b63-89c2-31944811759d: action "update-module" status is "aborted" (1) at step 95cleanup_images

One thing I couldn't fully explain: 05pullimages (the step before this one) is supposed to populate PREV_IMAGE_URL from os.getenv('IMAGE_URL') via agent.mset_env() before overwriting IMAGE_URL itself. On a second attempt, IMAGE_URL was confirmed present and correct in state/environment going in, so PREV_IMAGE_URL should have been computed as a real, non-None value that run — but 95cleanup_images still hit the identical KeyError afterward. I didn't dig further into 05pullimages/mset_env itself since the crash site above is the one clearly worth fixing regardless (a None/missing previous value is always a legitimate state — nothing to clean up, not an error), but it may be worth a second look in case there's also a persistence issue in the 05pullimages → 95cleanup_images handoff.

Workaround

Manually seed the missing fields before retrying, from the module's own context:

runagent -m <module_id> python3 -c "
import sys, os
sys.path.insert(0, '/usr/local/agent/pypkg')
import agent
agent.mset_env({
    'PREV_IMAGE_URL': os.environ['IMAGE_URL'],
    'PREV_IMAGE_ID': os.environ['IMAGE_ID'],
    'PREV_IMAGE_DIGEST': os.environ['IMAGE_DIGEST'],
    'PREV_IMAGE_REOPODIGEST': os.environ['IMAGE_REOPODIGEST'],
})
"
update-module <image_url> <module_id>

Components

NS8 core, module ghcr.io/danb35/automx (own module, not core-provided) — but the bug is entirely in core's inherited update-module action, not the module being updated.

See also

Found while updating the ghcr.io/danb35/automx module on a real node.