fix(k8s): handle DeletedFinalStateUnknown in informer delete handlers

[mayankpande88]

Problem

Every DeleteFunc in common/ip_resolver.go asserted the informer payload straight to its concrete type:

DeleteFunc: func(obj interface{}) {
    pod := obj.(*v1.Pod)     // panics on a tombstone
    ...
}

client-go does not always deliver the object on a delete. When a watch is interrupted and the final delete event is missed, the informer resyncs and delivers a cache.DeletedFinalStateUnknown tombstone wrapping the last known state. The assertion panics on it.

These handlers run on the shared informer's goroutine, so the panic reaches apimachinery's runtime handler — which is fatal. The process exits and the pod restarts:

panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod
  common.(*K8sIPResolver).addPodHandlers.func3
level=fatal msg="Panic in Kubernetes runtime handler" subsys=k8s

Reported from a customer cluster where this accounted for 14 of 15 node-agent restarts in 12 hours across two nodes (Terminated / Error / exit 1).

This is not load- or memory-related. One missed delete event is enough, so it fires on any cluster where the watch connection is disrupted — more likely on clusters with flaky control-plane connectivity or high object churn.

Scope

All nine delete handlers had the same defect, not just the reported pod one:

ReplicaSet · DaemonSet · StatefulSet · Job · CronJob · Service · Deployment · Pod · Node

Fixing only the reported handler would leave eight identical landmines, each capable of the same fatal panic.

Fix

func deletedObject[T any](obj interface{}) (T, bool)

Unwraps the tombstone, and returns ok=false for anything it cannot interpret — wrong type, nil payload, unrelated value — so the delete is skipped rather than killing the agent.

Skipping is safe: these maps are keyed by UID and the object is already gone. A missed delete leaves a stale entry that the next resync reconciles, which is strictly better than terminating the process.

Testing

go build ./...        ok  (whole repo)
go vet ./common/      clean
--- PASS: TestDeletedObjectUnwrapsTombstone      (plain object, tombstone)
--- PASS: TestDeletedObjectRejectsBadPayloads    (nil, wrong type, tombstone-wrapping-wrong-type,
                                                  tombstone-wrapping-nil, unrelated value)
--- PASS: TestDeletedObjectOtherTypes            (Service, Node)

The bad-payload cases assert no panic specifically, since panicking is the failure mode being fixed.

./common is not in CI's test exclusion (only ./containers is, for NVML reasons), so go test $(go list ./... | grep -v '/containers$') will run these and gate regressions.

Note on the same incident

The customer also reported 1 OOM out of those 15 restarts. That is a separate matter and is not addressed here. Heap profile from the affected pod shows 120MB total with a healthy distribution — no runaway allocation — so the OOM looks like a distinct, much rarer event rather than the crash-loop cause.

[gemini-code-assist]

Code Review

This pull request introduces a generic helper function deletedObject in common/ip_resolver.go to safely extract deleted Kubernetes objects from informer DeleteFunc payloads. This prevents application panics and crash loops when client-go delivers a cache.DeletedFinalStateUnknown tombstone instead of the actual object. Direct type assertions in the delete handlers for various Kubernetes resources (such as Pods, Services, and ReplicaSets) have been updated to use this helper. Additionally, comprehensive unit tests have been added in common/ip_resolver_delete_test.go to verify correct behavior under different scenarios. There are no review comments, and I have no additional feedback to provide.