Skip to content

Hold bpfKubeProxyHealthzPort=0 until every calico-node accepts it - #5314

Closed
tomastigera wants to merge 1 commit into
masterfrom
tomas/core-13540-healthz-version-gate
Closed

tomastigera wants to merge 1 commit into
masterfrom
tomas/core-13540-healthz-version-gate

Conversation

@tomastigera

Copy link
Copy Markdown
Contributor

Problem

The operator defaults FelixConfiguration's bpfKubeProxyHealthzPort to 0 when BPF is on and it is not managing kube-proxy (AKS, where the platform owns the kube-proxy DaemonSet).

0 is only a valid value for Calico >= v3.32.0 and Enterprise >= v3.23.0-2.0. Older calico-node rejects it at startup, so nodes being upgraded go into CrashLoopBackOff, and a manifest-to-operator migration deadlocks outright: the nodes it has to replace cannot start.

Change

Gate the write on every calico-node already running the version the operator targets:

  • fresh install — nothing is running, so write immediately
  • pending kube-system migration — manifest-install pods are still serving nodes, so wait
  • otherwise — the status has to report both the target variant and version, which the operator writes together once the stack is available

The DaemonSet lookup that the bpfEnabled defaulting already did moves up so both checks share it, rather than fetching twice.

Relationship to the in-tree copy

Cherry-pick of projectcalico/calico#13843 (0d65aae5de), the same change in the in-tree copy of this controller. It supersedes #5303, which was closed unmerged; this branch is force-updated from that stale head.

Three adaptations were needed, since the two copies have drifted:

  • this repo has a variant-extension block (r.ext.DefaultFelixConfiguration) between the two hunks, and declares err earlier, so the DaemonSet fetch had to be rewritten by hand rather than patched
  • the governing test fixture here uses Variant: CalicoEnterprise where the in-tree one uses Calico, so the positive spec now asserts cr.Status.Variant = cr.Spec.Variant instead of hard-coding the variant
  • three comments were shortened to fit a 20-word limit, so they read slightly differently from the in-tree copy

Tests

Three specs, all driving real reconciles:

  • older version still running → port not set
  • kube-system migration pending → port not set
  • status reports the target variant and version → port set to 0

Mutation-tested: replacing the gate with if true && fails the first two.

Suite: 313 passed / 22 failed. The 22 are all cel_validation_test.go [BeforeEach] envtest setup failures; a control run on unmodified origin/master in the same worktree fails the identical 22 (303 passed / 22 failed), so they are pre-existing and environmental.

CORE-13540

AI assistance

This PR was written in part with the assistance of generative AI.

Release Note

Fix calico-node CrashLoopBackOff on upgrade, and a deadlocked manifest-to-operator migration, caused by the operator writing bpfKubeProxyHealthzPort=0 before every node was running a version that accepts it.

The operator defaults FelixConfiguration's bpfKubeProxyHealthzPort to 0 when BPF
is on and it does not manage kube-proxy. 0 is only a valid value for Calico
>= v3.32.0 and Enterprise >= v3.23.0-2.0; older calico-node rejects it at
startup, so upgrading nodes CrashLoopBackOff, and a manifest-to-operator
migration deadlocks because the nodes it has to replace cannot start.

Gate the write on every calico-node already running the version the operator
targets. A fresh install has nothing running and writes immediately; a pending
kube-system migration means manifest-install pods are still serving nodes, so it
waits; otherwise the status has to report both the target variant and version,
which the operator writes together once the stack is available.

The DaemonSet lookup that the bpfEnabled defaulting already did moves up so both
checks share it.

Cherry-pick of projectcalico/calico#13843 (0d65aae5de), which is the same
change in the in-tree copy of this controller.

CORE-13540
Copilot AI lite review requested due to automatic review settings September 14, 2026 19:58
@tomastigera
tomastigera requested a review from a team as a code owner September 14, 2026 19:58
@marvin-tigera marvin-tigera added this to the v1.45.0 milestone Sep 14, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants