Repository navigation
Gate nodesets on their deployOnAllNodeSets service jobs - #2139
Conversation
A deployOnAllNodeSets service runs as a single AnsibleEE Job for the whole deployment. DedupeServices kept such a service in the first nodeSet's service list only, so every later nodeSet lost its ordering relationship to it: dependsOn resolves against the nodeSet's own list and silently skipped the missing entry. Those nodeSets advanced past the shared Job while it was still mutating their hosts. Keep the service in every listing nodeSet's plan, where it acts as a barrier. The deployment starts its Job before the nodeSet loop, once every service ordered before it in each listing nodeSet has completed, and a nodeSet blocks until that Job succeeded. Readiness is read from the Job rather than from conditions, which are rebuilt every reconcile, so a restart mid-deployment resumes from cluster state. A nodeSet that does not list the service neither waits for it nor delays it. Applies to both serial and parallel execution. Related: OSPRH-37768 Signed-off-by: Rabi Mishra <ramishra@redhat.com>
OpenStackControlPlane CRD Size Report
Threshold reference
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (8)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 SummarySummary by CodeRabbit
WalkthroughThe controller now runs each ChangesGlobal service barriers
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~45 minutes Change: Bug fix Sequence Diagram(s)sequenceDiagram
participant Reconciler
participant ServicePlan
participant GlobalServiceGate
participant NodeSetDeployer
participant GlobalServiceJob
Reconciler->>ServicePlan: Build service plan and service levels
Reconciler->>GlobalServiceGate: Check global service barrier
GlobalServiceGate-->>Reconciler: Return whether preceding jobs completed
Reconciler->>GlobalServiceJob: Start global service when barrier opens
Reconciler->>NodeSetDeployer: Deploy NodeSet services
NodeSetDeployer->>GlobalServiceJob: Check global service completion
GlobalServiceJob-->>NodeSetDeployer: Return job status
NodeSetDeployer-->>Reconciler: Return readiness or barrier wait
Merge Risk: ⚪ Minimal · up to No actionable merge-blocking issue remains from the reviewed change. The reported Job backoff-limit panic does not occur on the controller’s lookup path. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
Build succeeded (check pipeline). ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 16m 00s |
|
Could we account for AnsibleEE Jobs expiring when Would it make sense to persist completion state somewhere durable, or retain these Jobs until the relevant barriers have passed? I may be missing a cleanup detail that already covers this. |
Yeah, that's an existing issue not related to this.. There is a separate PR for it #2159 |
Normally we don't see |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: abays, rabi The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
455746f
into
openstack-k8s-operators:main
2 participants
A deployOnAllNodeSets service runs as a single AnsibleEE Job for the whole deployment.
DedupeServices kept such a service in the first nodeSet's service list only, so every later nodeSet lost its ordering relationship to it: dependsOn resolves against the nodeSet's own list and silently skipped the missing entry. Those nodeSets advanced past the shared Job while it was still mutating their hosts.
Keep the service in every listing nodeSet's plan, where it acts as a barrier. The deployment starts its Job before the nodeSet loop, once every service ordered before it in each listing nodeSet has completed, and a nodeSet blocks until that Job succeeded. Readiness is read from the Job rather than from conditions, which are rebuilt every reconcile, so a restart mid-deployment resumes from cluster state. A nodeSet that does not list the service neither waits for it nor delays it.
Applies to both serial and parallel execution.
Jira: OSPRH-37768