fix(nebius): sweep orphaned validation CI resources and make TerminateInstance idempotent - #156
hakhandelwal11 wants to merge 3 commits into
Conversation
| return res, errors.New("SweepOrphanedInstances: RunID is required") | ||
| } | ||
|
|
||
| instances, err := client.ListInstances(ctx, v1.ListInstancesArgs{}) |
There was a problem hiding this comment.
can we not pass the instance-id of the instance created during the test so we don't have to search through all instances to find a specific one?
There was a problem hiding this comment.
@patelspratik,
Quick clarification: the value the sweep matches on there is ci-run-id — a per-run label stamped on every resource a CI run creates, not an instance id. So it isn't looking up one specific instance; it selects every instance carrying this run's label.
On pushing that match into the list call itself: we could set ListInstancesArgs.TagFilters to {ci-run-id: [runID]} so ListInstances returns only this run's instances. We've kept the explicit match deliberately, because provider support for TagFilters is inconsistent, only some providers apply it, while others ignore it or discard the arguments entirely. Since this is a delete path, relying on the filter is unsafe: against a provider that ignores it, ListInstances would return every instance and the sweep would terminate non-matching — including production — resources. The explicit inst.Tags[ci-run-id] == RunID check keeps that guarantee independent of any provider's implementation.
There was a problem hiding this comment.
ok, how does this prevent concurrent runs from being deleted by each other?
There was a problem hiding this comment.
@patelspratik , Each workflow run gets a unique github.run_id, which we stamp as the ci-run-id label on every resource that run creates. The sweep only matches ci-run-id to its own CI_RUN_ID, so a run can only ever delete resources it created. Two concurrent runs have different run_ids and therefore different labels so run A's sweep does list run B's resources in the shared tenant, but skips them on the mismatch, and vice versa. So they're isolated by the per-run label.
accc87b to
1d9de38
Compare
Problem
The Nebius validation/integration CI provisions real VMs, each with an isolated VPC, subnet, and boot disk. On failure paths these resources are never terminated and leak indefinitely.
Fix
Source fix (P0)
registerInstanceCleanupschedules termination viat.Cleanupbefore any assertion, on a freshcontext.Background()with bounded retry. A terminate that ultimately fails is surfaced ast.Errorf("LEAKED INSTANCE …")— a red test, never swallowed.TerminateInstancemaps gRPCNotFound→v1.ErrInstanceNotFound, wrapping both the sentinel and the raw error soerrors.Ismatches and the gRPC status code / message survive for callers that classify on them.Backstop (P1)
ci-run-idlabel on every VM and its network/subnet/disk at create time.if: always()step runsTestSweepOrphans, which first sweeps VMs viaSweepOrphanedInstances(terminate cascades their attached network/disk), thensweepStandaloneResourcesreaps the leftover network/subnet/disk in dependency order (disk → subnet → VPC). Both match onlyci-run-id == CI_RUN_ID.PageSize=1000+PageToken) across all projects in the tenant — the SDK filter is client-side, and resources can land in any project/region, so a single page/project isn't sufficient.Prod-safety analysis
labels[ci-run-id] == CI_RUN_ID. In the shared tenant the sweep lists prod and other-run resources but skips everything without a matchingci-run-id.ci-run-idonly (ciRunLabels). In productionattrs.Tagsnever containsci-run-id, sociRunLabelsreturnsniland prod VPC/subnet/disk labels are byte-identical to before. The VM's own labels (fullattrs.Tags) are unchanged.