Skip to content

fix(server): warn on dangling provider refs at startup - #4661

Open
ericcurtin wants to merge 2 commits into
NVIDIA:mainfrom
ericcurtin:fix/4657-dangling-provider-startup/ericcurtin
Open

ericcurtin wants to merge 2 commits into
NVIDIA:mainfrom
ericcurtin:fix/4657-dangling-provider-startup/ericcurtin

Conversation

@ericcurtin

Copy link
Copy Markdown
Contributor

Summary

A deleted provider no longer blocks gateway startup. The preflight warns and keeps validating.

Related Issue

Closes #4657

Changes

  • Preflight looks up each provider and warns per sandbox on missing ones.
  • Split load_provider_environment_record out of load_provider_environment_records to share the lookup.

Testing

  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Ran cargo clippy and the startup_preflight unit tests.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ericcurtin

Copy link
Copy Markdown
Contributor Author

@johntmyers @elezar PTAL, and /ok to test 1ed3a28e09724eae20ff8ed886ff1cf641858ef3 if it looks good. Thank you!

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
@ericcurtin

Copy link
Copy Markdown
Contributor Author

Merged main to fix conflicts. @johntmyers @elezar PTAL, and /ok to test 30c1f3b952de6bc334f529ed0338470e4daff6da if it looks good. Thank you!

@mavromixaliskoullis-beep

Copy link
Copy Markdown

Is ok

@mrunalp mrunalp added the test:e2e Requires end-to-end coverage label Oct 17, 2026
@mrunalp

mrunalp commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator

/ok to test 30c1f3b

@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/4661 does not exist yet. A maintainer needs to comment /ok to test 30c1f3b952de6bc334f529ed0338470e4daff6da to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

@ericcurtin

Copy link
Copy Markdown
Contributor Author

E2E failures look unrelated to this change: k3s-ha pod gating timeout, ssh_host_identity container exit, and workspace operator lbl-sb1 already exists. Could a maintainer re-run the failed jobs? Thank you!

@EmilienM EmilienM left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. This does what #4657 asks: a missing provider gets one warning per sandbox, and store errors and the #2962 invalid-composition guard stay fatal. I left three small comments, none blocking: test ordering, the sequential lookups, and the UpdateConfig path that shares this helper. Heads up that my approval doesn't count toward required reviews, so you'll still need a maintainer's.

)
.await?
.layers;
// A provider deleted under a sandbox leaves a dangling reference. The

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helper also backs the global-policy delete in UpdateConfig (around L4398), so that RPC stops failing on a dangling reference too. I think that's the right call since the sandbox fails closed either way, but it deserves a line in the PR description, and maybe a test next to deleting_global_policy_rejects_reactivated_ambiguous_provider_binding.

// sandbox already fails closed, so warn and validate the rest.
let mut records = Vec::with_capacity(provider_names.len());
let mut missing = Vec::new();
for name in provider_names {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this loads providers one at a time, while load_provider_environment_records fans out with try_join_all (#4320). It's on the startup path before the listener binds, so each extra provider adds two sequential round trips per sandbox. You could try_join_all over load_provider_environment_record and zip the results with provider_names to split found from missing. That keeps the concurrency, and store errors still short-circuit.

async fn startup_preflight_still_rejects_invalid_policy_beside_dangling_reference() {
let state = test_server_state().await;
install_dangling_provider_sandbox(&state).await;
install_ambiguous_provider_binding(&state, "beside-dangling").await;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this test always exercises the skip. created_at_ms is the insert wall clock, and the list orders by (created_at_ms, name). When both inserts land in the same millisecond (easy on in-memory SQLite), sandbox-beside-dangling sorts before sandbox-dangling, so the invalid sandbox fails first and the test can pass on main too. A suffix that sorts after dangling (say z-beside-dangling) pins the order. A case with both providers on one sandbox would also catch a regression that skips the whole sandbox.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(server): report dangling provider references instead of failing startup

4 participants