Skip to content

fix(server): report dangling provider references instead of failing startup #4657

Open

Description

@EmilienM

User Story

I run multi-replica OpenShell gateways with PostgreSQL. A sandbox whose spec references a provider that was later deleted shouldn't stop every gateway restart, on any release, until someone edits the database by hand.

Problem Statement

Before #4147, DeleteProvider took no mutation guard, so a delete racing an attach could leave a sandbox whose spec names a provider that no longer exists. #4147 stops new dangling references but doesn't repair the ones a database already has.

When no global policy is set, validate_provider_composition_startup_preflight (crates/openshell-server/src/grpc/policy.rs) runs before any listener binds (crates/openshell-server/src/lib.rs) and any error is fatal. It pages every sandbox in every workspace through validate_provider_composition_for_existing_sandboxes and builds each sandbox's provider layers. A missing provider fails the record lookup with FailedPrecondition ("provider 'X' not found"); the ? ends the loop, and startup fails.

Impact / Why This Matters

Every gateway restart against that database fails, so the fleet crashloops and the upgrade that would prevent new dangling references can't roll out. The affected sandbox already fails closed on its own: create fails fast on missing providers, and a supervisor can't sync an admitted configuration for it, so the workload stays unstarted.

Proposed Design

The preflight detects a missing provider from the lookup result (not by matching the error string), logs one warning per affected sandbox naming the workspace, sandbox, and providers, and continues validating the remaining sandboxes. Store and catalog errors stay fatal, and a sandbox whose composed policy would be invalid still blocks startup (the #2962 guard).

Acceptance Criteria

  • A database holding a dangling provider reference no longer stops the gateway from starting.

Alternatives Considered

  • A repair step that removes the missing references at startup: it needs a mutation guard and a CAS per sandbox, and it changes user data at startup, while the affected sandbox already fails closed on its own.
  • Dropping the preflight: rejected; it guards upgrades where a persisted sandbox's composed policy would be invalid (feat(providers): make profiles authoritative #2962), and that case must stay fatal.

Part of #4307.

Activity

  1. added
    state:acceptedA maintainer decided OpenShell should pursue this issue
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:acceptedA maintainer decided OpenShell should pursue this issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions