Improve PostgreSQL role separation, provisioning, and connection management

[Jeremy Alvis (iplay88keys)]

Fixes #997

Summary

Note

See docs/postgres.md for information about how this works in practice.

• Give ateapi separate PostgreSQL owner and read/write connections. Migrations and owner maintenance use the owner pool; application reads and writes use the read/write pool. Each connection assumes its configured group role, so an operator can replace a login without changing object ownership.
• Add optional first-install identity bootstrap using an administrator credential. Bootstrap creates the fixed development logins, group roles, schema, memberships, and default privileges from embedded SQL. The binary owns the fixed development usernames and passwords, checks the application connection strings against them, and does not reset existing passwords. Direct ateapi runs default to bootstrap disabled.
• Accept @file:/absolute/path for both owner and read/write connection strings. Substrate rereads each projected Secret file when opening a new physical connection, so rotated credentials can take effect without restarting the process. The --postgres-max-conn-lifetime CLI flag sets the maximum connection lifetime, bounding how long old connections remain in use; endpoint or database changes still require a restart.
• Wire the bundled PostgreSQL credentials and the two application connections through ate-setup and the installation manifests. The shell installer now delegates to ate-setup. Its previous single-connection environment variable still supplies both pools when used alone and remains the owner connection when a separate read/write DSN is added. Bundled PostgreSQL requires password authentication plus its existing client certificate check.
• Standalone Substrate now defaults to the substrate schema, following best practices to not use the public schema from the Postgres Wiki.
• Update the Cloud SQL guide to provision the owner and read/write roles, IAM login membership, and default object grants before deployment; point the setup command at that guide.
• Keep the identity SQL beside the migration sources so operators can run it directly with their own transaction-local login, schema, and role-name settings when provisioning users outside the application. The optional group role settings default to the bundled names.

Scope

This identity layout requires a fresh PostgreSQL database. The PR does not migrate existing users, grants, or tables into it.

The bundled-database installers enable bootstrap to provision Substrate's fixed identities. ateapi also accepts explicit administrator and application connections for a managed shared database; it has no way to infer which deployment owns PostgreSQL. Deployments using operator-managed users disable bootstrap and provide precreated roles, memberships, schema, and grants.

Cloud SQL intentionally uses one IAM database login by default. The owner connection defaults to the read/write connection, and each pool assumes its configured role. Separate logins remain supported through separate connection strings, but requiring two IAM logins would add proxy identity setup. Two logins limit exposure if only the read/write credential leaks; because ateapi holds both pools, they do not isolate a compromised ateapi pod. Stronger isolation would require moving owner operations out of that pod.

The bootstrap usernames and passwords are fixed, published development credentials. Do not enable fixed-identity bootstrap against a production database. For production, provision unique logins and permissions outside ateapi, supply their connection strings, and leave bootstrap disabled.

Reviewer Notes

• I have manually tested the cloud sql setup guide and successfully attached an iam user with both roles to ateapi. The current implementation has a single iam account which has both roles. Each pool does a SET ROLE to the appropriate role (owner for migrations and outbox partition management or read/write for runtime access).

[Eitan Yarmush (EItanya)]

🤖 AI-generated review.

I found three reproducible configuration issues, detailed inline. The five affected package suites passed, and an additional PostgreSQL 18 probe passed the actual OpenFGA migrations and initialization with the restricted roles. Cluster E2E was not run locally.

[Jeremy Alvis (iplay88keys)]

I have local changes that add a second Cloud SQL Auth Proxy sidecar to ate-api-server for the owner GSA, which I have also tested manually. Under the current Workload Identity setup, identity is configured at the pod level: the pod’s Kubernetes service account maps to the runtime GSA. The owner proxy therefore uses that runtime identity to impersonate the owner GSA, which is a documented approach here.

This provides separate PostgreSQL logins for runtime operations and owner operations, although it is not a hard security boundary still because the runtime identity retains permission to impersonate the owner GSA. I also considered the other way around and having the pod's KSA be the owner, which would impersonate the runtime GSA, but that would make the more privileged owner GSA the pod's ambient identity. The least-privileged option there was to have the runtime KSA.

Achieving stronger separation would require moving owner operations out of ate-api-server: migrations would need to run through a separate Job or migrate command and outbox partition maintenance (which currently uses the owner connection) would also need a separately identified workload or a different privilege model.

Julian Gutierrez Oschmann (@juli4n) Eitan Yarmush (@EItanya) Joe Betz (@jpbetz), I’m happy to include the two-proxy approach in this PR, but I think the broader separation should be handled in a follow-up PR with an agreed-upon design and scope.