mctl-api

Control plane API for the mctl.ai Internal Developer Platform — dual-protocol (REST + MCP) access to manage services, workspaces, databases, previews, and platform operations.

What It Does

mctl-api is the central gateway for the mctl.ai platform. It exposes every platform operation — deploying services, creating team workspaces, provisioning databases, managing custom domains — as both REST endpoints and MCP tools. AI coding assistants (Claude, Copilot, Cursor, Gemini, and others) connect via the MCP protocol, while Backstage and CLI tools use the REST API. All write operations are executed through Argo Workflows and reconciled by ArgoCD through a GitOps repository.

Architecture

AI Clients (Copilot CLI / Claude / Cursor / VS Code / Windsurf / Gemini / Continue.dev)
    │
    ▼
MCP Server (/mcp endpoint, Streamable HTTP)
    │
    ▼
REST API (/api/v1/*)  ◄── Backstage / CLI / HTTP clients
    │
    ├──► Argo Workflows    (submits deploy/provision/rollback jobs)
    ├──► GitOps Repo       (reads service configs, commits changes)
    ├──► ArgoCD            (sync status, health checks)
    ├──► Kubernetes        (resource quota usage)
    ├──► Loki              (service log queries)
    └──► PostgreSQL        (audit log)

Tech Stack

Category	Details
Language	Go 1.25
HTTP Router	chi v5.2
MCP Server	mcp-go v0.31 (Streamable HTTP transport)
Auth	OAuth 2.1 + PKCE (sign-in through ZITADEL), federated OIDC JWTs, static service tokens
Database	PostgreSQL via pgx v5.9 (audit logs)
Kubernetes	client-go v0.32 (quota reader)
Rate Limiting	httprate v0.15 (300 req/min authenticated including POST /mcp, loopback MCP-internal REST excluded from that cap, 20 req/min write)
Container Registry	ghcr.io/mctlhq/mctl-api
Linting	golangci-lint (errcheck, govet, staticcheck, gosec, bodyclose)

Project Structure

mctl-api/
├── cmd/api/main.go              # Entry point, server bootstrap
├── internal/
│   ├── api/                     # REST handlers & router
│   │   ├── handlers_read.go     # Read operations (list, status, config)
│   │   ├── handlers_write.go    # Write operations (deploy, create, retire)
│   │   ├── handlers_domains.go  # Custom domain management
│   │   ├── handlers_repos.go    # Repository access management
│   │   ├── oauth_handlers.go    # OAuth 2.0 Authorization Code + PKCE
│   │   └── router.go            # Route definitions & middleware
│   ├── auth/                    # GitHub + OIDC + OAuth authentication
│   ├── mcp/                     # MCP server implementation (tool registry)
│   ├── operations/              # Platform operation registry & executor
│   ├── argocd/                  # ArgoCD API client
│   ├── gitops/                  # GitOps repo reader (clone, parse values)
│   ├── audit/                   # Audit logging (PostgreSQL-backed)
│   ├── k8s/                     # Kubernetes quota reader
│   ├── loki/                    # Loki log query client
│   └── openapi/                 # Embedded OpenAPI spec
├── e2e/                         # End-to-end tests (build tag: e2e)
├── helm/                        # Kubernetes Helm chart
├── Dockerfile                   # Multi-stage Alpine build
├── Makefile                     # Dev commands (run, test, lint, build)
├── .golangci.yml                # Linter configuration
├── .env.example                 # Environment variable template
├── mcp-config.example.json      # MCP client config examples
└── .github/workflows/build.yml  # CI/CD pipeline

Getting Started

Prerequisites

Go 1.25+
GitHub CLI (gh) — for obtaining auth tokens
Access to the mctlhq GitHub organization

Local Development

# Run API against local gitops clone
make run

# Run MCP stdio server pointing at local API
make run-mcp

# Both together
make run &
make run-mcp

Auth is bypassed locally (AUTH_REQUIRED defaults to false when unset). Set AUTH_REQUIRED=true to test authentication.

Docker

docker build -t mctl-api .
docker run -p 8080:8080 mctl-api

Configuration

Environment Variables

Variable	Description	Default	Required
`PORT`	HTTP server port	`8080`	No
`SHUTDOWN_DRAIN_DELAY`	Go duration the pod keeps serving after SIGTERM with `/readyz` at 503 (`checks.shutdown=draining`) before the listener closes. Keep it plus the 10s shutdown timeout below the pod's `terminationGracePeriodSeconds`. Invalid or non-positive values use the default.	`5s`	No
`AUTH_REQUIRED`	Enable authentication	`true`	No
`ADMIN_USERS`	Admin GitHub usernames (comma-separated)	—	No
`GITOPS_REPO_URL`	GitOps repository URL	`https://github.com/mctlhq/mctl-gitops.git`	No
`GITOPS_BRANCH`	GitOps branch	`main`	No
`GITOPS_LOCAL_PATH`	Local cache path for gitops clone	`/tmp/mctl-gitops`	No
`GITHUB_APP_TOKEN_FILE`	Path to a mounted GitHub App installation token. Takes precedence over `GITOPS_REPO_TOKEN` and `GITOPS_ACTIONS_TOKEN`, and is re-read per use so a rotation is picked up without a restart.	—	No
`GITOPS_REPO_TOKEN`	GitHub token for HTTPS clone. Fallback when `GITHUB_APP_TOKEN_FILE` is unset; read once at startup, so only for credentials that do not rotate.	—	No
`GITOPS_ACTIONS_TOKEN`	GitHub token that starts `workflow_dispatch` runs in mctl-gitops. Fallback when `GITHUB_APP_TOKEN_FILE` is unset; read once at startup.	—	No
`GITHUB_TOKEN`	Last-resort fallback for the clone, after `GITHUB_APP_TOKEN_FILE` and `GITOPS_REPO_TOKEN`.	—	No
`GITOPS_SSH_KEY_PATH`	SSH key path for git clone. Takes precedence over every token above.	—	No
`ARGOCD_URL`	ArgoCD API endpoint	`https://ops.mctl.ai`	No
`ARGOCD_TOKEN`	ArgoCD auth token (from Vault)	—	Yes
`ARGO_WORKFLOWS_NAMESPACE`	Kubernetes namespace for workflows	`argo-workflows`	No
`GITHUB_ORG`	Required GitHub organization	`mctlhq`	No
`DEX_ISSUER_URL`	Legacy, unset in production since Dex was removed (mctl-gitops#1741). Dex OIDC issuer URL; still read by the legacy `dex` federation shim. Unset turns Dex off: no Dex JWTs, no `dex` federation shim, no Dex check in `/readyz`.	—	No
`DEX_CLIENT_ID`	Legacy, unset in production (see `DEX_ISSUER_URL`). JWT audience for Dex. Empty skips the audience check (compatibility) and counts `federation_audience_check_skipped_total{provider="dex"}` on every verification.	—	No
`MCTL_OIDC_PROVIDERS`	JSON array of federation identity providers (mctl-api#374, see docs/federation.md): `[{"name":"dex","issuer":"https://…","audiences":["mctl-api"],"audience_enforcement":"audit"}]`. An entry named `dex` replaces the legacy shim built from `DEX_ISSUER_URL`/`DEX_CLIENT_ID` above, whatever its issuer; an entry under any other name on the `DEX_ISSUER_URL` issuer collides with the shim and refuses startup. `audiences` is required, `audience_enforcement` is `audit` or `enforce` (default `enforce`). A malformed value, a missing `audiences`, an invalid `audience_enforcement` or `kind`, a reserved or surface-colliding name, or a duplicate name/issuer refuses startup — unless `MCTL_FEDERATION_DISABLED` is on.	—	No
`MCTL_FEDERATION_DISABLED`	Kill switch (mctl-api#374): any value except `false`/`f`/`0`/`no`/`off` restores the pre-registry authentication chain unchanged, including its plain (non-constant-time) service-token compare. While it is on, `MCTL_OIDC_PROVIDERS` is neither validated nor built, so it also recovers a value that would otherwise refuse startup. Same shape as `PRINCIPALS_DISABLED`.	unset	No
`MCTL_GITHUB_ACTIONS_OIDC`	JSON object enabling GitHub Actions OIDC for CI deploys (mctl-api#530, see docs/federation.md): `{"repository_owners":["mctlhq"],"audience":"https://api.mctl.ai","branches":["refs/heads/main"]}`. The principal `ci:<owner>/<repo>` may only call `POST /api/v1/operations/deploy-service/execute` with `action=deploy` for a component whose gitops `github.com/source-repo` is that repository. `repository_owners` is required; a malformed value refuses startup. Unset: the provider is off and such tokens get 401.	unset	No
`SELF_URL`	Public base URL	`https://api.mctl.ai`	No
`ALLOWED_ORIGINS`	CORS allowed origins	—	No
`OAUTH_GITHUB_CLIENT_ID`	GitHub OAuth app client ID	—	No
`OAUTH_GITHUB_CLIENT_SECRET`	GitHub OAuth app client secret	—	No
`OAUTH_JWT_SECRET`	JWT signing secret for OAuth tokens	—	No
`OAUTH_UPSTREAM`	Where `/oauth/authorize` signs people in (mctl-api#467, see docs/principals.md): `github`, `zitadel` (as the linked principal's GitHub login; under `TENANT_AUTHZ_MODE=principal` an identity with no GitHub login signs in as its own principal), or `both` (a chooser, ZITADEL first). With `customers_org_id` and `ZITADEL_HOOK_GITHUB_IDP_ID` set, `zitadel` and `both` also offer customers "Sign in with GitHub" (mctl-web#158). `zitadel`/`both` refuse startup without `OAUTH_JWT_SECRET`, both `OAUTH_ZITADEL_CLIENT_` and the provider entry, and `both` also without both `OAUTH_GITHUB_CLIENT_`. An unknown value refuses startup	`github`	No
`OAUTH_ZITADEL_CLIENT_ID` / `OAUTH_ZITADEL_CLIENT_SECRET`	Confidential ZITADEL client of the ZITADEL upstream (redirect `<SELF_URL>/oauth/zitadel/callback`); not the link client. Helm: `oauthZitadelSecret`	—	No
`OAUTH_ZITADEL_PROVIDER`	`MCTL_OIDC_PROVIDERS` entry whose issuer the ZITADEL upstream verifies against; must be the one linking writes (`ZITADEL_LINK_PROVIDER`)	`zitadel`	No
`OAUTH_ZITADEL_REDIRECT_BASE_URL`	Development only, never set in a deployment: origin that replaces `SELF_URL` in the ZITADEL redirect_uri, e.g. `http://localhost:8080` for a local run behind a quick tunnel (https, or http on loopback)	—	No
`OAUTH_ALLOWED_REDIRECT_URIS`	Comma-separated redirect URIs accepted at `/oauth/authorize` and by Dynamic Client Registration (`POST /oauth/register`). An entry is an exact string compare unless it ends in `/`, which makes it a prefix match; prefer exact entries (the Cloudflare MCP portal's `https://mcp.mctl.ai/servers-callback` and its dashboard `…/mcp-server/oauth-callback/api` callback are listed exactly, never as a `dash.cloudflare.com/` prefix). Loopback `http://` callbacks are always accepted (RFC 8252). A registration naming any other URI gets `400 invalid_redirect_uri`	glama, smithery, `chatgpt.com/connector/oauth/*`	No
`OAUTH_DB_URL`	PostgreSQL for OAuth refresh tokens, dynamic client registrations, and the pending authorizations and authorization codes of sign-ins in flight (table `oauth_flow_entries`, shared by every replica) (falls back to `AUDIT_DB_URL`). With a database, a client registered through `POST /oauth/register` survives restarts, and registration is idempotent: the same `client_name` and redirect-URI set get the same `client_id` back. Registrations that have never completed a token exchange are capped at 1000 (least recently seen evicted); a client that has completed one is never evicted by the cap. Any row is dropped after 90 days unseen, where a registration, token exchange or refresh counts as seen. Without one, registrations are in-memory and lost on restart. Every registration is granted scope `mctl`, the only scope	`AUDIT_DB_URL`	No
`OAUTH_PREREGISTERED_CLIENTS`	JSON array of static public OAuth clients for counterparts that cannot register dynamically (the Cloudflare MCP portal): `[{"client_id":"…","client_name":"…","redirect_uris":["https://…"]}]`. Exact redirect match, no secret field, never evicted; a malformed value or an unknown key refuses startup	—	No
`OAUTH_CIMD_ACCEPT`	`true` makes the server accept OAuth Client ID Metadata Documents: an `https://` `client_id` URL is resolved from the JSON document at that URL (its `redirect_uris` alone decide the callback and must be `https` or loopback IP-literal `http`; RFC 8252 loopback IP-literal entries match on any port), and the human sees a consent page on every authorization. It does not announce CIMD in discovery; that is `OAUTH_CIMD_ADVERTISE`. A document is cached for its `Cache-Control: max-age` (default 1 h, at least 30 s, at most 24 h; `no-store`, `no-cache` or `max-age=0` give the 30 s floor; an invalid `max-age` is ignored), so a change to it takes effect within that lifetime. DCR and pre-registered clients are unchanged. Turning it on is a breaking change for any existing client whose `client_id` is an `https://` URL and is not pre-registered: such a client_id is then resolved as a metadata document ahead of `OAUTH_ALLOWED_REDIRECT_URIS` and the loopback rule, including when its host is not on `OAUTH_CIMD_ALLOWED_HOSTS`, in which case the authorization is refused outright with `400` and never falls back. Before turning it on, confirm no live client uses a URL-shaped `client_id`, or add its host to the allowlist. See `docs/federation.md`. Strict `true`/`false`: any other value refuses startup. Replaces the retired `OAUTH_CIMD_ENABLED`, which now refuses startup whenever it is set	`false`	No
`OAUTH_CIMD_ADVERTISE`	`true` makes `/.well-known/oauth-authorization-server` advertise `client_id_metadata_document_supported`. Clients that prefer CIMD (ChatGPT, for one) then switch to it, and a CIMD client whose host is off `OAUTH_CIMD_ALLOWED_HOSTS` or whose document is refused gets `400` with no DCR fallback, so turn it on only when the allowlist covers every CIMD client in use (mctl-api#693). Requires `OAUTH_CIMD_ACCEPT=true`, otherwise startup is refused. Strict `true`/`false`	`false`	No
`OAUTH_CIMD_ALLOWED_HOSTS`	Comma-separated hosts a CIMD `client_id` may be fetched from (exact host, or `.example.com` for subdomains). Mandatory when `OAUTH_CIMD_ACCEPT=true`: an empty or malformed list refuses startup, and so does an empty entry, including the one a trailing comma makes (`app.example.com,`). In production, list the exact host of a known client rather than a `.suffix`	—	When CIMD is on
`OAUTH_GROUPS_MAX_STALENESS`	Strict mode for tenant-group resolution (mctl-api#411, see "Authorization group freshness" below): a Go duration beyond which a gitops checkout that has not synced recently is treated as a failed resolution instead of merely a logged warning. Unset (or malformed) disables strict mode.	unset	No
`OAUTH_GROUPS_CACHE_TTL`	How long a successful tenant-group resolution is memoized per login, bounding the added cost of re-resolving on every JWT validation. Unset or malformed falls back to the default.	`30s`	No
`AUDIT_DB_URL`	PostgreSQL connection string (falls back to in-memory). `sslmode=disable` is upgraded to `require`, or `verify-full` when a CNPG CA is mounted.	—	No
`DB_POOL_MAX_CONNS`	Size of the connection pool each distinct `_DB_URL` gets. Stores whose URLs are identical (every one that falls back to `AUDIT_DB_URL`) share one pool, so this caps the connections per database per replica; `pool_max_conns` in a URL overrides it for that URL. A value that is not a positive integer refuses boot. Keep pods (replicas plus the surge pod of a rolling update) x (distinct URLs x this value + 3 unpooled connections) at or under the database role's connection limit. Startup logs the resulting pools (`pgshared: pools`), and `/metrics` exports their use as `mctl_api_db_pool_` by host and database; a full pool shows as `mctl_api_db_pool_empty_acquires_total` rising while `acquired_conns` sits at `max_conns`.	`8`	No
`HUMAN_INPUT_DB_URL`	PostgreSQL for the human-input delivery ledger (`POST /api/v1/human-input/{request_id}/response`). Falls back to `AUDIT_DB_URL`; with neither, that endpoint answers 503 (there is no in-memory fallback). The raw answer is kept only until its delivery resolves, or until the request expires.	—	No
`WORK_ITEMS_DB_URL`	PostgreSQL for the `workitem/v1` store (`/api/v1/work-items*`). Falls back to `AUDIT_DB_URL`; with neither, every work-items route answers 503.	—	No
`COMMERCE_DB_URL`	PostgreSQL for the commerce ledger of ADR 020 (mctl-api#547 and #548, see docs/commerce-ledger.md): terms, quotes, invocation receipts and history, tenant budgets, holds and the simulated payment operations. Falls back to `AUDIT_DB_URL`; with neither, `/api/v1/commerce*` answers 503. Deploying it enables no payment behaviour: no terms exist until an admin publishes them, no budget exists until an admin grants one, a budget is a limit and never a balance, and `funds_mode: live` is refused everywhere.	—	No
`COMMERCE_DISABLED`	Kill switch for the commerce ledger: routes answer 503 without touching `AUDIT_DB_URL` for other stores. Same values as the other kill switches.	unset	No
`COMMERCE_ACCEPT_DISABLED`	Accept-only kill switch (mctl-api#570): new quotes and accepts answer 422 `unsupported-execution`, while reads, budgets, reserve, dispatch, finalize, release and the reconciler stay live so outstanding liability can drain. `COMMERCE_DISABLED` takes precedence. Same values as the other kill switches. See docs/commerce-ledger.md.	unset	No
`COMMERCE_DISPATCH_KEY_ID`	Identifier of the Ed25519 key that signs dispatch authorizations for paid capabilities (ADR 020 sec. 9, mctl-agents#619). Set together with `COMMERCE_DISPATCH_SIGNING_KEY`; one without the other, or an unusable seed, refuses startup unless `COMMERCE_DISABLED` is on.	unset	No
`COMMERCE_DISPATCH_SIGNING_KEY`	Base64 Ed25519 seed (32 bytes) for dispatch authorizations. Without both values, `…/invocations/{id}/dispatch` and `/api/v1/commerce/dispatch-keys` answer 503 `dispatch-unavailable`, and no paid call can be dispatched. Enables no payment behaviour by itself.	unset	No
`WORK_ITEMS_DISABLED`	Kill switch: any value except `false`/`f`/`0`/`no`/`off` leaves the work-items store off (routes answer 503) without touching `AUDIT_DB_URL` for other stores.	unset	No
`MCTL_AGENTS_APPROVE_LEGACY_APPROVER`	Whether the `mctl-agent` service principal may relay a free-text `approver` on `mctl-agents-approve` (mctl-api#571). `true`/empty keeps the relay and audits every use as `approver.legacy_relay`; `false` refuses it (400 `legacy_approver_disabled`, audited), and a DevLoop approve with no pending `mctl-agents-approve` action approval for the workflow's `we_` answers 409 `approval_grant_missing` instead of signalling the legacy `{approver}` payload. Every use is also counted in `mctl_agents_approve_legacy_relay_total{outcome="relayed"	"refused"}`, since an audit write is best-effort. Turned off only after mctl-agents approves through action-approval grants (flow and error codes: [docs/principals.md](docs/principals.md#devloop-approval-by-grant-571)). A relay carries no content digest, so since mctl-api#722 it is refused with 400` approval_content_unbound`even when the flag is on, and counted as`refused`. Any other value refuses startup	`true`
`MCTL_AGENT_RECEIPT_GATE`	Server-side receipt gate for governed MCP tool calls from the `mctl-agent` service principal (mctl-api#723). Governed are the tools mctl-agents' checkpoint marks REQUIRE_APPROVAL (policy `mctl-api/agent-receipt-gate/v1`: every `mctl_` tool except the `mctl_get_`/`list_`/`read_`/`search_`/`describe_` reads, `whoami`, `incident_summary`, `resolve_agent`, `resolve_incident` and `acknowledge_incident`). A receipt is the reserved tool arguments `action_approval_ref` (an `aar_` request that `service:mctl-agent` itself requested and consumed, with `action_kind` `mcp.tool.call:mcp__mctl__<tool>` and target `mctl`) and `action_approval_artifact_hash`; both are removed from every call before a handler sees them. `off`: no check. `observe`: every governed call runs and is audited as `agent-receipt-gate` (status `succeeded` or `observed` with a code); a presented receipt is checked read-only. `enforce`: a governed call without a valid receipt is refused with a tool error naming the code (`agent_receipt_missing`, `_malformed`, `_not_found`, `_operation_mismatch`, `_target_mismatch`, `_not_consumed`, `_already_executed`, `_artifact_mismatch`, `_unverifiable`), and a valid one is spent (consumed → executed). Each decision also increments `mcp_agent_receipt_gate_total{mode,code}` (code `valid` for an authorizing receipt), which counts even when the best-effort audit write fails. A `_malformed` row stores no part of what was sent except a bounded `malformed` parameter: whether the ref has the `aar_` prefix, both lengths, and at most eight letters or digits after the prefix. Humans, surfaces, relayed and agent principals are not gated. What v1 guarantees:* a human approved an intent for this tool, requested by this principal, and the approval is used once. What it does not: the server does not recompute the arguments digest, so it does not bind the call to the exact approved arguments (the artifact hash is the agent's); the receipt is spent before the handler runs, so a failed call still uses it up; the age of a consumed receipt is not bounded; and only the hosted `/mcp` endpoint is gated, not the stdio `cmd/mcp` server, which reaches the platform over REST. `enforce` without the work-items store refuses startup. An unknown value refuses startup	`off`	No
`MCTL_SURFACE_TELEGRAM_TOKEN` / `MCTL_SURFACE_PORTAL_TOKEN`	Bearer tokens of the per-surface principals `surface:telegram` / `surface:portal` (mctl-api#350): non-admin, no tenant, confined to the surface routes. At least 32 characters; a token shared by two surfaces or equal to `MCTL_AGENT_SERVICE_TOKEN` is refused.	—	No
`MCTL_REGISTRY_PUBLISHER_TOKEN`	Bearer token of the registry-publisher principal `service:mctl-agents-registry-publisher` (mctlhq/mctl-agents#470): not an admin, and may call only `POST /api/v1/agents`, `POST /api/v1/agents/{name}/versions` and `POST /api/v1/agents/{name}/releases` (see docs/agent-platform-registry.md). At least 32 characters; a token equal to `MCTL_AGENT_SERVICE_TOKEN`, a surface token or a writer token is refused. Unset: the principal does not exist and only admins can publish.	—	No
`MCTL_USAGE_WRITER_TOKEN`	Bearer token of the usage-writer principal `service:mctl-agents-usage` (mctlhq/.github#50): holds only `usage:write` and may call only `POST /api/v1/usage/records` (see docs/model-usage-ledger.md). At least 32 characters; a token equal to `MCTL_AGENT_SERVICE_TOKEN` or to a surface token is refused. Unset: only admins can write usage.	—	No
`SURFACE_IDENTITY_DB_URL`	PostgreSQL for surface identity links and challenges, and for the canonical principal model (`principals`, `external_identities`; see docs/principals.md), which mirrors links in the same transaction. Falls back to `AUDIT_DB_URL`; with neither, `/api/v1/surface-identities*` answers 503 and callers carry no principal id.	—	No
`SURFACE_LINK_TTL`	Go duration after which a new link expires (e.g. `2160h`); unset means until revoked. Malformed disables the feature.	unset	No
`SURFACE_IDENTITY_DISABLED`	Kill switch: any value except `false`/`f`/`0`/`no`/`off`.	unset	No
`PRINCIPALS_DISABLED`	Kill switch for the canonical principal model: callers carry no `prn_` id, links are not mirrored, no backfill. Same values as the other kill switches.	unset	No
`ROADMAP_STATE_REPO_URL`	Repository holding the generated roadmap publication read by `/api/v1/roadmap/*` and the `mctl_get_epic_status` / `mctl_get_ready_work_items` tools. mctl-api only reads it; it never evaluates readiness.	`https://github.com/mctlhq/.github.git`	No
`ROADMAP_STATE_BRANCH`	Branch of the publication.	`roadmap-state`	No
`ROADMAP_STATE_LOCAL_PATH`	Local checkout, refreshed every 5 minutes.	`/tmp/roadmap-state`	No
`ROADMAP_STATE_TOKEN`	Credential for a private publication repository; the default repository is public.	—	No
`ROADMAP_STATE_DISABLED`	Kill switch: any value except `false`/`f`/`0`/`no`/`off` leaves the roadmap reader off (routes answer 503).	unset	No
`ROADMAP_WAVE_MAX_AGE`	Maximum age of the latest publication's observation when `POST /api/v1/roadmap/waves/execute` starts a wave (Go duration). An invalid value turns wave execution off.	`30m`	No
`ROADMAP_WAVE_CALL_TIMEOUT`	Per-Temporal-call bound during wave execution (Go duration, `500ms` to `10s`); it sets `max_items_per_execute`. An invalid value turns wave execution off.	`3s`	No
`TRUSTED_PROXY_CIDRS`	Comma-separated Traefik CIDRs/IPs trusted for `X-Forwarded-For` on audit events	—	No
`ALLOW_INSECURE_DB`	Permit `sslmode=disable` (tests/local only)	—	No
`BACKSTAGE_GITHUB_APP_CONNECT_TOKEN`	Backstage service token, scoped to the `github-app-connect` plugin (repos list/sync/install-url)	—	No
`DOMAINS_DB_URL`	PostgreSQL connection string for the custom domains registry (falls back to `AUDIT_DB_URL`). Without either, `/api/v1/domains*` returns 503.	—	No
`GITHUB_WEBHOOK_SECRET`	HMAC secret for `POST /api/v1/webhooks/github` (`X-Hub-Signature-256`). Unset: the webhook answers 401. Pull request events are queued as reference-only envelopes for Claude Remote (mctlhq/.github#87).	—	No
`GITHUB_WEBHOOK_OWNERS`	Comma-separated repository owners whose webhook events are queued; others are ignored.	`mctlhq`	No
`EVENTS_VALKEY_URL`	`redis://user@host:port/db` of the platform Valkey, using the producer's ACL user. Needed together with a database: when the secret is set but Valkey or the outbox database is not configured, the webhook answers 503.	—	No
`EVENTS_VALKEY_PASSWORD`	Password of that ACL user. A URL naming a user without a password is rejected.	—	No
`EVENTS_DB_URL`	PostgreSQL connection string for the event outbox (falls back to `AUDIT_DB_URL`).	—	No
`PLATFORM_DOMAIN`	The platform's own domain. Hostnames equal to or ending in `.<PLATFORM_DOMAIN>` are rejected by `POST /api/v1/domains` — those stay GitOps-only via `ingress.hosts`.	`mctl.ai`	No
`DNS_RESOLVER_ADDR`	`host:port` of the recursive resolver used to verify custom domain TXT/CNAME records	`1.1.1.1:53`	No
`LOKI_URL`	Loki base URL for log queries	—	No

API / Endpoints

Authentication

Bearer tokens are verified by the federation registry (docs/federation.md):

Method	Detection	Validation
MCTL OAuth access token (MCP clients, `mctl auth login`)	JWT whose `iss` is this server	Local signing key; groups re-resolved from gitops (see below)
ZITADEL OIDC JWT	JWT whose `iss` is `auth.mctl.ai`	JWKS and audience (`MCTL_OIDC_PROVIDERS`)
Service and surface tokens	Exact match of a configured static secret	Constant-time comparison
GitHub token (deprecated, #525)	Opaque token no static secret matches	GitHub API

Get a token: sign in with the mctl CLI (mctl auth login, browser sign-in through auth.mctl.ai; see REST API for using it with curl), or visit mctl.ai/mcp to connect an MCP client through OAuth. A GitHub token as the bearer still works but is deprecated and will stop being accepted (#525).

Authorization group freshness

mctl-api#411. Local OAuth 2.0 sessions (the JWTs ValidateJWT verifies, and every refresh-token grant) re-resolve tenant group membership and admin status from the gitops checkout on every use, instead of freezing them at first login:

Grant: create or join a tenant in tenants/<t>/values.yaml, and the next POST /oauth/token refresh or the next authenticated request carries it — no re-login, within the gitops refresh interval (60s) plus OAUTH_GROUPS_CACHE_TTL (30s), so about 90s at worst.
Revoke: remove someone from tenants/<t>/values.yaml, or delete the tenant, and their live session loses access within one OAUTH_TOKEN_TTL-sized "degraded grace" window at worst (see below), not after the full OAUTH_REFRESH_TOKEN_TTL (30 days) the old frozen-groups behaviour allowed.
admins is derived from ADMIN_USERS fresh on every resolution too, never taken from a stored or claimed snapshot: granting or revoking admin status behaves like any other group change.

What happens to a live session when the gitops checkout is unreachable for two hours:

For the first "degraded grace" window (not independently configurable; it always equals OAUTH_TOKEN_TTL, so 1h by default) since the last process-wide successful resolution, the session keeps its previously known tenant groups, with admins still recomputed fresh, and a slog.Warn is logged at most once a minute. The failure itself is cached for 5s, so during an outage the tenant files are re-read at most once every 5s, not on every authenticated request.
Past that window, the session fails closed: it keeps working (never signed out), but with tenant groups dropped, keeping only admins if the login is a configured admin. This is what bounds a resolver outage from defeating a tenant removal for longer than one access-token TTL.
The moment the checkout resolves again, resolution succeeds and the session's real groups are restored immediately — nothing needs a restart or a re-login.

A gitops checkout that is merely stale but present — the far more common case, since a git fetch failure most often means GitHub itself is down, which also blocks pushing a removal — is treated as a successful resolution by default, not a failure: fail-closed after 15 minutes would lock every tenant user out about 75 minutes into a GitHub incident, for no security benefit, since the same outage blocks the removal that would defeat. It is logged (rate-limited to once a minute) and exposed as the mctl_api_gitops_last_sync_age_seconds gauge so it can be alerted on (-1 while the checkout has never synced). Set OAUTH_GROUPS_MAX_STALENESS to opt into strict mode, where a checkout older than that duration is treated as a failed resolution instead — a stricter operator setting always wins over the fixed 15-minute default.

A suggested alert:

mctl_api_gitops_last_sync_age_seconds > 900

Per-request group resolution does not block on the gitops reader. A refresh holds the reader's write lock while it runs git fetch/reset (each git command is bounded by a 2-minute timeout). Within the degraded grace window, a lookup that would have to wait instead uses the session's own groups for that request, without a warning (a busy reader says nothing about its health). Beyond grace, the session snapshot is no longer trusted: the request retries for up to 2s, so a normal short refresh still answers correctly, and if the reader is still busy it fails closed like any other failure, with the rate-limited warning. A busy result is not cached. Login itself waits out a refresh and ignores the 5s failure cache, because the groups resolved at login become the refresh-token snapshot that later fallbacks rely on. LastSync() and the sync-age gauge are lock-free, so /metrics keeps answering during a hung fetch.

A never-synced checkout (gauge reads -1, or LastSync() is the zero value) is always a failed resolution, never "this user has no tenants" — that distinction is what stops a pod whose first gitops clone failed from silently stripping every session's access.

REST API

# Health check
curl https://api.mctl.ai/healthz

# Sign in once with the mctl CLI (mctl-gitops cli/mctl): browser sign-in
# through auth.mctl.ai. `mctl auth status` renews an expired access token.
mctl auth login
mctl auth status
# The access token is stored in <user config dir>/mctl/api-token.json:
# ~/Library/Application Support/mctl on macOS, ~/.config/mctl on Linux.
TOKEN=$(jq -r .token.access_token "$HOME/.config/mctl/api-token.json")

# List tenants
curl -H "Authorization: Bearer $TOKEN" https://api.mctl.ai/api/v1/tenants

# List services
curl -H "Authorization: Bearer $TOKEN" https://api.mctl.ai/api/v1/services

# Service status
curl -H "Authorization: Bearer $TOKEN" https://api.mctl.ai/api/v1/status/billing/payment-api

# Service logs (last 50 lines, past 30 minutes)
curl -H "Authorization: Bearer $TOKEN" \
     "https://api.mctl.ai/api/v1/logs/billing/payment-api?lines=50&since=30m"

# Trigger a deploy
curl -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"team_name":"billing","component_name":"payment-api","action":"deploy","git_tag":"v1.2.0"}' \
     https://api.mctl.ai/api/v1/operations/deploy-service/execute

Agent Registry

The agent registry (internal/agentregistry) has two layers on the same Postgres-backed store:

v1 — image/release: POST /api/v1/agents/{name}/versions, POST /api/v1/agents/{name}/releases, GET /api/v1/agents/{name}/resolve?environment= — versions a single agent image/release (agent_versions + agent_releases). Unchanged by the v1alpha2 layer below.
v1alpha2 — definition/profile/binding: GET /api/v1/agents, GET /api/v1/agents/{name}, POST /api/v1/agents/{name}/definition-versions, POST /api/v1/agent-profiles/{profile}/versions, POST /api/v1/agents/{name}/bindings, GET /api/v1/agents/{name}/bindings/resolve, POST /api/v1/agents/{name}/bindings/rollback — immutable AgentDefinition and ExecutionProfile versions plus an append-only ReleaseBinding ledger per (agent, environment), per ADR 007. See docs/agent-platform-registry.md for the full contract (compatibility-range grammar, error codes, gitops reconciliation).

Both layers are admin-only and share the seven v1 MCP tools plus seven new v1alpha2 tools (mctl_publish_agent_definition_version, mctl_publish_agent_profile_version, mctl_set_agent_version_lifecycle, mctl_bind_agent_release, mctl_rollback_agent_binding, mctl_list_agents, mctl_get_agent); mctl_resolve_agent gained an optional api_version argument selecting between the two layers.

Work Items

workitem/v1 (docs/work-context-contract.md) is the durable record of a piece of work a human asked for, shared by every surface. Surfaces correlate to it; they never own its lifecycle.

POST /api/v1/work-items opens one (idempotent per tenant Idempotency-Key, dedupes open work on external_key); GET /api/v1/work-items[/{id}] reads.
PATCH /api/v1/work-items/{id} completes, archives, supersedes or parks it (wait); POST .../resume starts the next execution. Both require expected_state_version and answer 409 with the current state on a mismatch.
POST .../intents (≤ 8 KiB, secret-scanned), GET|POST .../executions, POST .../surface-refs, GET .../events.
POST .../execution-requests asks the platform to start or resume the item without naming an engine run (mctl-api#368): a person or a relaying surface requests, the service principal claims it (POST /api/v1/execution-requests/claim) and fulfils it (POST /api/v1/execution-requests/{request_id}/fulfil {claim_token, engine, engine_ref}) or rejects it. A surface relays the request routes, never /resume.
POST .../executions/{execution_id}/snapshot seals the execution's ContextSnapshot (service principal only). The table is insert-only: one snapshot per execution, the same bytes again return the stored one, and different bytes answer 409 snapshot_divergence. GET the same path, GET .../snapshots[/{snapshot_id}] to read; the item view carries latest_snapshot.
POST /api/v1/action-approvals (service principal only) requests the single-use approval of one hashed runtime side effect; a human admin answers it with POST .../{id}/decision, and the requesting service spends it once with POST .../{id}/consume {intent_hash}, then marks it executed once with POST .../{id}/execute {target, artifact_hash}. GET /api/v1/action-approvals[/{id}] reads, with lazy expiry (mctl-api#366).
The acting principal is always the authenticated caller. A body naming an actor (actor, owner_principal, on_behalf_of, ...) gets 400 actor_not_accepted; a surface acting for its users needs mctl-api#350.

OpenAPI Documentation

URL	Description
`https://api.mctl.ai/openapi.yaml`	Raw OpenAPI 3.0 spec
`https://api.mctl.ai/docs`	Interactive Swagger UI

MCP Setup

The MCP endpoint is https://api.mctl.ai/mcp (Streamable HTTP). Clients connect with the URL alone and sign in through MCTL (auth.mctl.ai) with the standard MCP authorization flow (OAuth 2.1, PKCE, dynamic client registration). No config carries a token, and no Client ID or Secret is needed. The same per-client steps are on docs.mctl.ai. A GitHub token as the bearer is deprecated (#525).

Claude.ai and Claude Desktop: Settings → Connectors → Add custom connector, URL https://api.mctl.ai/mcp, leave the OAuth Client ID and Secret empty, then Connect. A connector added on Claude.ai shows up in Claude Desktop too.

Claude Code:

claude mcp add --transport http mctl https://api.mctl.ai/mcp

Then run /mcp, choose mctl → Authenticate.

Cursor: add to ~/.cursor/mcp.json (or Cursor Settings → MCP → Add server), then sign in to mctl in Cursor's MCP settings:

{
  "mcpServers": {
    "mctl": { "url": "https://api.mctl.ai/mcp" }
  }
}

VS Code: create .vscode/mcp.json, start the server from the file and accept the sign-in prompt (needs a recent VS Code with GitHub Copilot Chat):

{
  "servers": {
    "mctl": { "type": "http", "url": "https://api.mctl.ai/mcp" }
  }
}

Gemini CLI: add to ~/.gemini/settings.json, then run /mcp auth mctl. The sign-in needs a browser on the same machine, so it does not work over SSH.

{
  "mcpServers": {
    "mctl": { "httpUrl": "https://api.mctl.ai/mcp", "trust": true }
  }
}

Windsurf and Copilot CLI: MCTL sign-in is not enabled for them yet. Until it is, connect from Claude Code, Cursor, VS Code or Gemini CLI.

Other clients: any client that implements MCP authorization finds the sign-in on its own from the first 401 (discovery: https://api.mctl.ai/.well-known/oauth-protected-resource). Clients without it cannot connect. Continue.dev has not been tested with MCTL sign-in.

Local stdio binary (development): cmd/mcp serves the same tools over stdio and forwards each call to the REST API with the token in MCTL_API_TOKEN, for example the access token from mctl auth login (see REST API; it expires, so refresh it with mctl auth status).

go install github.com/mctlhq/mctl-api/cmd/mcp@latest

{
  "mcpServers": {
    "mctl": {
      "command": "/Users/<you>/go/bin/mcp",
      "env": { "MCTL_API_URL": "https://api.mctl.ai", "MCTL_API_TOKEN": "<access token>" }
    }
  }
}

MCP Tools

Read operations (safe, no side effects):

Tool	Description
`mctl_list_tenants`	List all team workspaces with quotas
`mctl_list_services`	List catalogue services and ArgoCD applications in a team's namespace, optional team filter
`mctl_get_tenant`	Get workspace details and members
`mctl_get_service_status`	ArgoCD health + sync state
`mctl_get_service_config`	Full service config from GitOps repo
`mctl_get_resource_usage`	Live CPU/memory/pods from K8s ResourceQuota
`mctl_get_service_logs`	Recent log lines from Loki

Write operations (trigger Argo Workflows):

Tool	Description
`mctl_deploy_service`	Onboard, deploy, or update-config a service
`mctl_create_tenant`	Create team workspace (namespace, quotas, Vault)
`mctl_provision_database`	Provision PostgreSQL on shared CNPG cluster
`mctl_rollback_service`	Roll back to a previous image tag
`mctl_retire_service`	Permanently remove a service
`mctl_create_preview` / `mctl_delete_preview`	Ephemeral preview environments
`mctl_add_custom_domain` / `mctl_remove_custom_domain`	Custom domain management
`mctl_list_repos` / `mctl_sync_repos` / `mctl_grant_repo_access`	Repository access

Testing

# Unit tests
make test

# Lint
make lint

# End-to-end tests (requires running platform)
go test -tags=e2e ./e2e/ -run TestE2E_FullPlatformSmokeTest

Linting uses golangci-lint with errcheck, govet, staticcheck, gosec, bodyclose, and other checkers configured in .golangci.yml.

CI/CD

GitHub Actions:

validate.yml (pull requests): golangci-lint, go build, go test
security.yml: govulncheck and a Trivy filesystem scan (CRITICAL, fail closed)
release-please.yml: tagged releases, GHCR image, gitops bump
Registry: ghcr.io/mctlhq/mctl-api
Dependabot: weekly go.mod, GitHub Actions, and Dockerfile updates

Deployment

Deployed to Kubernetes via Helm chart (helm/). ArgoCD syncs the desired state from the mctl-gitops repository.

Domain: api.mctl.ai

Vault secrets required before first deploy:

# ArgoCD API token
vault kv put platform/mctl-api/argocd-token token="<token>"

# Backstage service token
vault kv put platform/mctl-api/backstage-token token="<token>"

Release Process

git tag 0.2.0 && git push origin 0.2.0
# → CI builds ghcr.io/mctlhq/mctl-api:0.2.0
# → CI commits new tag to mctl-gitops → ArgoCD deploys

Tags use no v prefix. Every push to main also builds a latest image for local development.

Related Projects

Repository	Description
mctl-gitops	GitOps repository — Helm values, ArgoCD app definitions
mctl-portal	Web portal for token management and MCP config generation
mctl-web	Marketing site (mctl.ai)
mctl-agent	Platform agent components

License

Apache 2.0

Portal tool allowlist

api.mctl.ai/mcp is one upstream of the private aggregate portal mcp.mctl.ai (mctlhq/.github#35). The portal exposes a tool only when its mapping carries an explicit enabled: true for it and hides one it has an entry for with enabled: false; the mapping is kept at default_disabled: true plus an explicit entry for every tool. docs/portal-allowlist.json is the source of truth; internal/mcp/portal_allowlist_test.go fails the build when the set of registered tools differs from the file, when an enabled tool carries no reason, or when an enabled tool that is not recorded read-only in internal/mcp/annotations_test.go is missing from the mutatingOnPortal map in that test. Exposing a mutating tool is therefore a reviewed Go change naming what it changes, not a JSON edit — and it grants visibility only: the API's own authentication, team scope and role checks decide whether the call succeeds. Publishing the decision is automatic. When a change to docs/portal-allowlist.json lands on main, .github/workflows/portal-allowlist-dispatch.yml sends the file to mctlhq/mctl-gitops. There it is vendored byte-identical and opened as a PR whose cloudflare-plan names every tool that flips; the portal changes only when that PR merges and an approved cloudflare-apply runs (mctlhq/mctl-gitops#1370). Nothing in this repository talks to Cloudflare any more. The gitops side refuses a file that leaves a synced tool undecided, lets a widening through only once a human edits its baseline.json in that PR, and checks nightly that its vendored copy still matches this file. A portal access token carries the tool set as of its issuance; re-issue tokens after the apply.

Health endpoints

GET /healthz is liveness. GET /readyz returns 503 only when a store failed its startup init, the pod's own gitops checkout has never synced (checks.gitops_sync=never_synced), or the pod is draining on shutdown; dependency probe failures (gitops, postgres, dex, vault) never fail readiness because every replica shares them. Dependency health is reported in the /readyz body (checks, dependencies) and in the mctl_api_dependency_up{check} gauge.

Name		Name	Last commit message	Last commit date
.github		.github
cmd		cmd
docs		docs
e2e		e2e
helm		helm
internal		internal
scripts		scripts
.env.example		.env.example
.gitignore		.gitignore
.golangci.yml		.golangci.yml
.release-please-manifest.json		.release-please-manifest.json
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
LLMS.md		LLMS.md
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
context7.json		context7.json
go.mod		go.mod
go.sum		go.sum
mcp-config.example.json		mcp-config.example.json
release-please-config.json		release-please-config.json

Navigation Menu

Repository navigation

About

Repository files navigation

mctl-api

What It Does

Architecture

Tech Stack

Project Structure

Getting Started

Prerequisites

Local Development

Docker

Configuration

Environment Variables

API / Endpoints

Authentication

Authorization group freshness

REST API

Agent Registry

Work Items

OpenAPI Documentation

MCP Setup

MCP Tools

Testing

CI/CD

Deployment

Release Process

Related Projects

License

Portal tool allowlist

Health endpoints

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

mctl-api

What It Does

Architecture

Tech Stack

Project Structure

Getting Started

Prerequisites

Local Development

Docker

Configuration

Environment Variables

API / Endpoints

Authentication

Authorization group freshness

REST API

Agent Registry

Work Items

OpenAPI Documentation

MCP Setup

MCP Tools

Testing

CI/CD

Deployment

Release Process

Related Projects

License

Portal tool allowlist

Health endpoints

About

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages