Docs audit — 2026-08-20 (PRs #2740–#2743)

[SebConejo]

Audited: mnfst/manifest PRs #2740 to #2743 (3 merged PRs: #2740, #2742, #2743), against the docs at 24ee403.

Findings

1 · [MISSING-CASE] The page says a fallback starts on an HTTP status of 400 or more. A provider answer that succeeds but carries no content now starts one too, and the page does not say so

Page: https://manifest.build/docs/llm-gateway (Fallback, the "What triggers a fallback" section)

What actually happens:

1. Your agent sends a request. Manifest routes it to a model.
2. The provider answers 200 OK, but the answer carries no text and no tool call.
3. Manifest treats that answer as a provider failure and moves the request to the next fallback model in the tier.
4. If a fallback model is left, your agent gets that model's answer and sees no error.
5. If no fallback model is left, or if the request pinned one model, your agent gets HTTP 502 with the message Upstream provider returned bad gateway. The provider's own dashboard shows a 200 for that same call.
6. On the Requests page, the attempt shows as Failed with status 502, and the error card carries Upstream provider returned an empty Chat Completions response. That message never reaches your agent.

Options:

1. Add one sentence to the trigger section, after the status table. Recommended (it gives the trigger and the outcome, and it stays true on every route).
2. Add that sentence plus its scope. Accurate, but it puts back the streaming scope you rejected on issue Docs audit — llm gateway (routing, tiers, fallbacks) — 2026-08-19 #82 finding 2.
3. Change nothing. Readers with a fallback model never notice, and the dashboard already prints the reason.

Suggested wording (option 1), as a new paragraph after the status table:

A provider that returns a success status with an empty response can trigger a fallback as well.

Suggested wording (option 2), same place:

A provider that returns a success status with an empty response can trigger a fallback as well. Manifest runs that check on non-streaming calls to /v1/chat/completions, against endpoints that speak the OpenAI format.

Evidence (for the agent)

• llm-gateway.mdx:97 — "Any HTTP status code >= 400 triggers a fallback." Section heading at :95, status table at :99-108, lowest row 400.
• llm-gateway.mdx:93 — "When a model fails (provider outage, rate limit, bad request), Manifest retries with a backup model from the same tier."
• Corpus grep for empty across *.mdx, 14 hits: the word is used about a payload on the M002 and M300 pages and their rows (errors/M002.mdx, errors/M300.mdx, errors.mdx:28,61, errors/M001.mdx:29). The rest mean an empty instance or harness (introduction.mdx:81, self-hosted.mdx:128, errors/M101.mdx:7) or a deploy option (deploy/coolify.mdx:41). No page says a successful answer can be treated as a failure.
• Code, PR #2743 at clone HEAD 061d351f5: provider-client.ts:136-153 — a 200 body whose choices all carry finish_reason: 'stop', a message object and no output field (every key but role null, empty string or empty array) is detected as empty.
• provider-client.ts:155-178 — Manifest records the real upstream body against the attempt, then returns a synthetic 502 with body {"error":{"message":"Upstream provider returned an empty Chat Completions response","type":"server_error","code":"empty_response"}}.
• provider-client.ts:422-425 — the gate: !stream && endpoint.format === 'openai' && (opts.apiMode === undefined || opts.apiMode === 'chat_completions'). Callers on /v1/messages and /v1/responses are not covered (proxy.controller.ts:205-226).
• fallback-status-codes.ts:1-3 (status >= 400) and proxy.service.ts:501-508 — the synthetic 502 enters the fallback chain like any provider error.
• Step 5, pinned-model or no-chain branch: proxy.service.ts:502 (!explicitModelOverride) and proxy-response-handler.ts:305-319 (source: 'provider' at :313).
• Step 5, exhausted-chain branch: proxy-response-handler.ts:398-425 — the response keeps the primary's 502 (:403) and sets X-Manifest-Fallback-Exhausted: true (:405).
• Both branches collapse to the same text: proxy-error-sanitizer.ts:88-92 (parseStructuredProviderError returns null for status >= 500) and :191-201 (production returns the generic Upstream provider returned bad gateway). So empty_response never reaches the client.
• proxy-message-recorder.ts:694-699 — the attempt's error_message stores the synthetic body verbatim. :979-989 writes the same body on a primary failure that a later fallback recovers.
• RequestDrawer.tsx:82-97 — the attempt renders as Failed with the stored HTTP status. :637-682 — the Error card prints the stored body, above an Origin / Type / HTTP status table.
• Scope of the change: buildCustomEndpoint defaults to format: 'openai' (provider-endpoints.ts:585-606) and ollama is format: 'openai' (:516-522), so custom OpenAI-compatible providers and Ollama are covered. Anthropic-format, Google/Vertex-format and Responses-format endpoints are not.
• Impact: a reader with no fallback model configured gets a 502 for a call the provider reports as a success, and the page gives no way to connect the two.
• Ordering: docs PR docs: gateway routing and fallback boundaries (issues #82, #84) #89 (issues Docs audit — llm gateway (routing, tiers, fallbacks) — 2026-08-19 #82 and Docs audit — 2026-08-19 (PRs #2734–#2738) #84, not merged) adds a paragraph to this same section, about a tier whose pinned model leaves the model list. This finding is a different trigger, a different signal and a different outcome, but it edits the same block. The suggested sentence is written to sit after PR docs: gateway routing and fallback boundaries (issues #82, #84) #89's paragraph, so the two ship in one wave or this one rebases onto it.

2 · [IMAGE] The Usage-based screenshot shows 23 provider cards. The dashboard now shows 24: Google Vertex AI is missing from the picture

Page: https://manifest.build/docs/providers/api-key-providers (the frame under "Add a provider", captioned "The Usage-based page: one card per provider, each with a Connect button.")

Evidence (for the agent)

• The shot: the Providers → Usage-based page with the full "Supported usage-based providers" grid, including the Google Vertex AI card with its Vertex logo, between Google and Groq. It replaces images/providers-usage-based.png in place. The frame's caption and the alt text do not change.
• providers/api-key-providers.mdx:69-71 — the <Frame caption="The Usage-based page: one card per provider, each with a Connect button."> around <img src="/images/providers-usage-based.png" ... />, under ## Add a provider at :52.
• images/providers-usage-based.png, captured 2026-08-18: 23 cards over 8 rows, Alibaba Cloud to Z.ai, with Google and Groq adjacent in row 3.
• The grid renders PROVIDERS.filter((provider) => !provider.subscriptionOnly && !provider.localOnly) (ProviderConnectionsPage.tsx:123). PROVIDER_ORDER holds 35 ids (frontend/src/services/providers.ts:505-541); localOnly comes from the shared registry through buildProviderDef (frontend/src/services/providers.ts:486-500, packages/shared/src/providers.ts:371,384,397) and marks 3 ids; the PROVIDER_UI map marks 8 ids subscriptionOnly (frontend/src/services/providers.ts:194,217,242,261,288,384,406,447). 24 cards render today, and those 24 are the PNG's 23 plus vertex.
• vertex sits between gemini and groq in that order (frontend/src/services/providers.ts:517-519), and is a real API-key card (packages/shared/src/providers.ts:105-116, requiresApiKey: true, localOnly: false).
• PR #2740 is why this is raised now: ProviderIcon.tsx:204 returns the Vertex mark as inline SVG, with packages/frontend/public/icons/vertex.svg. The re-shoot was parked during the #2736 audit for two reasons: the caption sells page shape rather than an inventory, and the card rendered as a bare letter tile until its icon PR landed. The second reason has lapsed. The first still argues for leaving the picture alone, which is why this is a finding you can decline.
• Docs PR docs: Google Vertex AI (issue #85) #91 (issue Docs audit — 2026-08-19 (PRs #2736–#2741) #85, not merged) adds the Google Vertex AI rows to this page's tables without touching the image. Shooting into that PR keeps the page consistent on the day it merges.

Questions, not findings

1. glm-5.3 was added to the Z.ai GLM Coding Plan model list (PR #2742), but it was never confirmed against a live Z.ai request: the platform PR's own smoke test hit HTTP 429, "Insufficient balance or no resource package". If Z.ai does not serve that id, a user who pins it gets a provider error at chat time. No docs page names it, so no doc change waits on the answer. One routed request on a GLM Coding Plan connection settles it.
2. The exact message a Cloud caller sees when a fallback chain runs out depends on the deployed NODE_ENV (proxy-error-sanitizer.ts:191-201). The code branch is verified, and every documented install pins it to production (docker/Dockerfile:77, docker/docker-compose.yml:80). The deployed Cloud value is not verified. Finding 1's wording does not depend on it.

Cleared this run (recorded so it is not re-derived)

• PR #2742 (glm-5.3 added to the Z.ai plan list): no finding. Listing what another vendor's plan serves is what you rejected on 2026-08-20 for issue Docs audit — 2026-08-19 (PR #2733) #81. The Manifest-side fact it exposes, that a new plan model does not reach an existing connection on its own, is already the subject of providers/subscription-based-providers.mdx:56. That sentence has its own accuracy problem, parked in the backlog for the providers audit; this window does not change it.
• reference/api.mdx:208 — "| 5xx | Upstream provider error (triggers fallback) |" survives PR #2743: the synthetic 502 carries source: 'provider' and the standard bad-gateway message.
• autofix.mdx:28 — "It deliberately skips 401, 403, 429, and every 5xx. A bad key, a rate limit, or a provider outage isn't something a rewritten body can fix, so those go straight to fallback." stays true: the empty-completion 502 is never repaired (autofix.service.ts:75-86 filters to 400-499).
• request-logs.mdx:25 — "| Response | The parsed JSON body, or for a streamed attempt, the raw SSE stream as it arrived |" stays true: the recorder stores the provider's real empty body, not the synthetic error (provider-client.ts:168).
• Internal-leak sweep: clean. No corpus hit for any internal name on the paths this window touches. Every operator environment variable named in the docs stays inside a self-hosted section.
• observability.mdx:27 ("a failed attempt can still burn tokens"): not raised, because issue Docs audit — observability (requests page, analytics) — 2026-08-19 #83 finding 2 is open on that same line. One constraint for whoever writes the replacement: since PR #2743 a provider can run a call to completion, invoice it, and have Manifest record zero tokens and zero cost. The replacement must not swing to "a failed attempt costs you nothing".

---

Reply with the finding number and the chosen option
(e.g. "3 → option 2", or "3: other idea, let's discuss").
PRs are then made manually with the agent, never by the harness.
Rejected findings: reply "N → reject" and they will never be re-raised.