diff --git a/apps/website/content/docs/chat/guides/generative-ui.mdx b/apps/website/content/docs/chat/guides/generative-ui.mdx index 5aef319f4..1a5014e96 100644 --- a/apps/website/content/docs/chat/guides/generative-ui.mdx +++ b/apps/website/content/docs/chat/guides/generative-ui.mdx @@ -4,65 +4,65 @@ description: How the generative UI example authors a dashboard spec as message c # Generative UI -An agent can return the interface rather than describe it. It writes a declarative json-render specification as the assistant message content, and the `` composition detects that content, parses it while it is still streaming, and mounts it through the `views` registry you supply. The running example is an airline operations dashboard, and this guide walks the three files that build it. +An agent can return an interface rather than describe it. In this example, the model proposes a declarative json-render layout and the backend validates the operation before publishing the complete specification as assistant message content. The `` composition classifies that content and mounts it through the `views` registry you supply. The running example is an airline operations dashboard. -This default Angular guide describes progressive rendering. Select **React** in Example UI for the experimental native preview, which publishes confirmed layouts and data together after saved-state verification. +Angular is the default Example UI. Its render surface can update progressively as message content and state events arrive. Select **React** for the experimental native preview, which publishes confirmed layouts and data together only after verifying the exact saved checkpoint and tool evidence. Graph-state publication does not guarantee that browser painting happens atomically. -The detection and the rendering are adapter-neutral. This page uses `@threadplane/langgraph`; the same registry and the same store work over the [AG-UI adapter](/docs/ag-ui/guides/json-render), where only the way the backend delivers the data differs. +The detection and rendering are adapter-neutral. This page uses `@threadplane/langgraph`; the same registry and store work over the [AG-UI adapter](/docs/ag-ui/guides/json-render), where the backend delivers data differently. ## What the demo does -The Run tab shows the prebuilt `` composition. Choose the welcome suggestion "Airline operations dashboard" and the agent calls one tool to author the layout — four KPI cards, a line chart, a bar chart, and a table — and, in the same turn, the data tools that back it. The cards mount as skeletons and take their values when the data arrives. +The Run tab shows the prebuilt `` composition. Choose "Airline operations dashboard" or write your own request. The model authors a layout from six supported view types: `dashboard_grid`, `container`, `stat_card`, `line_chart`, `bar_chart`, and `data_grid`. A layout may use any valid subset of this catalog; it does not need six elements. -The second suggestion, "Filter to cancelled flights", is the follow-up worth trying. The agent does not author a new layout for it. It calls one data tool again with new arguments, the new result replaces that slice of the store, and the dashboard already on screen updates in place. +Try "Filter to cancelled flights" after creating a dashboard. A data update executes one query and replaces its validated dashboard slice while keeping the existing layout and its render owner. A request to change the arrangement instead requires a new layout. Interpretive questions execute no dashboard tools and answer from confirmed data. Unsupported requests make no changes. -## How it is built - -Three files carry the feature: a LangGraph graph that authors the layout and streams the data, an application config that registers the agent, and a component that supplies the view registry and the store. Open the Code tab to read them in full. +The model still interprets freeform language, so the selected category and interpretive prose are probabilistic. The backend's independent validation bounds what can execute; it does not guarantee that the model understood every request correctly. -### The tool that authors the layout +## How it is built -The layout arrives as a tool call. `render_spec` takes the flat element dictionary and the id of the root element, and returns them serialized as JSON. +The Code tab includes four backend modules. `graph.py` owns the turn, messages, publication, and response. `dashboard_tools.py` supplies the airline queries. `operations.py` admits bounded operations against the accepted prior layout, and `dashboard_contract.py` validates layouts, data, and bindings. The prompt is a separate asset. The frontend config registers the agent, and its component supplies the view registry and store. - +### Choosing a bounded operation -The docstring is the contract the model reads, which is why it says to call the tool at most once per turn: layout changes go through this tool, and data refreshes go through the data tools instead. +Every turn requires one named `plan_dashboard_operation` decision: `create`, `data_update`, `restructure`, `interpret`, or `unsupported`. The decision itself executes nothing. Independent admission checks the category, tool arguments, prior layout, and bindings before the real tool node runs a single batch. -### Moving the specification into the message +A data update requires an existing accepted layout and exactly one query. Create and restructure require one `render_spec` call plus only the queries needed for bindings missing from the applicable data, with at most five unique calls in total. Interpret and unsupported decisions have no calls. There is no agent retry loop or iteration cap. -A tool result is not message content, so a post-processing node moves it. `wrap_spec_into_ai` finds the most recent `render_spec` tool message and the assistant message whose tool call produced it. +### The tool that authors the layout - +`render_spec` returns a flat element dictionary and root id as serialized JSON. The model proposes the layout; the graph controls whether it becomes a published dashboard. -The node returns early when this turn produced no `render_spec` result at all; the guard that makes it safe on every pass of the loop is the content check in the next region. + -### Rewriting the message in place +### Moving the accepted specification into the message -The rest of `wrap_spec_into_ai` replaces both messages, keeping their ids so LangGraph's `add_messages` reducer matches and replaces rather than appends. The early return on an already-filled parent message is what stops a second pass from wrapping the same payload twice. +The graph assesses the entire current tool batch before wrapping a layout or emitting data. Every result must belong to the admitted call and graph-owned action parent, report success, and satisfy the dashboard contract. A layout must match the admitted arguments and resolve its bindings against the candidate data. A failed or malformed result prevents partial dashboard publication, retaining the prior board and genuine raw tool results. - +Only an accepted, correlated render result can move into its assistant parent. `wrap_spec_into_ai` preserves message ids, transfers the complete JSON text into the empty parent, and changes that render tool result to `rendered`. Data-only updates leave the prior render pair and owner intact. -The tool message becomes a short placeholder and the assistant message takes the specification JSON, stripped of a code fence if the model wrapped one around it. + - -The specification travels as message content because that is what the client classifies. `DashboardState` extends `MessagesState` with persisted dashboard data and completion evidence. The layout stays in the message. The same tool data is saved in the dashboard field and sent through the custom updates below, which keep the Angular views progressive. + +The assistant message contains the JSON that the client classifies and renders. The graph also persists an owned layout, its exact render owner, dashboard data, and an internal operation receipt. That receipt records the submitted human message, admitted calls, observed results, validated slices, and outcome; it is completion evidence rather than a separate public receipt API. -### Sending the tool data as state updates +### Sending accepted data as state updates -The data tools return their results as tool messages, which the browser would otherwise only see as tool output. `emit_state` processes successful known tool results from the current turn in chronological order, with the newest result for each dashboard slice winning. It turns each result into JSON Pointer paths, and writes them to the stream as one custom event named `state_update`. +`emit_state` rechecks the accepted messages and results before recording the dashboard, layout ownership, and operation receipt together in graph state. Only validated slices from an applied batch produce a custom `state_update` event. The newest validated result for a slice defines its value; admission forbids duplicate query names within an executable batch. -The keys it builds, such as `/on_time/value` and `/recent_disruptions`, are exactly the pointers the specification binds to with `$state`. +The event contains JSON Pointer keys such as `/on_time/value` and `/recent_disruptions`, matching the specification's `$state` bindings. The Angular store applies these events as they arrive. React additionally verifies the saved checkpoint and exact tool evidence before publishing a confirmed board. -### The graph wiring +### Outcomes and graph wiring -The wiring shows the loop. `agent` calls the tools and the `tools` node runs them; `wrap_spec_into_ai` post-processes the result and returns to the agent; the turn ends through `emit_state`, a short conversational summary, and background title generation. `should_continue` routes to `finalize` instead of `tools` when the six-iteration cap is hit, so orphaned `tool_calls` are stripped from the last AI message before the turn ends. See the [persistence guide](/docs/langgraph/guides/persistence) for the no-checkpointer rule the last line follows and its one exception. +The flow is prepare, required decision, admission, bounded execution when needed, whole-batch assessment, wrapping, state publication, response, and best-effort title generation. The graph owns the action, call, result, and answer identities. Its input schema accepts messages; the LangGraph API server supplies persistence, so the graph calls `compile()` without an explicit checkpointer. -The last line calls `compile()` with no checkpointer, because this graph is served by the LangGraph API server, which supplies persistence itself. +Action responses are derived from the executed receipt: an applied operation reports the actual layout or stored data, while rejected decisions, unsupported requests, and unfulfilled operations report no dashboard changes. Read-only interpretation uses model prose followed by an explicit no-change footer. Title generation is best effort and does not stream into the action response. + +A malformed decision with no executable calls can be saved as an honest no-op and the conversation can continue. A real failed or malformed tool batch does not become a confirmed native turn merely because the backend retained its prior state or recorded an outcome. The React preview fails closed, keeps its prior confirmed board, and requires **New** before further submissions when saved evidence cannot be confirmed. ### Registering the agent @@ -87,9 +87,9 @@ Both travel as inputs on ``: `[views]` for the registry and `[store]` for The pieces meet in the browser in a fixed order. -The content classifier in `@threadplane/chat` inspects each assistant message as it streams and decides from the start of the content: a `{` is a json-render specification, the `---a2ui_JSON---` sentinel is A2UI, and anything else is markdown. That is why the system prompt insists on raw JSON with no prose and no code fence around it. +The content classifier in `@threadplane/chat` inspects each assistant message as it streams and decides from the start of the content: a `{` is a json-render specification, the `---a2ui_JSON---` sentinel is A2UI, and anything else is markdown. This backend supplies complete validated JSON through the correlated wrapper, rather than streaming raw model-authored JSON into the render surface. -From there the content goes through `@cacheplane/partial-json`, and a `ParseTreeStore` materializes a best-effort `Spec` from the parse tree on every chunk, so a signal holds a usable specification long before the JSON closes. The composition mounts that specification as ``, which renders it through `` from `@threadplane/render` using the registry you passed. The surface also receives a `loading` input, true while the agent is still streaming, which a view component can read to draw a placeholder. +For general streaming specifications, the content goes through `@cacheplane/partial-json`, and a `ParseTreeStore` materializes a best-effort `Spec` from the parse tree on every chunk, so a signal holds a usable specification long before the JSON closes. The composition mounts that specification as ``, which renders it through `` from `@threadplane/render` using the registry you passed. The surface also receives a `loading` input, true while the agent is still streaming, which a view component can read to draw a placeholder. The data takes the other path. The LangGraph adapter surfaces a custom event named `state_update` on the agent's event stream, and the composition writes its payload straight into the render store, keys included. Because `emit_state` already emits JSON Pointer keys, a binding of `{ "$state": "/on_time/value" }` resolves the moment the event lands. @@ -132,7 +132,7 @@ export class StatCardComponent { } ``` -Give every input a default. A prop bound with `$state` is `undefined` until its event arrives, and a prop that is still streaming can be a partial string, so a component that requires its inputs will fail on the first frame. +Give every input a default, including when using the renderer with other backends that stream partial specifications. A prop bound with `$state` is `undefined` until its event arrives, and a prop that is still streaming can be a partial string, so a component that requires its inputs will fail on the first frame. The specification that drives it is flat. One `root` id and one `elements` map, each element naming a `type`, optional `props`, and optional `children` as an array of ids: @@ -157,7 +157,7 @@ The specification that drives it is flat. One `root` id and one `elements` map, } ``` -The full format, including `$bindState`, `$computed` and visibility conditions, is covered in [Specs and elements](/docs/render/guides/specs). +The general render format, including `$bindState`, `$computed` and visibility conditions, is covered in [Specs and elements](/docs/render/guides/specs). ## Event handlers @@ -171,7 +171,7 @@ protected readonly handlers = { }; ``` -An element wires it as `"on": { "click": { "action": "submitForm", "params": {} } }`. Handlers may return a value or a `Promise`, which lets an action await a network call before the surface advances. The dashboard example registers no handlers, because its dashboard is read-only. See [Events and actions](/docs/render/guides/events) for the dispatch rules. +An element wires it as `"on": { "click": { "action": "submitForm", "params": {} } }`. Handlers may return a value or a `Promise`, which lets an action await a network call before the surface advances. The general render library supports actions and computed expressions. This bounded dashboard demo registers no handlers and rejects action bindings and computed expressions in model-authored layouts. See [Events and actions](/docs/render/guides/events) for the dispatch rules. ## A2UI protocol diff --git a/apps/website/e2e/react-chat-generative-ui-preview.spec.ts b/apps/website/e2e/react-chat-generative-ui-preview.spec.ts index 6612d8bb6..1ad1d878a 100644 --- a/apps/website/e2e/react-chat-generative-ui-preview.spec.ts +++ b/apps/website/e2e/react-chat-generative-ui-preview.spec.ts @@ -99,6 +99,17 @@ test('native Chat Generative UI keeps canonical Docs, Code and Run aligned', asy await expect(page.getByRole('tabpanel')).toContainText( 'createConnectedApplication' ); + for (const name of ['graph.py', 'dashboard_tools.py', 'operations.py', 'dashboard_contract.py'] as const) { + const file = page.getByRole('complementary', { name: 'File tree' }).getByRole('button', { name, exact: true }); + await expect(file).toBeVisible(); + await file.click(); + await expect(page.getByRole('tabpanel')).toContainText({ + 'graph.py': 'plan_dashboard_operation', + 'dashboard_tools.py': 'query_recent_disruptions', + 'operations.py': 'def admit(', + 'dashboard_contract.py': 'def validate_spec(', + }[name]!); + } await expect(page).toHaveURL(/(?:\?|&)mode=code(?:&|$)/); await page .locator('[data-workspace-desktop-navigation]') diff --git a/cockpit/chat/footprint.spec.ts b/cockpit/chat/footprint.spec.ts index f99816ead..5fe450b21 100644 --- a/cockpit/chat/footprint.spec.ts +++ b/cockpit/chat/footprint.spec.ts @@ -53,7 +53,9 @@ describe('Chat footprint', () => { expect(graphSource).toContain('async def generate_title'); expect(graphSource).toContain('metadata={"title": title}'); - expect(graphSource).toContain('add_node("generate_title", generate_title)'); + const explicitRegistration = graphSource.includes('add_node("generate_title", generate_title)'); + const loopRegistration = /for name, node in \([^\n]*\("generate_title", generate_title\)[^\n]*\):\n _builder\.add_node\(name, node\)/.test(graphSource); + expect(explicitRegistration || loopRegistration).toBe(true); expect(graphSource).toContain('add_edge("generate_title", END)'); } }); diff --git a/cockpit/chat/generative-ui/angular/e2e/c-generative-ui.spec.ts b/cockpit/chat/generative-ui/angular/e2e/c-generative-ui.spec.ts index c84323342..f1f3988e5 100644 --- a/cockpit/chat/generative-ui/angular/e2e/c-generative-ui.spec.ts +++ b/cockpit/chat/generative-ui/angular/e2e/c-generative-ui.spec.ts @@ -1,8 +1,13 @@ import { test, expect } from '@playwright/test'; import { submitAndWaitForResponse } from '@threadplane-internal/e2e-harness'; -test('c-generative-ui: dashboard prompt renders chat-generative-ui surface', async ({ page }) => { - await submitAndWaitForResponse(page, 'Show me a dashboard of airline operations.'); +test('c-generative-ui: dashboard prompt renders chat-generative-ui surface', async ({ + page, +}) => { + await submitAndWaitForResponse( + page, + 'Show me a dashboard of airline operations.' + ); // The render_spec tool call returns a dashboard JSON spec which the // content-classifier in @threadplane/chat mounts as a tree of @@ -14,13 +19,37 @@ test('c-generative-ui: dashboard prompt renders chat-generative-ui surface', asy await expect(page.locator('chat-generative-ui')).not.toHaveCount(0); }); -test('c-generative-ui: filter prompt produces assistant turn', async ({ page }) => { - await submitAndWaitForResponse(page, 'Filter to only the cancelled flights.'); +test('c-generative-ui: filter prompt produces assistant turn', async ({ + page, +}) => { + const runThreads: string[] = []; + page.on('request', (request) => { + const run = /\/threads\/([^/]+)\/runs\/stream/.exec(request.url()); + if (request.method() === 'POST' && run) runThreads.push(run[1]); + }); + await submitAndWaitForResponse( + page, + 'Show me a dashboard of airline operations.' + ); + await expect(page.locator('chat-generative-ui').first()).toBeVisible(); + // Keep the accepted dashboard's conversation; the one-turn helper navigates + // on every call and would make this a rejected first-turn filter. + const send = page.getByRole('button', { name: /send message/i }); + await page + .getByRole('textbox', { name: /message|prompt/i }) + .fill('Filter to only the cancelled flights.'); + await expect(send).toBeEnabled(); + await send.click(); // The query_recent_disruptions tool returns data the assistant uses to // narrow the dashboard. Distinctive surface here is just that the // assistant turn finalized — the dashboard view update is internal state. await expect( - page.locator('chat-message[data-role="assistant"]').last(), + page.locator('chat-message[data-role="assistant"]').last() ).toBeVisible(); + await expect( + page.locator('chat-message[data-role="assistant"]').last() + ).toContainText('Stored 3 cancelled disruption rows (limit 5).'); + expect(runThreads).toHaveLength(2); + expect(new Set(runThreads).size).toBe(1); }); diff --git a/cockpit/chat/generative-ui/angular/e2e/fixtures/c-generative-ui.json b/cockpit/chat/generative-ui/angular/e2e/fixtures/c-generative-ui.json index bccea6269..aa1de776b 100644 --- a/cockpit/chat/generative-ui/angular/e2e/fixtures/c-generative-ui.json +++ b/cockpit/chat/generative-ui/angular/e2e/fixtures/c-generative-ui.json @@ -7,8 +7,8 @@ "response": { "toolCalls": [ { - "name": "render_spec", - "arguments": "{\"elements\":{\"root\":{\"type\":\"dashboard_grid\",\"children\":[\"stats_row\",\"charts_row\",\"table_section\"]},\"stats_row\":{\"type\":\"container\",\"props\":{\"direction\":\"row\"},\"children\":[\"on_time_card\",\"flights_card\",\"delay_card\",\"load_card\"]},\"on_time_card\":{\"type\":\"stat_card\",\"props\":{\"label\":\"On-time %\",\"value\":{\"$state\":\"/on_time/value\"},\"delta\":{\"$state\":\"/on_time/delta\"}}},\"flights_card\":{\"type\":\"stat_card\",\"props\":{\"label\":\"Flights Today\",\"value\":{\"$state\":\"/flights_today/value\"},\"delta\":{\"$state\":\"/flights_today/delta\"}}},\"delay_card\":{\"type\":\"stat_card\",\"props\":{\"label\":\"Avg Delay (min)\",\"value\":{\"$state\":\"/avg_delay/value\"},\"delta\":{\"$state\":\"/avg_delay/delta\"}}},\"load_card\":{\"type\":\"stat_card\",\"props\":{\"label\":\"Load Factor\",\"value\":{\"$state\":\"/load_factor/value\"},\"delta\":{\"$state\":\"/load_factor/delta\"}}},\"charts_row\":{\"type\":\"container\",\"props\":{\"direction\":\"row\"},\"children\":[\"trend_chart\",\"airline_chart\"]},\"trend_chart\":{\"type\":\"line_chart\",\"props\":{\"title\":\"On-time % Trend\",\"data\":{\"$state\":\"/on_time_trend\"},\"xKey\":\"month\",\"yKey\":\"on_time_pct\"}},\"airline_chart\":{\"type\":\"bar_chart\",\"props\":{\"title\":\"Flights by Airline (Daily)\",\"data\":{\"$state\":\"/flights_by_airline\"},\"labelKey\":\"airline\",\"valueKey\":\"count\"}},\"table_section\":{\"type\":\"data_grid\",\"props\":{\"title\":\"Recent Disruptions\",\"rows\":{\"$state\":\"/recent_disruptions\"},\"columns\":[\"flight_number\",\"type\",\"minutes\",\"route\",\"date\"]}}},\"root\":\"root\"}", + "name": "plan_dashboard_operation", + "arguments": "{\"kind\":\"create\",\"calls\":[{\"name\":\"render_spec\",\"args\":{\"elements\":{\"root\":{\"type\":\"dashboard_grid\",\"children\":[\"stats_row\",\"charts_row\",\"table_section\"]},\"stats_row\":{\"type\":\"container\",\"props\":{\"direction\":\"row\"},\"children\":[\"on_time_card\",\"flights_card\",\"delay_card\",\"load_card\"]},\"on_time_card\":{\"type\":\"stat_card\",\"props\":{\"label\":\"On-time %\",\"value\":{\"$state\":\"/on_time/value\"},\"delta\":{\"$state\":\"/on_time/delta\"}}},\"flights_card\":{\"type\":\"stat_card\",\"props\":{\"label\":\"Flights Today\",\"value\":{\"$state\":\"/flights_today/value\"},\"delta\":{\"$state\":\"/flights_today/delta\"}}},\"delay_card\":{\"type\":\"stat_card\",\"props\":{\"label\":\"Avg Delay (min)\",\"value\":{\"$state\":\"/avg_delay/value\"},\"delta\":{\"$state\":\"/avg_delay/delta\"}}},\"load_card\":{\"type\":\"stat_card\",\"props\":{\"label\":\"Load Factor\",\"value\":{\"$state\":\"/load_factor/value\"},\"delta\":{\"$state\":\"/load_factor/delta\"}}},\"charts_row\":{\"type\":\"container\",\"props\":{\"direction\":\"row\"},\"children\":[\"trend_chart\",\"airline_chart\"]},\"trend_chart\":{\"type\":\"line_chart\",\"props\":{\"title\":\"On-time % Trend\",\"data\":{\"$state\":\"/on_time_trend\"},\"xKey\":\"month\",\"yKey\":\"on_time_pct\"}},\"airline_chart\":{\"type\":\"bar_chart\",\"props\":{\"title\":\"Flights by Airline (Daily)\",\"data\":{\"$state\":\"/flights_by_airline\"},\"labelKey\":\"airline\",\"valueKey\":\"count\"}},\"table_section\":{\"type\":\"data_grid\",\"props\":{\"title\":\"Recent Disruptions\",\"rows\":{\"$state\":\"/recent_disruptions\"},\"columns\":[\"flight_number\",\"type\",\"minutes\",\"route\",\"date\"]}}},\"root\":\"root\"}},{\"name\":\"query_airline_kpis\",\"args\":{}},{\"name\":\"query_on_time_trend\",\"args\":{}},{\"name\":\"query_flights_by_airline\",\"args\":{}},{\"name\":\"query_recent_disruptions\",\"args\":{}}],\"interpretation\":\"\"}", "id": "call_offZhaDPm8tbblqv1sNHarnc" } ] @@ -21,12 +21,12 @@ "response": { "toolCalls": [ { - "name": "query_recent_disruptions", - "arguments": "{\"limit\":5,\"type\":\"cancelled\"}", + "name": "plan_dashboard_operation", + "arguments": "{\"kind\":\"data_update\",\"calls\":[{\"name\":\"query_recent_disruptions\",\"args\":{\"limit\":5,\"type\":\"cancelled\"}}],\"interpretation\":\"\"}", "id": "call_ivGTFO6CX8sl4bkOprs633gH" } ] } } ] -} \ No newline at end of file +} diff --git a/cockpit/chat/generative-ui/python/prompts/generative-ui.md b/cockpit/chat/generative-ui/python/prompts/generative-ui.md index 85b0d2e6a..f6f1c7920 100644 --- a/cockpit/chat/generative-ui/python/prompts/generative-ui.md +++ b/cockpit/chat/generative-ui/python/prompts/generative-ui.md @@ -1,6 +1,10 @@ # Airline Operations Dashboard Agent -You are a dashboard agent that builds interactive airline-operations KPI dashboards. You have five tools: +You choose one operation for an interactive airline-operations dashboard. You must call `plan_dashboard_operation(kind, calls, interpretation)` exactly once. That is your only directly available tool. Do not write an action promise or a summary in your response. Your decision is independently validated before the graph executes any action. + +`kind` is exactly one of `create`, `data_update`, `restructure`, `interpret`, or `unsupported`. `calls` is a list of objects containing exactly `name` and `args` (no IDs). Use an empty string for `interpretation` on action and unsupported decisions; for `interpret`, provide the bounded question/interpretation text. Read-only kinds have an empty calls list. Decisions are bounded to five calls, with unique names and at most one render. + +The graph can execute these five actions after admission: - `render_spec(elements, root)` — Author or update the dashboard layout. `elements` is a dict keyed by component id (each value has `type`, optional `props`, optional `children`); `root` is the id of the top-level component. See the schema below. - `query_airline_kpis()` — Snapshot of operational KPIs: on-time %, flights today, avg delay, load factor. @@ -10,19 +14,20 @@ You are a dashboard agent that builds interactive airline-operations KPI dashboa ## Workflow -### When no dashboard exists yet (first turn) +### For a request to create/show a dashboard when none exists -1. Call `render_spec` ONCE with a complete dashboard layout — stat cards, charts, table — using `$state` bindings to the slots the data tools populate (see "State Path Conventions" below). -2. In the SAME turn (same tool_calls array), call EACH data tool that backs a component in your spec. Do NOT call tools whose data your spec doesn't reference. -3. After the tools return, return WITHOUT any further tool calls. A separate node will write a brief conversational summary. **Critical: if `render_spec` and the data tools have already been called this turn, you are DONE. Return with no tool_calls. Do NOT call `render_spec` again. Do NOT re-call the data tools.** +Choose `create`. Include exactly one `render_spec` call with a complete layout and exactly the queries required for its `$state` bindings. Literal or null data needs no query. The graph will execute this batch once, assess every result, and publish the entire operation atomically. You will not receive another planning turn or produce its completion summary. Interpretive and unsupported requests remain read-only regardless of whether a layout exists; do not force creation solely because the checkpoint lacks one. `data_update` and `restructure` require an accepted existing layout. ### When the dashboard exists (follow-up turn) -Categorize the user's request and act ONCE. DO NOT ask clarifying questions — pick the most reasonable interpretation and act. +The confirmed checkpoint appended to this prompt provides the accepted layout and owned data. Existing dashboard means an accepted layout, including a literal-only layout with empty data. Choose one decision without asking clarifying questions. + +- **Filter / scope** (e.g. "filter to cancelled flights only", "last 6 months", "top 3"): choose `data_update` with exactly one query and the requested arguments. Do not include a render call. A cancelled filter means `query_recent_disruptions` with `{"type":"cancelled"}`; the graph validates returned rows before applying them. +- **Structural change** (e.g. "add a card for X", "remove the table"): choose `restructure` with one modified render layout. Reuse resolved checkpoint bindings; include queries only for genuinely missing binding paths. A stored null array resolves its array pointer; a null metric does not resolve `/value` or `/delta`. +- **Interpretive question** (e.g. "why is on-time % low?"): choose `interpret`, no calls, and the question in `interpretation`. The graph produces read-only prose and an explicit no-change status. +- **Unsupported request**: choose `unsupported`, no calls. Never claim that a change occurred. -- **Filter / scope** (e.g. "filter to cancelled flights only", "last 6 months", "top 3"): call EXACTLY ONE data tool — the one that backs the affected component — with the new parameters. Do NOT call `render_spec`. -- **Structural change** (e.g. "add a card for X", "remove the table"): call `render_spec` with the modified layout, then call data tools only for the NEW components. -- **Interpretive question** that no tool could resolve (e.g. "why is on-time % low?"): respond in plain prose with no tool calls. +Query arguments use only their named fields. `months` and `limit` are integers from 1 through 100 (defaults 12 and 5); null is invalid. `airlines` may be absent/null/empty for all airlines or a list of at most 100 nonempty strings. Disruption `type` may be absent/null for all or exactly `delayed`/`cancelled`. All dashboard text is nonempty and at most 512 UTF-16 characters. Component IDs are at most 80 characters and cannot be `__proto__`, `prototype`, or `constructor`. Layouts contain 1–100 elements and form one connected tree without duplicate children, shared nodes, or cycles. Use only the six component types below and their exact documented props. Chart keys must be `month`/`on_time_pct` or `airline`/`count`; grid columns come from `flight_number`, `type`, `minutes`, `route`, `date`, without duplicates. Stat values/deltas and chart/grid data may be literal values or null; safe state bindings use only the paths below. ## JSON Render Spec Format diff --git a/cockpit/chat/generative-ui/python/src/dashboard_contract.py b/cockpit/chat/generative-ui/python/src/dashboard_contract.py new file mode 100644 index 000000000..662a8de23 --- /dev/null +++ b/cockpit/chat/generative-ui/python/src/dashboard_contract.py @@ -0,0 +1,198 @@ +"""Private mirror of the native dashboard data/spec admission contract. + +Keep aligned with react/src/dashboard-data.ts and dashboard-spec.ts. +All returned values are owned copies; invalid data raises ValueError. +""" +import json +from math import isfinite + +METRICS = ("on_time", "flights_today", "avg_delay", "load_factor") +ARRAYS = ("on_time_trend", "flights_by_airline", "recent_disruptions") +DISRUPTION_KEYS = ("flight_number", "type", "minutes", "route", "date") +SLOT_TO_TOOL = {**{key: "query_airline_kpis" for key in METRICS}, + **{key: "query_" + key for key in ARRAYS}} + + +def require(condition, reason="Invalid dashboard contract"): + if not condition: + raise ValueError(reason) + + +def utf16(value): + return len(value.encode("utf-16-le", errors="surrogatepass")) // 2 + + +def capture(value): + nodes = characters = 0 + parents = set() + + def copy(item, depth): + nonlocal nodes, characters + nodes += 1 + require(nodes <= 20000 and depth <= 32, "Data exceeds budget") + if type(item) is str: + size = utf16(item) + characters += size + require(size <= 100000 and characters <= 1000000, "Text exceeds budget") + return item + if item is None or type(item) is bool: + return item + if type(item) in (int, float): + require(number(item), "Nonfinite number") + return item + require(type(item) in (dict, list) and id(item) not in parents, "Unsupported data") + require(len(item) <= 10000, "Container exceeds budget") + parents.add(id(item)) + result = {} if type(item) is dict else [] + for key, entry in (item.items() if type(item) is dict else enumerate(item)): + key_text = key if type(item) is dict else str(key) + require(type(key_text) is str) + characters += utf16(key_text) + require(utf16(key_text) <= 100000 and characters <= 1000000, "Keys exceed budget") + owned = copy(entry, depth + 1) + if type(item) is dict: + result[key] = owned + else: + result.append(owned) + parents.remove(id(item)) + return result + + return copy(value, 0) + + +def parse_data(text): + require(type(text) is str and utf16(text) <= 100000, "Result exceeds budget") + try: + return capture(json.loads(text)) + except (TypeError, json.JSONDecodeError, RecursionError) as error: + raise ValueError("Invalid tool JSON") from error + + +def number(value): + try: + return type(value) in (int, float) and isfinite(value) + except OverflowError: + return False + + +def text(value): + return type(value) is str and 1 <= utf16(value) <= 512 + + +def safe_key(value): + return text(value) and utf16(value) <= 80 and value not in ("__proto__", "prototype", "constructor") + + +def metric(value): + return (type(value) is dict and set(value) == {"value", "delta"} + and (text(value["value"]) or number(value["value"])) + and (value["delta"] is None or text(value["delta"]))) + + +def rows(slot, value): + if type(value) is not list or len(value) > 100: + return False + rules = { + "on_time_trend": {"month": text, "on_time_pct": lambda n: number(n) and 0 <= n <= 100}, + "flights_by_airline": {"airline": text, "count": lambda n: number(n) and n >= 0}, + "recent_disruptions": {**{key: text for key in DISRUPTION_KEYS if key != "minutes"}, + "minutes": lambda n: number(n) and n >= 0}, + }[slot] + return all(type(row) is dict and set(row) == set(rules) + and all(rule(row[key]) for key, rule in rules.items()) for row in value) + + +def validate_dashboard(value): + value = capture(value) + require(type(value) is dict and set(value) <= set(METRICS + ARRAYS)) + for key, item in value.items(): + require(item is None or (metric(item) if key in METRICS else rows(key, item))) + return value + + +def data_slots(name, value, args): + value = capture(value) + if name == "query_airline_kpis": + require(type(value) is dict and set(value) == set(METRICS) + and all(metric(item) for item in value.values())) + return value + slots = {"query_" + key: key for key in ARRAYS} + require(name in slots) + slot = slots[name] + require(rows(slot, value)) + if slot == "on_time_trend": + require(len(value) <= args.get("months", 12), "Trend exceeds requested months") + elif slot == "flights_by_airline": + airlines = args.get("airlines") + require(not airlines or all(row["airline"] in airlines for row in value), "Wrong airline") + else: + require(len(value) <= args.get("limit", 5), "Disruptions exceed requested limit") + require(args.get("type") is None or all(row["type"] == args["type"] for row in value), "Wrong disruption filter") + return {slot: value} + + +def is_binding(value, paths): + return type(value) is dict and set(value) == {"$state"} and type(value["$state"]) is str and value["$state"] in paths + + +def scalar(value, suffix): + return (value is None or text(value) or (suffix == "value" and number(value)) + or is_binding(value, [f"/{key}/{suffix}" for key in METRICS])) + + +def array_prop(value, slot): + return value is None or is_binding(value, ["/" + slot]) or rows(slot, value) + + +def validate_spec(value): + value = capture(value) + require(type(value) is dict and set(value) == {"root", "elements"} and safe_key(value["root"])) + elements = value["elements"] + require(type(elements) is dict and 1 <= len(elements) <= 100 and value["root"] in elements) + catalog = { + "dashboard_grid": ([], {}, True), + "container": ([], {"direction": lambda x: x in ("row", "column")}, True), + "stat_card": (["label"], {"label": text, "value": lambda x: scalar(x, "value"), "delta": lambda x: scalar(x, "delta")}, False), + "line_chart": (["data", "xKey", "yKey"], {"title": text, "data": lambda x: array_prop(x, "on_time_trend"), "xKey": lambda x: x == "month", "yKey": lambda x: x == "on_time_pct"}, False), + "bar_chart": (["data", "labelKey", "valueKey"], {"title": text, "data": lambda x: array_prop(x, "flights_by_airline"), "labelKey": lambda x: x == "airline", "valueKey": lambda x: x == "count"}, False), + "data_grid": (["rows", "columns"], {"title": text, "rows": lambda x: array_prop(x, "recent_disruptions"), "columns": lambda x: type(x) is list and 1 <= len(x) <= 5 and all(type(k) is str and k in DISRUPTION_KEYS for k in x) and len(set(x)) == len(x)}, False), + } + for key, element in elements.items(): + require(safe_key(key) and type(element) is dict and set(element) <= {"type", "props", "children"}) + require(type(element.get("type")) is str and element["type"] in catalog) + required, rules, can_have_children = catalog[element["type"]] + props = element.get("props", {}) + require(type(props) is dict and set(required) <= set(props) and set(props) <= set(rules)) + require(all(rules[name](item) for name, item in props.items())) + children = element.get("children", []) + require(type(children) is list and len(children) <= 100 and all(safe_key(child) for child in children) + and len(set(children)) == len(children) and (can_have_children or not children)) + visited = set() + + def visit(key): + require(key in elements and key not in visited, "Layout must be one complete tree") + visited.add(key) + for child in elements[key].get("children", []): + visit(child) + + visit(value["root"]) + require(len(visited) == len(elements), "Unreachable component") + return value + + +def bindings(spec): + return [value["$state"] for element in spec["elements"].values() + for value in element.get("props", {}).values() if type(value) is dict and "$state" in value] + + +def resolve(pointer, dashboard): + current = dashboard + for key in pointer.lstrip("/").split("/"): + if type(current) is not dict or key not in current: + return False + current = current[key] + return True + + +def bindings_resolve(spec, dashboard): + return all(resolve(pointer, dashboard) for pointer in bindings(spec)) diff --git a/cockpit/chat/generative-ui/python/src/graph.py b/cockpit/chat/generative-ui/python/src/graph.py index 32a56f4af..bed86f2b9 100644 --- a/cockpit/chat/generative-ui/python/src/graph.py +++ b/cockpit/chat/generative-ui/python/src/graph.py @@ -1,390 +1,354 @@ -"""Single-node agentic graph for the airline operations KPI dashboard. - -Flow: - START → agent ↔ tools → wrap_spec_into_ai → agent (loop) → emit_state → respond → END - -`agent` is a single LLM call with all 5 tools bound (render_spec + 4 data -tools). After tools run, `wrap_spec_into_ai` post-processes — if the LLM -called render_spec, it replaces the parent AI message's content with the -spec JSON (in place via add_messages' id-match reducer) so the chat-lib's -content-classifier mounts . Then loops back to agent -until the LLM returns no tool_calls (cap _MAX_TOOL_ITERATIONS per turn). - -Mirrors the emit_generated_surface pattern from examples/chat/python/src/graph.py, -adapted for a continuation loop instead of terminal dispatch. -""" - +"""Required decision, bounded execution, atomic dashboard publication.""" import json import os from copy import deepcopy -from math import isfinite from pathlib import Path from typing import Literal, NotRequired - +from uuid import uuid4 from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, ToolMessage from langchain_core.tools import tool from langchain_openai import ChatOpenAI -from langgraph.graph import StateGraph, MessagesState, END -from langgraph.prebuilt import ToolNode from langgraph.constants import TAG_NOSTREAM +from langgraph.graph import END, MessagesState, StateGraph +from langgraph.prebuilt import ToolNode from langgraph_sdk import get_client - +from src.dashboard_contract import bindings_resolve, capture, data_slots, parse_data, require, validate_dashboard, validate_spec from src.dashboard_tools import ALL_TOOLS as _DATA_TOOLS +from src.operations import admit _PROMPT = (Path(__file__).parent.parent / "prompts" / "generative-ui.md").read_text() -_MAX_TOOL_ITERATIONS = 6 - - class DashboardState(MessagesState): - """Layouts live in messages; owned data and completion survive checkpoints.""" dashboard: NotRequired[dict] completed_turn_id: NotRequired[str | None] completed_answer_id: NotRequired[str | None] completed_message_ids: NotRequired[list[str] | None] + operation_receipt: NotRequired[dict | None] _submitted_turn_id: NotRequired[str | None] + _submitted_human: NotRequired[dict | None] + _decision: NotRequired[dict | None] + _operation: NotRequired[dict | None] + _reason: NotRequired[str | None] + _parent_id: NotRequired[str | None] + _result_ids: NotRequired[list[str]] + _assessment: NotRequired[dict | None] + _owned_layout: NotRequired[dict | None] + _render_owner: NotRequired[dict | None] + _prior_dashboard: NotRequired[dict] + _prior_layout: NotRequired[dict | None] + _prior_owner: NotRequired[dict | None] + +def fresh(prefix): + return f"{prefix}-{uuid4()}" - -def prepare_turn(state: DashboardState) -> dict: - """Capture the submitted human before tools append intermediate messages.""" - last = state["messages"][-1] if state["messages"] else None - return { - "_submitted_turn_id": last.id if isinstance(last, HumanMessage) else None, - "completed_turn_id": None, - "completed_answer_id": None, - "completed_message_ids": None, - } - +@tool +def plan_dashboard_operation(kind: Literal["create", "data_update", "restructure", "interpret", "unsupported"], calls: list[dict], interpretation: str) -> dict: + """Choose one bounded operation. Calls contain only name and args. No action executes until independent admission.""" + return {"kind": kind, "calls": calls, "interpretation": interpretation} # region render-spec-tool @tool async def render_spec(elements: dict, root: str) -> str: - """Render an interactive dashboard layout. - - Use this tool to author or update the dashboard layout. See the system - prompt for the full component catalog and state binding conventions. - - Call this tool AT MOST ONCE per turn — only when the layout needs to - be created (first turn) or restructured (follow-up structural change). - Do NOT call it again to refresh data; the data tools handle that. - - Args: - elements: Dict keyed by component id. Each value has `type`, optional - `props`, and optional `children` (list of component ids). - root: The id of the top-level component (must be a key in `elements`). - - Returns: - The spec serialized as JSON. A post-process node (wrap_spec_into_ai) - wraps this payload into the AI message content where the - chat-lib's content-classifier picks it up. - """ + """Render a validated native dashboard layout. The graph owns publication.""" return json.dumps({"elements": elements, "root": root}) # endregion - _ALL_TOOLS = [render_spec, *_DATA_TOOLS] - -_llm_with_tools = ChatOpenAI( - model="gpt-5", - temperature=0, - streaming=True, - reasoning_effort="minimal", -).bind_tools(_ALL_TOOLS) - +_llm_with_tools = ChatOpenAI(model="gpt-5", temperature=0, streaming=True, reasoning_effort="minimal", tags=[TAG_NOSTREAM]).bind_tools( + [plan_dashboard_operation], tool_choice=plan_dashboard_operation.name, parallel_tool_calls=False, strict=False) _respond_llm = ChatOpenAI(model="gpt-5-mini", temperature=0, streaming=True) - -# ── generate_title node (inline; matches Pattern D from spec -# 2026-05-19-llm-generated-labels-design.md) ────────────────────────────── - -_TITLE_PROMPT = ( - "In 3-5 words, summarize what the user is asking about. " - "Output ONLY the title — no quotes, no period, no prefix." -) +_TITLE_PROMPT = "In 3-5 words, summarize what the user is asking about. Output ONLY the title — no quotes, no period, no prefix." _TITLE_MODEL = "gpt-5-mini" - async def generate_title(state: DashboardState, config) -> dict: - """Background title generation: on the first turn, summarize the user's - intent into 3-5 words and persist to LangGraph thread metadata. - - Idempotent — skips when metadata.title already exists. Errors are - swallowed because the title is a UX nicety, never a blocker. - """ + """Best-effort metadata title; never a blocking or streamed action.""" thread_id = (config.get("configurable") or {}).get("thread_id") if not thread_id: return {} - sdk_url = os.environ.get("LANGGRAPH_API_URL") try: - client = get_client(url=sdk_url) + client = get_client(url=os.environ.get("LANGGRAPH_API_URL")) thread = await client.threads.get(thread_id) if (thread.get("metadata") or {}).get("title"): return {} - first_user = next( - (m for m in state["messages"] if getattr(m, "type", None) == "human"), - None, - ) - if not first_user or not isinstance(first_user.content, str): - return {} - if first_user.content.lstrip().startswith("{"): + first_user = next((m for m in state["messages"] if isinstance(m, HumanMessage)), None) + if not first_user or not isinstance(first_user.content, str) or first_user.content.lstrip().startswith("{"): return {} llm = ChatOpenAI(model=_TITLE_MODEL, temperature=0, tags=[TAG_NOSTREAM]) - response = await llm.ainvoke([ - SystemMessage(content=_TITLE_PROMPT), - HumanMessage(content=first_user.content), - ]) + response = await llm.ainvoke([SystemMessage(content=_TITLE_PROMPT), HumanMessage(content=first_user.content)]) title = (response.content or "").strip().strip('"').strip("'")[:80] if title: await client.threads.update(thread_id, metadata={"title": title}) - except Exception as e: # noqa: BLE001 — title is a UX nicety; never block - print( - f"[generate_title] failed for thread {thread_id}: " - f"{type(e).__name__}: {e}", - flush=True, - ) + except Exception as error: + print(f"[generate_title] failed for thread {thread_id}: {type(error).__name__}: {error}", flush=True) return {} +def owned_layout(state): + """Accept only a checkpoint layout with its exact graph-owned wrapped pair.""" + layout, owner = state.get("_owned_layout"), state.get("_render_owner") + if layout is None or type(owner) is not dict: + return None, None + try: + spec = validate_spec(layout) + parent = next(m for m in state["messages"] if m.id == owner["parent_id"]) + result = next(m for m in state["messages"] if m.id == owner["result_id"]) + require(isinstance(parent, AIMessage) and isinstance(result, ToolMessage)) + require(result.name == "render_spec" and result.tool_call_id == owner["call_id"] and result.status == "success" and result.content == "rendered") + require(parse_data(parent.content) == spec) + require(any(c["id"] == owner["call_id"] and c["name"] == "render_spec" and c["args"] == spec for c in parent.tool_calls)) + return spec, deepcopy(owner) + except (ValueError, KeyError, StopIteration): + return None, None -async def agent(state: DashboardState) -> dict: - """Single agentic node: LLM bound with all 5 tools.""" - messages = [SystemMessage(content=_PROMPT)] + state["messages"] - response = await _llm_with_tools.ainvoke(messages) - return {"messages": [response]} - - -def should_continue(state: DashboardState) -> Literal["tools", "finalize", "emit_state"]: - """Loop while the agent emits tool_calls, up to _MAX_TOOL_ITERATIONS - this turn. After the cap, route through `finalize` (strips orphan - tool_calls from the last AI message) before emit_state — otherwise - `respond`'s LLM call rejects the message history because the AI - message had tool_calls without matching ToolMessages.""" - last = state["messages"][-1] - if not (hasattr(last, "tool_calls") and last.tool_calls): - return "emit_state" - - iter_count = 0 - for msg in reversed(state["messages"]): - if msg.type == "human": - break - if msg.type == "ai" and getattr(msg, "tool_calls", None): - iter_count += 1 - if iter_count >= _MAX_TOOL_ITERATIONS: - return "finalize" - return "tools" - - -async def finalize(state: DashboardState) -> dict: - """Strip tool_calls from the last AI message when the iteration cap - is hit, so respond's LLM call doesn't fail with 'tool_calls without - matching ToolMessage' errors. Replaces in place via add_messages - id-match reducer.""" - last = state["messages"][-1] - if not (isinstance(last, AIMessage) and last.tool_calls): +def prepare_turn(state: DashboardState) -> dict: + last = state["messages"][-1] if state["messages"] else None + layout, owner = owned_layout(state) + try: + dashboard = validate_dashboard(state.get("dashboard") or {}) + except ValueError: + dashboard = {} + return {"_submitted_turn_id": last.id if isinstance(last, HumanMessage) else None, + "_submitted_human": deepcopy(last.model_dump()) if isinstance(last, HumanMessage) else None, + "_decision": None, "_operation": None, "_reason": None, "_parent_id": None, + "_result_ids": [], "_assessment": None, "operation_receipt": None, + "completed_turn_id": None, "completed_answer_id": None, "completed_message_ids": None, + "_prior_dashboard": deepcopy(dashboard), "_prior_layout": deepcopy(layout), + "_prior_owner": deepcopy(owner), "dashboard": dashboard, "_owned_layout": layout, "_render_owner": owner} + + +def submitted_human_matches(state): + human = next((m for m in reversed(state["messages"]) if isinstance(m, HumanMessage)), None) + return (human is not None and human.id == state.get("_submitted_turn_id") + and human.model_dump() == state.get("_submitted_human")) + +async def decide(state: DashboardState) -> dict: + if not state.get("_submitted_turn_id") or not submitted_human_matches(state): + return {"_reason": "Missing submitted human identity"} + context = json.dumps({"accepted_layout": state.get("_prior_layout"), "dashboard": state["_prior_dashboard"]}) + response = await _llm_with_tools.ainvoke([SystemMessage(content=_PROMPT + "\nConfirmed checkpoint: " + context), *state["messages"]], config={"tags": [TAG_NOSTREAM]}) + try: + require(isinstance(response, AIMessage) and not response.invalid_tool_calls, "Invalid decision tool calls") + require(not response.additional_kwargs.get("refusal"), "Decision refused") + require(len(response.tool_calls) == 1 and response.tool_calls[0]["name"] == plan_dashboard_operation.name, "Exactly one required decision was not returned") + return {"_decision": capture(response.tool_calls[0]["args"])} + except ValueError as error: + return {"_reason": str(error)} + +def admit_operation(state: DashboardState) -> dict: + if state.get("_reason"): return {} - replacement_kwargs: dict = { - "content": last.content or "(iteration cap reached)", - "additional_kwargs": last.additional_kwargs or {}, - "response_metadata": last.response_metadata or {}, - } - if getattr(last, "id", None): - replacement_kwargs["id"] = last.id - return {"messages": [AIMessage(**replacement_kwargs)]} - + try: + human = state["messages"][-1] if state["messages"] else None + require(isinstance(human, HumanMessage) and human.model_dump() == state.get("_submitted_human"), "Submitted human changed before admission") + operation = admit(state.get("_decision"), state["_prior_dashboard"], state.get("_prior_layout")) + operation["calls"] = [{**call, "id": fresh("call"), "type": "tool_call"} for call in operation["calls"]] + parent_id = fresh("action") if operation["calls"] else None + update = {"_operation": deepcopy(operation), "_parent_id": parent_id} + if parent_id: + update["messages"] = [AIMessage(id=parent_id, content="", tool_calls=deepcopy(operation["calls"]))] + return update + except ValueError as error: + return {"_reason": str(error)} + +def route_operation(state): + return "tools" if (state.get("_operation") or {}).get("calls") else "assess_results" + +_tool_node = ToolNode(_ALL_TOOLS, handle_tool_errors="The dashboard tool failed.") + +async def execute_tools(state: DashboardState, config) -> dict: + """Real ToolNode execution with owned result IDs before transcript publication.""" + output = await _tool_node.ainvoke(state, config) + # LangChain treats [] as an empty multimodal-content list, rather than JSON. + # Preserve the genuine empty data result in the native text-result format. + results = [m.model_copy(deep=True, update={"id": fresh("result"), + "content": "[]" if m.content == [] else deepcopy(m.content)}) for m in output["messages"]] + return {"messages": results, "_result_ids": [m.id for m in results]} + +def current_results(state): + parent_index = next((i for i, m in enumerate(state["messages"]) if m.id == state.get("_parent_id")), None) + return [] if parent_index is None else [m for m in state["messages"][parent_index + 1:] if isinstance(m, ToolMessage)] + + +def result_projection(results): + return [{"id": m.id, "tool_call_id": m.tool_call_id, "name": m.name, + "status": m.status, "content": deepcopy(m.content)} for m in results] + + +def latest_data_slots(results, calls): + """Merge validated data in transcript order, owning every selected slot. + + Admission forbids repeated tool names in executable operations. This helper + still defines chronological replacement for historical/direct observations. + """ + slots = {} + by_id = {call["id"]: call for call in calls} + for result in results: + if result.name == "render_spec": + continue + call = by_id.get(result.tool_call_id) + require(call is not None and result.name == call["name"] and result.status == "success", "Uncorrelated data result") + slots.update(data_slots(result.name, parse_data(result.content), call["args"])) + return deepcopy(slots) + +def assess_results(state: DashboardState) -> dict: + operation = state.get("_operation") + if operation is None: + return {"_assessment": {"outcome": "rejected", "slots": {}, "layout": None, "reason": state.get("_reason") or "Invalid decision"}} + if not operation["calls"]: + return {"_assessment": {"outcome": operation["kind"], "slots": {}, "layout": None, "reason": None}} + try: + require(submitted_human_matches(state), "Submitted human changed before assessment") + calls, results = operation["calls"], current_results(state) + require(len(results) == len(calls) and len(state["_result_ids"]) == len(calls), "Missing or extra results") + require([m.id for m in results] == state["_result_ids"] and len(set(state["_result_ids"])) == len(results), "Unowned results") + slots, layout, render_result, render_text = {}, None, None, None + seen = set() + for result in results: + matching = [call for call in calls if call["id"] == result.tool_call_id] + require(len(matching) == 1 and result.tool_call_id not in seen, "Uncorrelated results") + call = matching[0] + seen.add(result.tool_call_id) + require(result.name == call["name"] and result.status == "success", "Failed or misnamed tool result") + payload = result.content + if call["name"] == "render_spec" and type(payload) is str and payload.strip().startswith("```"): + payload = "\n".join(line for line in payload.strip().split("\n") if not line.startswith("```")).strip() + parsed = parse_data(payload) + if call["name"] == "render_spec": + layout = validate_spec(parsed) + require(layout == call["args"], "Render differs from admitted arguments") + render_result = result.id + render_text = payload.strip() + slots = latest_data_slots(results, calls) + candidate = deepcopy(state["_prior_dashboard"]) + candidate.update(deepcopy(slots)) + if layout is not None: + require(bindings_resolve(layout, candidate), "Unresolved layout bindings") + parent = next(m for m in state["messages"] if m.id == state["_parent_id"]) + require(isinstance(parent, AIMessage) and parent.tool_calls == calls, "Unowned action parent") + require(parent.content == "", "Render/action parent already contains prose") + observed = result_projection(results) + return {"_assessment": {"outcome": "applied", "slots": deepcopy(slots), "layout": deepcopy(layout), "render_result_id": render_result, "render_text": render_text, "observed_results": observed, "reason": None}} + except (ValueError, KeyError, StopIteration) as error: + return {"_assessment": {"outcome": "unfulfilled", "slots": {}, "layout": None, "reason": str(error)}} # region wrap-spec-into-ai async def wrap_spec_into_ai(state: DashboardState) -> dict: - """Post-process that wraps the most recent render_spec ToolMessage - payload into the parent AI tool-call message's content (in place via - LangGraph's add_messages reducer matching by id). The chat-lib's - content-classifier then sees content starting with `{` and mounts - . - - Idempotent: if the parent AI message already has non-empty content - (already wrapped on a prior iteration), no-op. Also no-op if there - is no render_spec ToolMessage to process. - - Mirrors emit_generated_surface from examples/chat/python/src/graph.py, - adapted to loop back to `agent` instead of going to END. - """ - msgs = state["messages"] - - render_tool_msg: ToolMessage | None = None - parent_ai: AIMessage | None = None - for m in reversed(msgs): - if isinstance(m, ToolMessage) and m.name == "render_spec": - render_tool_msg = m - for prior in reversed(msgs): - if isinstance(prior, AIMessage) and prior.tool_calls: - if any(tc.get("id") == render_tool_msg.tool_call_id for tc in prior.tool_calls): - parent_ai = prior - break - break - - if render_tool_msg is None or parent_ai is None: + assessment = state.get("_assessment") or {} + if assessment.get("outcome") != "applied" or assessment.get("layout") is None: return {} + parent = next((m for m in state["messages"] if m.id == state["_parent_id"]), None) + result = next((m for m in current_results(state) if m.id == assessment["render_result_id"]), None) + if (not submitted_human_matches(state) + or not isinstance(parent, AIMessage) or parent.content != "" or result is None + or parent.tool_calls != state["_operation"]["calls"] + or result_projection(current_results(state)) != assessment["observed_results"] + or [m.id for m in current_results(state)] != state["_result_ids"]): + return {"_assessment": {"outcome": "unfulfilled", "slots": {}, "layout": None, "reason": "Render parent could not be applied"}} + return {"messages": [result.model_copy(deep=True, update={"content": "rendered"}), parent.model_copy(deep=True, update={"content": assessment["render_text"]})]} # endregion -# region rewrite-message-content - existing = parent_ai.content - if isinstance(existing, str) and existing.strip(): - return {} - - payload = render_tool_msg.content if isinstance(render_tool_msg.content, str) else "" - if not payload: - return {} - - stripped = payload.strip() - if stripped.startswith("```"): - lines = stripped.split("\n") - stripped = "\n".join(line for line in lines if not line.startswith("```")).strip() - - out: list = [] - - placeholder_kwargs: dict = { - "content": "rendered", - "tool_call_id": render_tool_msg.tool_call_id, - "name": "render_spec", - } - if getattr(render_tool_msg, "id", None): - placeholder_kwargs["id"] = render_tool_msg.id - out.append(ToolMessage(**placeholder_kwargs)) - - replacement_kwargs: dict = { - "content": stripped, - "tool_calls": parent_ai.tool_calls, - "additional_kwargs": parent_ai.additional_kwargs or {}, - "response_metadata": parent_ai.response_metadata or {}, - } - if getattr(parent_ai, "id", None): - replacement_kwargs["id"] = parent_ai.id - out.append(AIMessage(**replacement_kwargs)) - - return {"messages": out} -# endregion - +def make_receipt(state, assessment, owner): + operation = state.get("_operation") + results = current_results(state) if operation and operation["calls"] else [] + return {"submitted_human_id": state.get("_submitted_turn_id"), "submitted_human": deepcopy(state.get("_submitted_human")), "kind": operation["kind"] if operation else None, + "admitted_calls": deepcopy(operation["calls"]) if operation else [], + "results": result_projection(results), + "validated_slots": deepcopy(assessment.get("slots") or {}), "render_owner": deepcopy(owner), + "outcome": assessment["outcome"], "reason": assessment.get("reason")} # region emit-state -def _data_slots(name: str, data) -> dict: - """Accept only the four data tools' successful fixture-shaped payloads.""" - def number(value): - try: - return type(value) in (int, float) and isfinite(value) - except OverflowError: - return False - - if name == "query_airline_kpis": - fields = {"on_time", "flights_today", "avg_delay", "load_factor"} - if not isinstance(data, dict) or not data or not set(data) <= fields: - return {} - for value in data.values(): - if (not isinstance(value, dict) or set(value) != {"value", "delta"} - or not isinstance(value["delta"], str) - or not (isinstance(value["value"], str) or number(value["value"]))): - return {} - return deepcopy(data) - - contracts = { - "query_on_time_trend": ("on_time_trend", {"month": str, "on_time_pct": number}), - "query_flights_by_airline": ("flights_by_airline", {"airline": str, "count": number}), - "query_recent_disruptions": ("recent_disruptions", { - "flight_number": str, "type": str, "minutes": number, "route": str, "date": str}), - } - if name not in contracts or not isinstance(data, list): - return {} - slot, fields = contracts[name] - for row in data: - if not isinstance(row, dict) or set(row) != set(fields): - return {} - if any(not (isinstance(row[key], str) if check is str else check(row[key])) - for key, check in fields.items()): - return {} - return {slot: deepcopy(data)} - - async def emit_state(state: DashboardState) -> dict: - """Persist current-turn data and emit equivalent Angular state patches. - - Traverse in transcript order so the newest successful result for a slot - wins. Prior-turn results never replay; untouched prior slots are retained. - """ from langgraph.config import get_stream_writer - - messages = state["messages"] - start = next((i + 1 for i in range(len(messages) - 1, -1, -1) - if isinstance(messages[i], HumanMessage)), len(messages)) - slots: dict = {} - for msg in messages[start:]: - if not isinstance(msg, ToolMessage) or msg.status != "success": - continue + assessment = deepcopy(state["_assessment"]) + layout, owner = deepcopy(state["_prior_layout"]), deepcopy(state["_prior_owner"]) + slots = {} + if assessment["outcome"] == "applied": try: - data = json.loads(msg.content) if isinstance(msg.content, str) else msg.content - except (json.JSONDecodeError, TypeError): - continue - slots.update(_data_slots(msg.name, data)) - - dashboard = deepcopy(state.get("dashboard") or {}) - dashboard.update(slots) + require(submitted_human_matches(state), "Submitted human changed before publication") + operation = state["_operation"] + parent = next(m for m in state["messages"] if m.id == state["_parent_id"]) + require(isinstance(parent, AIMessage) and parent.tool_calls == operation["calls"], "Changed action ownership") + observed = deepcopy(assessment["observed_results"]) + if assessment["layout"] is not None: + next(item for item in observed if item["id"] == assessment["render_result_id"])["content"] = "rendered" + current = result_projection(current_results(state)) + require(current == observed and [item["id"] for item in current] == state["_result_ids"], "Results changed after assessment") + if assessment["layout"] is not None: + result = next(m for m in current_results(state) if m.id == assessment["render_result_id"]) + render = next(c for c in operation["calls"] if c["name"] == "render_spec") + require(result.content == "rendered" and result.status == "success" and result.name == "render_spec" and result.tool_call_id == render["id"] and parent.content == assessment["render_text"] and parse_data(parent.content) == assessment["layout"], "Render wrapper was not applied") + layout = deepcopy(assessment["layout"]) + owner = {"parent_id": parent.id, "call_id": render["id"], "result_id": result.id} + else: + require(parent.content == "", "Unexpected data action content") + slots = deepcopy(assessment["slots"]) + except (ValueError, StopIteration, KeyError) as error: + assessment = {"outcome": "unfulfilled", "slots": {}, "layout": None, "reason": str(error)} + layout, owner = deepcopy(state["_prior_layout"]), deepcopy(state["_prior_owner"]) + dashboard = deepcopy(state["_prior_dashboard"]) + dashboard.update(deepcopy(slots)) + receipt = make_receipt(state, assessment, owner if assessment["outcome"] == "applied" and assessment.get("layout") is not None else None) if slots: patches = {} for key, value in slots.items(): - if isinstance(value, dict): + if type(value) is dict: patches.update({f"/{key}/{field}": deepcopy(item) for field, item in value.items()}) else: patches[f"/{key}"] = deepcopy(value) - writer = get_stream_writer() - writer({"name": "state_update", "data": patches}) - - return {"dashboard": dashboard} + get_stream_writer()({"name": "state_update", "data": deepcopy(patches)}) + return {"dashboard": dashboard, "_owned_layout": layout, "_render_owner": owner, "operation_receipt": deepcopy(receipt), "_assessment": assessment} # endregion +def action_answer(receipt): + if receipt["outcome"] == "applied": + facts = [] + for call in receipt["admitted_calls"]: + name, args = call["name"], call["args"] + if name == "render_spec": + facts.append("Dashboard layout applied.") + elif name == "query_recent_disruptions": + facts.append(f"Stored {len(receipt['validated_slots']['recent_disruptions'])} {args.get('type') or 'all'} disruption rows (limit {args.get('limit', 5)}).") + elif name == "query_on_time_trend": + facts.append(f"Stored {len(receipt['validated_slots']['on_time_trend'])} months of on-time data (requested {args.get('months', 12)}).") + elif name == "query_flights_by_airline": + selection = ", ".join(args.get("airlines") or []) or "all airlines" + facts.append(f"Stored {len(receipt['validated_slots']['flights_by_airline'])} airline rows for {selection}.") + else: + facts.append("Operational KPI data applied.") + return " ".join(facts) + if receipt["outcome"] == "unsupported": + return "This request is unsupported. No dashboard changes were made." + if receipt["outcome"] == "unfulfilled": + return "The dashboard operation could not be applied. No dashboard changes were made." + return "No valid dashboard operation was selected. No dashboard changes were made." async def respond(state: DashboardState) -> dict: - """Generate a brief conversational summary of what just happened on - this turn. ALWAYS runs (no early-exit).""" - messages = [ - SystemMessage(content=( - "Provide a brief (1-2 sentence) conversational summary of what " - "you just did this turn. If you generated a dashboard, say so. " - "If you filtered data, say what you filtered. " - "Do NOT output JSON. Do NOT ask follow-up questions." - )) - ] + state["messages"] - response = await _respond_llm.ainvoke(messages) - canonical = [*state["messages"], response] - ids = [getattr(message, "id", None) for message in canonical] - human = next((m for m in reversed(state["messages"]) if isinstance(m, HumanMessage)), None) + receipt = state["operation_receipt"] + if receipt["outcome"] == "interpret": + result = await _respond_llm.ainvoke([SystemMessage(content="Answer the user's interpretive question using confirmed data. This is read-only: do not claim actions or changes. Do not output tools or JSON."), *state["messages"], SystemMessage(content="Confirmed dashboard: " + json.dumps(state["dashboard"]))], config={"tags": [TAG_NOSTREAM]}) + prose = result.content if isinstance(result.content, str) else "" + content = prose + "\n\nNo dashboard changes were made." + else: + content = action_answer(receipt) + response = AIMessage(id=fresh("answer"), content=content) + ids = [m.id for m in [*state["messages"], response]] completion = {} - if (human is not None and human.id == state.get("_submitted_turn_id") - and isinstance(response, AIMessage) - and all(isinstance(identity, str) and identity for identity in ids) - and len(ids) == len(set(ids))): - completion = { - "completed_turn_id": human.id, - "completed_answer_id": response.id, - "completed_message_ids": ids, - } + if submitted_human_matches(state) and all(type(identity) is str and identity for identity in ids) and len(ids) == len(set(ids)): + completion = {"completed_turn_id": state["_submitted_turn_id"], "completed_answer_id": response.id, "completed_message_ids": ids} return {"messages": [response], **completion} - # region graph-wiring -_builder = StateGraph(DashboardState) -_builder.add_node("prepare_turn", prepare_turn) -_builder.add_node("agent", agent) -_builder.add_node("tools", ToolNode(_ALL_TOOLS)) -_builder.add_node("wrap_spec_into_ai", wrap_spec_into_ai) -_builder.add_node("finalize", finalize) -_builder.add_node("emit_state", emit_state) -_builder.add_node("respond", respond) -_builder.add_node("generate_title", generate_title) - +_builder = StateGraph(DashboardState, input_schema=MessagesState) +for name, node in (("prepare_turn", prepare_turn), ("decide", decide), ("admit", admit_operation), ("tools", execute_tools), ("assess_results", assess_results), ("wrap_spec_into_ai", wrap_spec_into_ai), ("emit_state", emit_state), ("respond", respond), ("generate_title", generate_title)): + _builder.add_node(name, node) _builder.set_entry_point("prepare_turn") -_builder.add_edge("prepare_turn", "agent") -_builder.add_conditional_edges("agent", should_continue) -_builder.add_edge("tools", "wrap_spec_into_ai") -_builder.add_edge("wrap_spec_into_ai", "agent") -_builder.add_edge("finalize", "emit_state") +_builder.add_edge("prepare_turn", "decide") +_builder.add_edge("decide", "admit") +_builder.add_conditional_edges("admit", route_operation) +_builder.add_edge("tools", "assess_results") +_builder.add_edge("assess_results", "wrap_spec_into_ai") +_builder.add_edge("wrap_spec_into_ai", "emit_state") _builder.add_edge("emit_state", "respond") _builder.add_edge("respond", "generate_title") _builder.add_edge("generate_title", END) - graph = _builder.compile() # endregion diff --git a/cockpit/chat/generative-ui/python/src/index.ts b/cockpit/chat/generative-ui/python/src/index.ts index 0adb2d408..8a0228f2d 100644 --- a/cockpit/chat/generative-ui/python/src/index.ts +++ b/cockpit/chat/generative-ui/python/src/index.ts @@ -34,6 +34,9 @@ export const chatGenerativeUiPythonModule: CockpitCapabilityModule = { ], backendAssetPaths: [ 'cockpit/chat/generative-ui/python/src/graph.py', + 'cockpit/chat/generative-ui/python/src/dashboard_tools.py', + 'cockpit/chat/generative-ui/python/src/operations.py', + 'cockpit/chat/generative-ui/python/src/dashboard_contract.py', ], runtimeUrl: 'chat/generative-ui', devPort: 4508, diff --git a/cockpit/chat/generative-ui/python/src/operations.py b/cockpit/chat/generative-ui/python/src/operations.py new file mode 100644 index 000000000..b57c3d7d3 --- /dev/null +++ b/cockpit/chat/generative-ui/python/src/operations.py @@ -0,0 +1,53 @@ +"""Private, independently validated bounded operation admission.""" +from src.dashboard_contract import (SLOT_TO_TOOL, bindings, capture, require, + resolve, text, validate_dashboard, validate_spec) + +KINDS = ("create", "data_update", "restructure", "interpret", "unsupported") +QUERY_KEYS = {"query_airline_kpis": (), "query_on_time_trend": ("months",), + "query_flights_by_airline": ("airlines",), "query_recent_disruptions": ("limit", "type")} + + +def arguments(name, args): + args = capture(args) + require(type(args) is dict) + if name == "render_spec": + return validate_spec(args) + require(name in QUERY_KEYS and set(args) <= set(QUERY_KEYS[name]), "Unknown tool or arguments") + for key in ("months", "limit"): + if key in args: + require(type(args[key]) is int and 1 <= args[key] <= 100, "Invalid query bound") + if "airlines" in args and args["airlines"] is not None: + require(type(args["airlines"]) is list and len(args["airlines"]) <= 100 and all(text(item) for item in args["airlines"]), "Invalid airlines") + if "type" in args: + require(args["type"] is None or args["type"] in ("delayed", "cancelled"), "Invalid disruption type") + return args + + +def admit(decision, dashboard, layout): + decision = capture(decision) + require(type(decision) is dict and set(decision) == {"kind", "calls", "interpretation"}, "Invalid decision") + kind = decision["kind"] + require(type(kind) is str and kind in KINDS) + require(type(decision["interpretation"]) is str and (decision["interpretation"] == "" or text(decision["interpretation"]))) + require(kind != "interpret" or text(decision["interpretation"]), "Interpretation requires a question") + calls = decision["calls"] + require(type(calls) is list and len(calls) <= 5, "Operation exceeds batch bound") + owned = [] + names = set() + for call in calls: + require(type(call) is dict and set(call) == {"name", "args"} and type(call["name"]) is str) + require(call["name"] not in names, "Duplicate operation tool") + names.add(call["name"]) + owned.append({"name": call["name"], "args": arguments(call["name"], call["args"])}) + if kind in ("interpret", "unsupported"): + require(not owned, "Read-only decision cannot execute") + elif kind == "data_update": + require(layout is not None and len(owned) == 1 and "render_spec" not in names, "Data update requires existing layout and one query") + else: + require((layout is None) == (kind == "create"), "Category does not match existing layout") + require("render_spec" in names, "Layout operation requires one render") + spec = next(call["args"] for call in owned if call["name"] == "render_spec") + prior = validate_dashboard(dashboard) if kind == "restructure" else {} + necessary = {SLOT_TO_TOOL[pointer.split("/")[1]] for pointer in bindings(spec) if not resolve(pointer, prior)} + require(names - {"render_spec"} == necessary, "Only required backing queries may execute") + return {"kind": kind, "calls": owned, "interpretation": decision["interpretation"]} diff --git a/cockpit/chat/generative-ui/python/tests/test_operations.py b/cockpit/chat/generative-ui/python/tests/test_operations.py new file mode 100644 index 000000000..fafe1637b --- /dev/null +++ b/cockpit/chat/generative-ui/python/tests/test_operations.py @@ -0,0 +1,160 @@ +"""Offline admission and native-contract boundaries (no semantic routing claims).""" +import unittest +from copy import deepcopy + +from src.dashboard_tools import query_recent_disruptions + +SPEC = {"root": "root", "elements": { + "root": {"type": "dashboard_grid", "children": ["table"]}, + "table": {"type": "data_grid", "props": { + "rows": {"$state": "/recent_disruptions"}, "columns": ["flight_number", "type"]}}}} + + +class OperationsTests(unittest.TestCase): + def test_admission_rejects_invalid_decisions_and_arguments(self): + from src.operations import admit + for decision in [None, {}, {"kind": "unknown", "calls": [], "interpretation": "x"}, + {"kind": "data_update", "calls": [], "interpretation": "x"}]: + with self.subTest(decision=decision): + with self.assertRaises(ValueError): + admit(decision, {}, None) + for args in [{"limit": None}, {"limit": True}, {"limit": 0}, {"limit": 101}, + {"limit": 1.5}, {"type": "other"}, {"extra": 1}]: + with self.subTest(args=args), self.assertRaises(ValueError): + admit({"kind": "data_update", "interpretation": "filter", "calls": [ + {"name": "query_recent_disruptions", "args": args}]}, {}, SPEC) + + def test_only_missing_exact_dependencies_are_required(self): + from src.operations import admit + decision = {"kind": "create", "interpretation": "create", "calls": [ + {"name": "render_spec", "args": SPEC}, {"name": "query_recent_disruptions", "args": {}}]} + self.assertEqual(len(admit(decision, {}, None)["calls"]), 2) + with self.assertRaises(ValueError): + admit({**decision, "calls": decision["calls"][:1]}, {}, None) + reuse = {**decision, "kind": "restructure", "calls": decision["calls"][:1]} + self.assertEqual(len(admit(reuse, {"recent_disruptions": None}, SPEC)["calls"]), 1) + with self.assertRaises(ValueError): + admit({**decision, "kind": "restructure"}, {"recent_disruptions": []}, SPEC) + with self.assertRaises(ValueError): + admit(decision, {}, SPEC) + + def test_native_spec_literal_nulls_tree_and_binding_contract(self): + from src.dashboard_contract import validate_spec, bindings_resolve + literal = deepcopy(SPEC) + literal["elements"]["table"]["props"]["rows"] = None + self.assertEqual(validate_spec(literal), literal) + self.assertTrue(bindings_resolve(SPEC, {"recent_disruptions": None})) + metric = {"root": "m", "elements": {"m": {"type": "stat_card", "props": { + "label": "Metric", "value": {"$state": "/on_time/value"}}}}} + self.assertFalse(bindings_resolve(metric, {"on_time": None})) + for change in [lambda s: s["elements"]["root"].update(type="Stack"), + lambda s: s["elements"]["root"].update(children=["root"]), + lambda s: s["elements"].update(orphan={"type": "container"}), + lambda s: s["elements"]["table"]["props"].update(rows={"$state": "/foreign"}), + lambda s: s["elements"]["table"]["props"].update(extra=1)]: + value = deepcopy(SPEC) + change(value) + with self.assertRaises(ValueError): + validate_spec(value) + + def test_result_shapes_filters_limits_and_copy(self): + from src.dashboard_contract import data_slots + rows = deepcopy(query_recent_disruptions.invoke({"type": "cancelled"})) + copied = data_slots("query_recent_disruptions", rows, {"type": "cancelled"}) + self.assertEqual([r["flight_number"] for r in copied["recent_disruptions"]], ["AA456", "UA204", "UA640"]) + rows[0]["type"] = "changed" + self.assertEqual(copied["recent_disruptions"][0]["type"], "cancelled") + for name, data, args in [ + ("query_recent_disruptions", copied["recent_disruptions"], {"limit": 2}), + ("query_recent_disruptions", copied["recent_disruptions"], {"type": "delayed"}), + ("query_flights_by_airline", [{"airline": "United", "count": 2}], {"airlines": ["Delta"]}), + ("query_airline_kpis", {"on_time": {"value": 1, "delta": None}}, {}), + ("query_on_time_trend", [{"month": "x", "on_time_pct": 101}], {}), + ("query_recent_disruptions", [{"flight_number": "x", "type": "delayed", "minutes": -1, "route": "x", "date": "x"}], {})]: + with self.subTest(name=name, args=args), self.assertRaises(ValueError): + data_slots(name, data, args) + + def test_json_budgets_use_utf16(self): + from src.dashboard_contract import capture, parse_data + with self.assertRaises(ValueError): + capture("😀" * 50001) + with self.assertRaises(ValueError): + parse_data(" " * 100001) + value = None + for _ in range(34): + value = [value] + with self.assertRaises(ValueError): + capture(value) + + def test_exact_argument_names_defaults_bounds_and_all_airline_selections(self): + from src.operations import arguments + for name, args in [("query_airline_kpis", {}), ("query_on_time_trend", {"months": 100}), + ("query_recent_disruptions", {"limit": 1, "type": None}), + ("query_flights_by_airline", {}), ("query_flights_by_airline", {"airlines": None}), + ("query_flights_by_airline", {"airlines": []}), + ("query_flights_by_airline", {"airlines": ["United"]})]: + self.assertEqual(arguments(name, args), args) + for name, args in [("query_airline_kpis", {"foreign": 1}), + ("query_on_time_trend", {"months": None}), + ("query_on_time_trend", {"months": False}), + ("query_flights_by_airline", {"airlines": "United"}), + ("query_flights_by_airline", {"airlines": [""]}), + ("query_flights_by_airline", {"airlines": ["A"] * 101}), + ("query_flights_by_airline", {"airlines": ["😀" * 257]}), + ("query_recent_disruptions", {"type": ""})]: + with self.subTest(name=name, args=args), self.assertRaises(ValueError): + arguments(name, args) + + def test_full_native_catalog_batch_and_duplicate_bounds(self): + from src.operations import admit + from src.dashboard_contract import validate_spec + spec = {"root": "root", "elements": { + "root": {"type": "dashboard_grid", "children": ["container", "trend", "airlines", "table"]}, + "container": {"type": "container", "props": {"direction": "row"}, "children": ["metric"]}, + "metric": {"type": "stat_card", "props": {"label": "On time", "value": {"$state": "/on_time/value"}, "delta": {"$state": "/on_time/delta"}}}, + "trend": {"type": "line_chart", "props": {"data": {"$state": "/on_time_trend"}, "xKey": "month", "yKey": "on_time_pct"}}, + "airlines": {"type": "bar_chart", "props": {"data": {"$state": "/flights_by_airline"}, "labelKey": "airline", "valueKey": "count"}}, + "table": SPEC["elements"]["table"], + }} + selected = {"kind": "create", "calls": [{"name": "render_spec", "args": spec}, + *[{"name": name, "args": {}} for name in ("query_airline_kpis", "query_on_time_trend", "query_flights_by_airline", "query_recent_disruptions")]], "interpretation": ""} + self.assertEqual(len(admit(selected, {}, None)["calls"]), 5) + for value in [{**selected, "calls": selected["calls"] * 2}, + {**selected, "calls": selected["calls"][:2] + [selected["calls"][1]]}, + {**selected, "foreign": True}, + {**selected, "kind": "unsupported"}, + {**selected, "calls": [{"name": "render_spec", "args": SPEC}, {"name": "query_airline_kpis", "args": {}}]}]: + with self.subTest(value=value), self.assertRaises(ValueError): + admit(value, {}, None) + for modify in [lambda v: v["elements"].update(constructor={"type": "container"}), + lambda v: v["elements"]["root"].update(children=["container", "container"]), + lambda v: v["elements"]["metric"].update(children=["table"]), + lambda v: v["elements"]["trend"]["props"].update(yKey="wrong"), + lambda v: v["elements"]["table"]["props"].update(columns=["type", "type"]), + lambda v: v["elements"]["airlines"]["props"].update(data=[{"airline": "x", "count": -1}])]: + value = deepcopy(spec) + modify(value) + with self.assertRaises(ValueError): + validate_spec(value) + + def test_default_row_bounds_nonfinite_exact_metric_and_literal_data(self): + from src.dashboard_contract import data_slots, validate_dashboard, validate_spec + for name, value in [("query_recent_disruptions", deepcopy(query_recent_disruptions.invoke({"limit": 6}))), + ("query_on_time_trend", [{"month": "now", "on_time_pct": 1}] * 13), + ("query_flights_by_airline", [{"airline": "United", "count": float("nan")}])]: + with self.assertRaises(ValueError): + data_slots(name, value, {}) + metrics = {key: {"value": 1, "delta": None} for key in ("on_time", "flights_today", "avg_delay", "load_factor")} + self.assertEqual(data_slots("query_airline_kpis", metrics, {}), metrics) + for bad in [{"on_time": {"value": "", "delta": None}}, {"on_time": {"value": 1, "delta": ""}}, + {"on_time": {"value": True, "delta": None}}, {"foreign": None}, + {"on_time_trend": [{"month": "x", "on_time_pct": -1}]}]: + with self.assertRaises(ValueError): + validate_dashboard(bad) + literal = deepcopy(SPEC) + literal["elements"]["table"]["props"]["rows"] = [] + self.assertEqual(validate_spec(literal), literal) + + +if __name__ == "__main__": + unittest.main() diff --git a/cockpit/chat/generative-ui/python/tests/test_streaming.py b/cockpit/chat/generative-ui/python/tests/test_streaming.py index e9b3352cd..d2df1f728 100644 --- a/cockpit/chat/generative-ui/python/tests/test_streaming.py +++ b/cockpit/chat/generative-ui/python/tests/test_streaming.py @@ -19,10 +19,6 @@ from langchain_core.outputs import ChatGeneration, ChatGenerationChunk, ChatResult from langgraph.checkpoint.memory import InMemorySaver -LAYOUT = {"elements": {"root": {"type": "Stack", "children": ["metric"]}, - "metric": {"type": "Metric", "props": {"label": "Flights"}}}, "root": "root"} - - def call(name, args=None, identity=None): return {"name": name, "args": args or {}, "id": identity or name, "type": "tool_call"} @@ -62,6 +58,16 @@ def _stream(self, messages, stop=None, run_manager=None, **kwargs): yield ChatGenerationChunk(message=AIMessageChunk(id=answer.id, content="", chunk_position="last")) + +from tests.test_operations import SPEC +LAYOUT = SPEC + + +def decision(kind, calls=(), interpretation=""): + return call("plan_dashboard_operation", {"kind": kind, "calls": [ + {"name": c["name"], "args": c["args"]} for c in calls], "interpretation": interpretation}, "model-decision") + + class StreamingTests(unittest.IsolatedAsyncioTestCase): def setup_graph(self, *, fail_title=False, fail_read=False, fail_write=False): network = self.enterContext(patch.object(socket.socket, "connect", side_effect=AssertionError("Remote socket forbidden"))) @@ -71,7 +77,7 @@ def setup_graph(self, *, fail_title=False, fail_read=False, fail_write=False): get=AsyncMock(return_value={"metadata": {}}, side_effect=RuntimeError("Metadata read failure") if fail_read else None), update=AsyncMock(side_effect=RuntimeError("Metadata write failure") if fail_write else None)) self.enterContext(patch("langgraph_sdk.get_client", return_value=SimpleNamespace(threads=self.threads))) - self.agent_model = LocalModel() + self.agent_model = LocalModel(tags=["nostream"]) self.respond_model = LocalModel() self.enterContext(patch("langchain_openai.ChatOpenAI", side_effect=lambda **kw: (self.agent_model if kw["model"] == "gpt-5" else self.respond_model) if kw.get("streaming") else @@ -79,230 +85,543 @@ def setup_graph(self, *, fail_title=False, fail_read=False, fail_write=False): spec = importlib.util.spec_from_file_location("local_dashboard", Path(__file__).parents[1] / "src/graph.py") self.module = importlib.util.module_from_spec(spec) spec.loader.exec_module(self.module) - self.graph = self.module.graph.builder.compile(checkpointer=InMemorySaver()) + self.recompile() self.config = {"configurable": {"thread_id": "dashboard"}} - async def send(self, identity, batches=(), *, prose="", answer_id=None, messages=None, copy_events=True): - self.agent_model.responses = [AIMessage(id=f"{identity}-agent-{i}", content=prose if i == 0 else "", tool_calls=batch) - for i, batch in enumerate(batches)] + [AIMessage(id=f"{identity}-done", content="Ready.")] + def recompile(self): + self.graph = self.module.graph.builder.compile(checkpointer=InMemorySaver()) + + async def send(self, identity, selected=None, *, response=None, messages=None, input_extra=None, copy_events=True): + self.agent_model.responses = [response or AIMessage(id="historical-model-id", content="I refreshed everything.", tool_calls=[selected] if selected else [])] self.agent_model.cursor = 0 - self.respond_model.responses = [AIMessage(id=answer_id if answer_id is not None else f"{identity}-answer", content="Dashboard updated.")] + self.respond_model.responses = [AIMessage(id=identity, content="Confirmed data explanation.")] self.respond_model.cursor = 0 - events = [(deepcopy(event) if copy_events else event) async for event in self.graph.astream( - {"messages": messages or [HumanMessage(id=identity, content=identity)]}, self.config, - stream_mode=["messages", "values", "updates", "custom", "checkpoints"])] + events = [(deepcopy(e) if copy_events else e) async for e in self.graph.astream( + {"messages": messages if messages is not None else [HumanMessage(id=identity, content=identity)], **(input_extra or {})}, + self.config, stream_mode=["messages", "values", "updates", "custom", "checkpoints"])] saved = await self.graph.aget_state(self.config) self.assertFalse(saved.next) self.assertFalse(saved.tasks) - values = [event for kind, event in events if kind == "values"] - self.assertEqual(values[-1], saved.values) - self.assertTrue([event for kind, event in events if kind == "checkpoints"]) + self.assertEqual([e for k, e in events if k == "values"][-1], saved.values) + self.assertTrue([e for k, e in events if k == "checkpoints"]) return saved, events - def assert_completion(self, saved, identity): - ids = [m.id for m in saved.values["messages"]] - self.assertEqual(saved.values.get("completed_turn_id"), identity) - self.assertEqual(saved.values.get("completed_answer_id"), ids[-1]) - self.assertEqual(saved.values.get("completed_message_ids"), ids) - self.assertEqual(len(ids), len(set(ids))) - self.assertNotIn("run_id", saved.values) + async def initial(self, spec=LAYOUT): + queries = [call("query_recent_disruptions")] if spec == LAYOUT else [] + return await self.send("initial", decision("create", [call("render_spec", spec), *queries])) def patches(self, events): - return [event["data"] for kind, event in events if kind == "custom" and event["name"] == "state_update"] + return [e["data"] for k, e in events if k == "custom" and e["name"] == "state_update"] - async def test_initial_dashboard_persists_and_wrap_keeps_correlated_identity(self): + def assert_completion(self, saved, identity): + ids = [m.id for m in saved.values["messages"]] + self.assertEqual(saved.values["completed_turn_id"], identity) + self.assertEqual(saved.values["completed_answer_id"], ids[-1]) + self.assertEqual(saved.values["completed_message_ids"], ids) + self.assertEqual(len(set(ids)), len(ids)) + self.assertTrue(ids[-1].startswith("answer-")) + + async def test_no_tool_filter_cannot_claim_refresh(self): self.setup_graph() - batch = [call("render_spec", LAYOUT), *[call(t.name) for t in self.module._DATA_TOOLS]] - saved, events = await self.send("initial", [batch]) - self.assertIn("dashboard", saved.values) - self.assertEqual(saved.values["dashboard"]["on_time"]["value"], "84.2%") - self.assertEqual(len(saved.values["dashboard"]["on_time_trend"]), 12) - self.assertEqual(len(saved.values["dashboard"]["flights_by_airline"]), 4) + await self.initial() + saved, events = await self.send("Filter to only the cancelled flights.") + self.assertNotIn("updated", saved.values["messages"][-1].content.lower()) + self.assertIn("No dashboard changes", saved.values["messages"][-1].content) + self.assertEqual(self.patches(events), []) + self.assertEqual(saved.values["operation_receipt"]["outcome"], "rejected") self.assertEqual(len(saved.values["dashboard"]["recent_disruptions"]), 5) - self.assert_completion(saved, "initial") - observations = [v["messages"] for kind, v in events if kind == "values"] - before = next(ms for ms in observations if any(m.type == "tool" and m.name == "render_spec" for m in ms)) - after = next(ms for ms in observations if any(m.type == "tool" and m.name == "render_spec" and m.content == "rendered" for m in ms)) + self.assertEqual(self.respond_model.cursor, 0) + + async def test_cancelled_decision_executes_once_and_receipt_matches(self): + self.setup_graph() + initial, _ = await self.initial() + saved, events = await self.send("filter", decision("data_update", [call("query_recent_disruptions", {"type": "cancelled"})])) + rows = saved.values["dashboard"]["recent_disruptions"] + self.assertEqual([r["flight_number"] for r in rows], ["AA456", "UA204", "UA640"]) + self.assertEqual([r["type"] for r in rows], ["cancelled"] * 3) + self.assertEqual(self.patches(events), [{"/recent_disruptions": rows}]) + receipt = saved.values["operation_receipt"] + self.assertEqual(receipt["submitted_human_id"], "filter") + self.assertEqual(receipt["outcome"], "applied") + self.assertEqual(receipt["validated_slots"], {"recent_disruptions": rows}) + self.assertEqual(len(receipt["admitted_calls"]), 1) + self.assertEqual(len(receipt["results"]), 1) + self.assertEqual(receipt["admitted_calls"][0]["args"], {"type": "cancelled"}) + self.assertEqual(receipt["results"][0]["tool_call_id"], receipt["admitted_calls"][0]["id"]) + self.assertTrue(receipt["admitted_calls"][0]["id"].startswith("call-")) + self.assertEqual(saved.values["_owned_layout"], initial.values["_owned_layout"]) + self.assertEqual(saved.values["_render_owner"], initial.values["_render_owner"]) + self.assertEqual(saved.values["messages"][:len(initial.values["messages"])], initial.values["messages"]) + self.assertIn("3 cancelled", saved.values["messages"][-1].content) + self.assertEqual(self.respond_model.cursor, 0) + self.assert_completion(saved, "filter") + self.assertEqual((await self.graph.aget_state(initial.config)).values, initial.values) + + async def test_raw_rendered_wrapped_transitions_preserve_correlations(self): + self.setup_graph() + saved, events = await self.initial() + observations = [v["messages"] for k, v in events if k == "values"] + before = next(ms for ms in observations if any(isinstance(m, ToolMessage) and m.name == "render_spec" and m.content != "rendered" for m in ms)) + after = next(ms for ms in observations if any(isinstance(m, ToolMessage) and m.name == "render_spec" and m.content == "rendered" for m in ms)) self.assertEqual([m.id for m in before], [m.id for m in after]) - raw = next(m for m in before if m.type == "tool" and m.name == "render_spec") - parent = next(m for m in before if m.id == "initial-agent-0") - rewritten = next(m for m in after if m.id == parent.id) + raw = next(m for m in before if isinstance(m, ToolMessage) and m.name == "render_spec") + parent = next(m for m in before if isinstance(m, AIMessage) and m.tool_calls) + wrapped = next(m for m in after if m.id == parent.id) self.assertEqual(parent.content, "") - self.assertEqual(rewritten.content, raw.content) - self.assertEqual(json.loads(raw.content), LAYOUT) - self.assertEqual(rewritten.tool_calls, parent.tool_calls) + self.assertEqual(json.loads(wrapped.content), json.loads(raw.content)) + self.assertEqual(wrapped.tool_calls, parent.tool_calls) rendered = next(m for m in after if m.id == raw.id) - self.assertEqual((rendered.name, rendered.tool_call_id, rendered.status, rendered.content), - (raw.name, raw.tool_call_id, raw.status, "rendered")) + self.assertEqual((rendered.name, rendered.tool_call_id, rendered.status), (raw.name, raw.tool_call_id, raw.status)) + self.assertEqual(rendered.content, "rendered") for old, new in zip(before, after): if old.id not in (parent.id, raw.id): self.assertEqual(old, new) - self.assertEqual(self.patches(events)[0]["/on_time/value"], "84.2%") - terminal = [event for kind, event in events if kind == "checkpoints"][-1] - self.assertEqual(terminal["values"], saved.values) - self.assertFalse(terminal["next"]) - self.assertFalse(terminal["tasks"]) - callbacks = [event for kind, event in events if kind == "messages"] - self.assertFalse([event for event in callbacks if event[1]["langgraph_node"] == "wrap_spec_into_ai"]) - self.assertTrue(any(chunk.get("name") is None and chunk.get("args") for message, _ in callbacks - for chunk in getattr(message, "tool_call_chunks", []))) - - async def test_same_tool_latest_result_wins_in_custom_patch(self): + self.assertEqual(saved.values["operation_receipt"]["render_owner"], + {"parent_id": parent.id, "call_id": raw.tool_call_id, "result_id": raw.id}) + self.assert_completion(saved, "initial") + self.assertFalse([e for k, e in events if k == "messages" and e[1]["langgraph_node"] in ("decide", "wrap_spec_into_ai", "generate_title")]) + + async def test_absent_multiple_unknown_invalid_and_refused_decisions(self): + variants = [ + AIMessage(content="Done."), + AIMessage(content="", tool_calls=[decision("unsupported"), decision("unsupported")]), + AIMessage(content="", tool_calls=[call("unknown")]), + AIMessage(content="", invalid_tool_calls=[{"name": "plan_dashboard_operation", "args": "{", "id": "invalid", "error": "bad"}]), + AIMessage(content="", additional_kwargs={"refusal": "No."}), + ] + for response in variants: + with self.subTest(response=response): + self.setup_graph() + # _generate preserves invalid_tool_calls/refusal exactly; only model boundary mocked. + with patch.object(LocalModel, "ainvoke", AsyncMock(return_value=response)): + saved, events = await self.send("rejected", response=response) + self.assertEqual(saved.values["operation_receipt"]["outcome"], "rejected") + self.assertEqual(saved.values["operation_receipt"]["admitted_calls"], []) + self.assertFalse([m for m in saved.values["messages"] if isinstance(m, ToolMessage)]) + self.assertEqual(self.patches(events), []) + + async def test_initial_category_and_restructure_reuse_only_missing(self): self.setup_graph() - saved, events = await self.send("repeat", [[call("query_recent_disruptions", {"limit": 5}, "first")], - [call("query_recent_disruptions", {"type": "cancelled"}, "last")]]) - result = self.patches(events)[0]["/recent_disruptions"] - self.assertEqual([r["type"] for r in result], ["cancelled"] * 3) - self.assertEqual(saved.values["dashboard"]["recent_disruptions"], result) + rejected, _ = await self.send("wrong", decision("data_update", [call("query_recent_disruptions")])) + self.assertEqual(rejected.values["operation_receipt"]["outcome"], "rejected") + initial, _ = await self.initial() + changed = deepcopy(LAYOUT) + changed["elements"]["table"]["props"]["title"] = "Saved rows" + saved, events = await self.send("structure", decision("restructure", [call("render_spec", changed)])) + self.assertEqual(saved.values["operation_receipt"]["outcome"], "applied") + self.assertEqual(saved.values["dashboard"], initial.values["dashboard"]) + self.assertEqual(self.patches(events), []) + rejected, _ = await self.send("duplicate-create", decision("create", [call("render_spec", changed), call("query_recent_disruptions")])) + self.assertEqual(rejected.values["operation_receipt"]["outcome"], "rejected") - async def test_followups_preserve_slots_and_prior_messages(self): + async def test_literal_null_layout_owns_dashboard_without_data(self): self.setup_graph() - initial, _ = await self.send("initial", [[call("render_spec", LAYOUT), call("query_airline_kpis"), call("query_recent_disruptions")]]) - filtered, events = await self.send("filter", [[call("query_recent_disruptions", {"type": "cancelled"})]]) - self.assertEqual(filtered.values.get("dashboard", {}).get("on_time"), {"value": "84.2%", "delta": "+1.4%"}) - self.assertEqual(filtered.values["messages"][:len(initial.values["messages"])], initial.values["messages"]) - self.assertEqual(len(self.patches(events)[0]), 1) - layout = {"elements": {"new": {"type": "Stack"}}, "root": "new"} - structural, _ = await self.send("structure", [[call("render_spec", layout)]]) - self.assertEqual(structural.values["dashboard"], filtered.values["dashboard"]) - self.assertEqual(json.loads(next(m.content for m in structural.values["messages"] if m.id == "structure-agent-0")), layout) - prose, events = await self.send("prose") - self.assertEqual(prose.values["dashboard"], filtered.values["dashboard"]) + literal = {"root": "metric", "elements": {"metric": {"type": "stat_card", "props": {"label": "Literal", "value": None, "delta": None}}}} + saved, events = await self.initial(literal) + self.assertEqual(saved.values["dashboard"], {}) + self.assertEqual(saved.values["_owned_layout"], literal) self.assertEqual(self.patches(events), []) - self.assert_completion(prose, "prose") - self.assertEqual((await self.graph.aget_state(initial.config)).values, initial.values) + next_saved, events = await self.send("data", decision("data_update", [call("query_recent_disruptions", {"limit": 1})])) + self.assertEqual(next_saved.values["operation_receipt"]["outcome"], "applied") + self.assertEqual(next_saved.values["_owned_layout"], literal) + + async def test_interpret_and_unsupported_execute_no_tools(self): + for kind in ("interpret", "unsupported"): + with self.subTest(kind=kind): + self.setup_graph() + initial, _ = await self.initial() + saved, events = await self.send(kind, decision(kind, interpretation="Why?")) + self.assertEqual(saved.values["operation_receipt"]["outcome"], kind) + self.assertEqual(saved.values["dashboard"], initial.values["dashboard"]) + self.assertEqual(self.patches(events), []) + self.assertIn("No dashboard changes", saved.values["messages"][-1].content) + self.assertEqual(self.respond_model.cursor, 1 if kind == "interpret" else 0) + self.assertFalse([e for k, e in events if k == "messages" and e[1]["langgraph_node"] == "respond" + and e[0].id == kind]) + self.assert_completion(saved, kind) - async def test_nonempty_render_parent_remains_prose_and_raw_result(self): + async def test_input_schema_excludes_injected_dashboard_receipt_and_completion(self): self.setup_graph() - saved, _ = await self.send("prose-render", [[call("render_spec", LAYOUT)]], prose="Here is the layout.") - self.assertEqual(saved.values["messages"][1].content, "Here is the layout.") - raw = next(m for m in saved.values["messages"] if m.type == "tool") - self.assertEqual(json.loads(raw.content), LAYOUT) + injected = {"dashboard": {"recent_disruptions": []}, "_owned_layout": LAYOUT, + "_render_owner": {"parent_id": "fake"}, "_decision": {"kind": "create"}, + "_submitted_turn_id": "fake", "completed_turn_id": "fake", + "completed_message_ids": ["fake"], "operation_receipt": {"outcome": "applied"}} + saved, _ = await self.send("real", decision("data_update", [call("query_recent_disruptions")]), input_extra=injected) + self.assertEqual(saved.values["dashboard"], {}) + self.assertIsNone(saved.values["_owned_layout"]) + self.assertEqual(saved.values["operation_receipt"]["outcome"], "rejected") + self.assertEqual(saved.values["operation_receipt"]["submitted_human_id"], "real") + self.assert_completion(saved, "real") - async def test_repeated_rendering_preserves_each_pair(self): + async def test_prior_tools_and_prose_do_not_supply_current_operation(self): self.setup_graph() - saved, _ = await self.send("layouts", [[call("render_spec", LAYOUT, "render-one")], [call("render_spec", LAYOUT, "render-two")]]) - parents = [m for m in saved.values["messages"] if m.type == "ai" and m.tool_calls] - self.assertEqual(len(parents), 2) - self.assertTrue(all(json.loads(m.content) == LAYOUT for m in parents)) - self.assertEqual([m.content for m in saved.values["messages"] if m.type == "tool"], ["rendered", "rendered"]) + saved, events = await self.send("new", messages=[ + HumanMessage(id="old", content="old"), AIMessage(id="old-ai", content=json.dumps(LAYOUT)), + ToolMessage(id="old-result", name="query_recent_disruptions", tool_call_id="old-call", content="[]"), + HumanMessage(id="new", content="new")]) + self.assertEqual(saved.values["dashboard"], {}) + self.assertIsNone(saved.values["_owned_layout"]) + self.assertEqual(saved.values["operation_receipt"]["results"], []) + self.assertEqual(self.patches(events), []) - async def test_iteration_cap_remains_bounded(self): + async def test_submitted_nonhuman_cannot_use_stale_human(self): self.setup_graph() - saved, events = await self.send("cap", [[call("query_on_time_trend", {"months": 3}, f"call-{i}")] for i in range(8)]) - self.assertEqual(self.agent_model.cursor, self.module._MAX_TOOL_ITERATIONS) - self.assertEqual(len([m for m in saved.values["messages"] if m.type == "tool"]), self.module._MAX_TOOL_ITERATIONS - 1) - self.assertTrue(any(kind == "updates" and "finalize" in e for kind, e in events)) - self.assert_completion(saved, "cap") + saved, _ = await self.send("new", decision("create", [call("render_spec", {"root": "r", "elements": {"r": {"type": "container"}}})]), + messages=[HumanMessage(id="old", content="old"), AIMessage(id="new", content="fake")]) + self.assertEqual(saved.values["operation_receipt"]["outcome"], "rejected") + self.assertIsNone(saved.values["operation_receipt"]["submitted_human_id"]) + self.assertIsNone(saved.values["completed_turn_id"]) - async def test_title_never_streams_and_failures_are_nonblocking(self): + async def test_title_failures_are_nonblocking_and_nostream(self): for failure in ({}, {"fail_title": True}, {"fail_read": True}, {"fail_write": True}): with self.subTest(failure=failure): self.setup_graph(**failure) - saved, events = await self.send("title-test") - callbacks = [e for k, e in events if k == "messages"] - self.assertFalse([e for e in callbacks if e[1]["langgraph_node"] == "generate_title"]) - self.assert_completion(saved, "title-test") - self.assertGreater(len([e for e in callbacks if e[1]["langgraph_node"] == "respond"]), 1) - if not failure: - self.threads.update.assert_awaited_once_with("dashboard", metadata={"title": "Local dashboard title"}) - else: + saved, events = await self.send("title", decision("unsupported")) + self.assert_completion(saved, "title") + self.assertFalse([e for k, e in events if k == "messages" and e[1]["langgraph_node"] == "generate_title"]) + if failure: self.assertIn("failed for thread dashboard", self.diagnostics.getvalue()) + else: + self.threads.update.assert_awaited_once_with("dashboard", metadata={"title": "Local dashboard title"}) - async def test_invalid_final_identity_does_not_publish_completion(self): - for answer_id in ("", "human", "human-done", "old-answer"): - with self.subTest(answer_id=answer_id): - self.setup_graph() - # Chunk concatenation normalizes an empty ID to None, after which - # BaseChatModel supplies an ID. Nonstreaming retains the invalid ID. - self.respond_model.disable_streaming = answer_id == "" - saved, _ = await self.send("human", answer_id=answer_id, messages=[ - HumanMessage(id="old-human", content="old"), AIMessage(id="old-answer", content="old"), HumanMessage(id="human", content="new")]) - self.assertFalse(saved.values.get("completed_turn_id")) - - async def test_trailing_assistant_does_not_fabricate_completion(self): - self.setup_graph() - saved, _ = await self.send("new", messages=[HumanMessage(id="old-human", content="old"), AIMessage(id="old-answer", content="answered")]) - self.assertFalse(saved.values.get("completed_turn_id")) - - async def test_new_failed_turn_clears_stale_completion(self): + async def test_model_failure_clears_stale_completion_and_preserves_failed_run(self): self.setup_graph() - confirmed, _ = await self.send("confirmed") - self.assert_completion(confirmed, "confirmed") + initial, _ = await self.initial() self.agent_model.fail = True with self.assertRaisesRegex(RuntimeError, "Local title failure"): - await self.send("failed") + await self.send("failure") saved = await self.graph.aget_state(self.config) - self.assertEqual(saved.next, ("agent",)) + self.assertEqual(saved.values["dashboard"], initial.values["dashboard"]) + self.assertIsNone(saved.values["operation_receipt"]) self.assertIsNone(saved.values["completed_turn_id"]) - self.assertIsNone(saved.values["completed_answer_id"]) - self.assertIsNone(saved.values["completed_message_ids"]) - self.assertEqual((await self.graph.aget_state(confirmed.config)).values, confirmed.values) + self.assertEqual(saved.next, ("decide",)) - async def test_prior_turn_results_are_not_replayed(self): + async def test_graph_owned_ids_ignore_model_historical_ids(self): self.setup_graph() - saved, events = await self.send("new", messages=[HumanMessage(id="prior", content="old"), - ToolMessage(id="old-result", name="query_airline_kpis", tool_call_id="old-call", - content='{"on_time":{"value":"999%","delta":"old"}}'), HumanMessage(id="new", content="new")]) - self.assertEqual(saved.values["dashboard"], {}) - self.assertEqual(self.patches(events), []) + saved, _ = await self.initial() + self.assertNotIn("historical-model-id", [m.id for m in saved.values["messages"]]) + self.assertNotIn("model-decision", [c["id"] for c in saved.values["operation_receipt"]["admitted_calls"]]) + self.assert_completion(saved, "initial") - async def test_malformed_error_and_unknown_current_results_are_ignored(self): - from langchain_core.tools import StructuredTool + async def test_invalid_transcript_identity_prevents_completion(self): + self.setup_graph() + saved, _ = await self.send("human", decision("unsupported"), + messages=[AIMessage(id="", content="invalid"), HumanMessage(id="human", content="human")]) + self.assertIsNone(saved.values["completed_turn_id"]) + + async def test_empty_and_idempotent_updates_are_applied(self): + self.setup_graph() + await self.initial() + for identity, args in [("same", {}), ("empty", {"type": "cancelled", "limit": 1})]: + if identity == "empty": + from langchain_core.tools import StructuredTool + from langgraph.prebuilt import ToolNode + def empty(limit: int = 5, type: str | None = None) -> list: + return [] + self.module._tool_node = ToolNode([StructuredTool.from_function(empty, name="query_recent_disruptions", description="Empty observation")]) + saved, events = await self.send(identity, decision("data_update", [call("query_recent_disruptions", args)])) + self.assertEqual(saved.values["operation_receipt"]["outcome"], "applied") + self.assertEqual(len(self.patches(events)), 1) + if identity == "empty": + self.assertEqual(saved.values["dashboard"]["recent_disruptions"], []) + + async def test_copy_ownership_between_patch_receipt_and_checkpoint(self): + self.setup_graph() + saved, events = await self.send("initial", decision("create", [call("render_spec", LAYOUT), call("query_recent_disruptions")]), copy_events=False) + expected = deepcopy(saved.values["dashboard"]) + patch_data = self.patches(events)[0] + patch_data["/recent_disruptions"][0]["type"] = "changed" + self.assertEqual(saved.values["dashboard"], expected) + self.assertEqual(saved.values["operation_receipt"]["validated_slots"], expected) + saved.values["operation_receipt"]["validated_slots"]["recent_disruptions"][0]["type"] = "changed" + saved.values["dashboard"]["recent_disruptions"][0]["type"] = "changed" + self.assertEqual((await self.graph.aget_state(self.config)).values["dashboard"], expected) + + async def test_malformed_failed_missing_or_uncorrelated_results_roll_back_entire_batch(self): from langgraph.prebuilt import ToolNode - for name, content, status in [ - ("query_airline_kpis", "[]", "success"), ("query_airline_kpis", "null", "success"), - ("query_airline_kpis", "not-json", "success"), ("query_airline_kpis", '{"error":"failed"}', "success"), - ("query_airline_kpis", '{"on_time":{"value":"999%","delta":"bad"}}', "error"), - ("query_on_time_trend", '{"error":"failed"}', "success"), - ("query_recent_disruptions", '"bad"', "success"), ("query_flights_by_airline", "[1]", "success"), - ("query_on_time_trend", '[{"month":"now","on_time_pct":NaN}]', "success"), - ("query_on_time_trend", json.dumps([{"month": "now", "on_time_pct": 10**400}]), "success"), - ("query_airline_kpis", '{"foreign":{"value":1,"delta":"x"}}', "success"), - ("unknown", '{"on_time":{"value":"999%"}}', "success")]: - with self.subTest(name=name, content=content, status=status): + from langchain_core.tools import StructuredTool + variants = ["malformed", "error", "missing", "wrong-name", "wrong-call", "overlimit", "wrongfilter"] + for variant in variants: + with self.subTest(variant=variant): self.setup_graph() - def result(): - return ToolMessage(name=name, tool_call_id=name, content=content, status=status) - controlled = StructuredTool.from_function(result, name=name, description="Controlled observation") - self.module.graph.builder.nodes["tools"].runnable = ToolNode([controlled]) - self.graph = self.module.graph.builder.compile(checkpointer=InMemorySaver()) - saved, events = await self.send("bad", [[call(name)]]) - self.assertEqual(saved.values["dashboard"], {}) + initial, _ = await self.initial() + spec = deepcopy(LAYOUT) + spec["elements"]["root"]["children"].append("trend") + spec["elements"]["trend"] = {"type": "line_chart", "props": {"data": {"$state": "/on_time_trend"}, "xKey": "month", "yKey": "on_time_pct"}} + # Missing trend requires exactly this query; make its actual observed output invalid. + def observed(months=12): + if variant == "error": + raise ValueError("Unavailable") + if variant == "overlimit": + return [{"month": "x", "on_time_pct": 1}] * 13 + return "not-json" + self.module._tool_node = ToolNode([self.module.render_spec, StructuredTool.from_function(observed, name="query_on_time_trend", description="Controlled result")], handle_tool_errors=True) + if variant in ("missing", "wrong-name", "wrong-call"): + real_executor = self.module.execute_tools + async def controlled(state, config): + output = await real_executor(state, config) + if variant == "missing": + output["messages"] = output["messages"][:1] + output["_result_ids"] = output["_result_ids"][:1] + elif variant == "wrong-name": + output["messages"][1].name = "query_recent_disruptions" + else: + output["messages"][1].tool_call_id = "unrelated" + return output + self.module.graph.builder.nodes["tools"].runnable = controlled + # runnable must use LangGraph's normal coercion via a new graph node registration. + from langchain_core.runnables import RunnableLambda + self.module.graph.builder.nodes["tools"].runnable = RunnableLambda(controlled) + self.recompile() + # preserve initial checkpoint in the newly compiled graph. + await self.graph.aupdate_state(self.config, initial.values, as_node="generate_title") + saved, events = await self.send("bad", decision("restructure", [call("render_spec", spec), call("query_on_time_trend")])) + self.assertEqual(saved.values["operation_receipt"]["outcome"], "unfulfilled") + self.assertEqual(saved.values["dashboard"], initial.values["dashboard"]) + self.assertEqual(saved.values["_owned_layout"], initial.values["_owned_layout"]) self.assertEqual(self.patches(events), []) + current = saved.values["operation_receipt"]["results"] + self.assertTrue(any(r["name"] == "render_spec" and r["status"] == "success" and r["content"] != "rendered" for r in current)) + self.assertFalse(any(json.loads(m.content) == spec for m in saved.values["messages"] if isinstance(m, AIMessage) and m.content.startswith("{"))) - async def test_custom_and_persisted_data_do_not_share_mutable_references(self): + async def test_genuine_nonempty_parent_fallback_no_wrapper_or_data(self): self.setup_graph() - saved, events = await self.send("owned", [[call("query_recent_disruptions")]], copy_events=False) - original = deepcopy(saved.values["dashboard"]) - self.patches(events)[0]["/recent_disruptions"][0]["type"] = "mutated" - self.assertEqual(saved.values["dashboard"], original) - terminal = [event for kind, event in events if kind == "values"][-1] - self.assertEqual(terminal["dashboard"], original) - saved.values["dashboard"]["recent_disruptions"][0]["type"] = "mutated" - self.assertEqual((await self.graph.aget_state(self.config)).values["dashboard"], original) - - async def test_parallel_same_tool_results_follow_transcript_order(self): + real_admit = self.module.admit_operation + def prose_parent(state): + output = real_admit(state) + if output.get("messages"): + output["messages"][0].content = "Controlled genuine fallback." + return output + from langchain_core.runnables import RunnableLambda + self.module.graph.builder.nodes["admit"].runnable = RunnableLambda(prose_parent) + self.recompile() + saved, events = await self.initial() + self.assertEqual(saved.values["operation_receipt"]["outcome"], "unfulfilled") + self.assertEqual(saved.values["dashboard"], {}) + self.assertIsNone(saved.values["_owned_layout"]) + self.assertEqual(self.patches(events), []) + self.assertTrue(any(m.content == "Controlled genuine fallback." for m in saved.values["messages"])) + self.assertTrue(any(isinstance(m, ToolMessage) and m.name == "render_spec" and m.status == "success" and m.content != "rendered" for m in saved.values["messages"])) + + async def test_current_only_wrapper_does_not_rewrap_historical_render(self): self.setup_graph() - saved, events = await self.send("parallel", [[call("query_recent_disruptions", {"type": "delayed"}, "one"), - call("query_recent_disruptions", {"type": "cancelled"}, "two")]]) - self.assertEqual([row["type"] for row in saved.values["dashboard"]["recent_disruptions"]], ["cancelled"] * 3) - self.assertEqual(self.patches(events)[0]["/recent_disruptions"], saved.values["dashboard"]["recent_disruptions"]) + initial, _ = await self.initial() + saved, events = await self.send("new", decision("unsupported")) + self.assertEqual(saved.values["messages"][:len(initial.values["messages"])], initial.values["messages"]) + self.assertFalse([e for k, e in events if k == "updates" and (e.get("wrap_spec_into_ai") or {}).get("messages")]) - async def test_newest_kpi_sections_preserve_other_prior_slots(self): + async def test_fenced_minified_render_transfers_exact_validated_text(self): self.setup_graph() from langchain_core.tools import StructuredTool from langgraph.prebuilt import ToolNode - outputs = iter([{"on_time": {"value": "10%", "delta": "old"}, "flights_today": {"value": 312, "delta": "+8"}}, - {"on_time": {"value": "90%", "delta": "new"}}]) - def result(): - return next(outputs) - controlled = StructuredTool.from_function(result, name="query_airline_kpis", description="Controlled KPI observation") - self.module.graph.builder.nodes["tools"].runnable = ToolNode([controlled]) - self.graph = self.module.graph.builder.compile(checkpointer=InMemorySaver()) - saved, events = await self.send("kpis", [[call("query_airline_kpis", identity="first")], [call("query_airline_kpis", identity="last")]]) - self.assertEqual(saved.values["dashboard"], {"on_time": {"value": "90%", "delta": "new"}, "flights_today": {"value": 312, "delta": "+8"}}) - self.assertEqual(self.patches(events), [{"/on_time/value": "90%", "/on_time/delta": "new", "/flights_today/value": 312, "/flights_today/delta": "+8"}]) + rendered = json.dumps(LAYOUT, separators=(",", ":"), ensure_ascii=False) + def fenced(elements: dict, root: str) -> str: + return "```json\n" + rendered + "\n```" + self.module._tool_node = ToolNode([StructuredTool.from_function(fenced, name="render_spec", description="Fenced render"), *self.module._DATA_TOOLS]) + saved, _ = await self.initial() + parent = next(m for m in saved.values["messages"] if isinstance(m, AIMessage) and m.tool_calls) + self.assertEqual(parent.content, rendered) + self.assertEqual(saved.values["operation_receipt"]["outcome"], "applied") + + async def test_mixed_genuine_success_and_runtime_error_is_atomic(self): + self.setup_graph() + initial, _ = await self.initial() + from langchain_core.tools import StructuredTool + from langgraph.prebuilt import ToolNode + def unavailable() -> dict: + raise RuntimeError("Provider secret must not escape") + self.module._tool_node = ToolNode([self.module.render_spec, + next(t for t in self.module._DATA_TOOLS if t.name == "query_on_time_trend"), + StructuredTool.from_function(unavailable, name="query_airline_kpis", description="Failing observation")], handle_tool_errors="The dashboard tool failed.") + spec = deepcopy(LAYOUT) + spec["elements"]["root"]["children"] += ["trend", "metric"] + spec["elements"]["trend"] = {"type": "line_chart", "props": {"data": {"$state": "/on_time_trend"}, "xKey": "month", "yKey": "on_time_pct"}} + spec["elements"]["metric"] = {"type": "stat_card", "props": {"label": "On time", "value": {"$state": "/on_time/value"}}} + saved, events = await self.send("mixed", decision("restructure", [call("render_spec", spec), call("query_on_time_trend", {"months": 3}), call("query_airline_kpis")])) + self.assertEqual(saved.values["operation_receipt"]["outcome"], "unfulfilled") + self.assertEqual(saved.values["dashboard"], initial.values["dashboard"]) + self.assertEqual(saved.values["_owned_layout"], initial.values["_owned_layout"]) + self.assertEqual(self.patches(events), []) + results = saved.values["operation_receipt"]["results"] + success = next(r for r in results if r["name"] == "query_on_time_trend") + self.assertEqual(success["status"], "success") + self.assertEqual(len(json.loads(success["content"])), 3) + failure = next(r for r in results if r["name"] == "query_airline_kpis") + self.assertEqual((failure["status"], failure["content"]), ("error", "The dashboard tool failed.")) + self.assertEqual(next(r for r in results if r["name"] == "render_spec")["status"], "success") + for k, v in events: + if k == "values": + self.assertEqual(v["dashboard"], initial.values["dashboard"]) + self.assert_completion(saved, "mixed") + + async def test_post_assessment_result_changes_cannot_publish(self): + self.setup_graph() + initial, _ = await self.initial() + original = self.module.wrap_spec_into_ai + async def change_result(state): + output = await original(state) + result = self.module.current_results(state)[0] + output["messages"] = [result.model_copy(deep=True, update={"content": "[]"})] + return output + from langchain_core.runnables import RunnableLambda + self.module.graph.builder.nodes["wrap_spec_into_ai"].runnable = RunnableLambda(change_result) + self.recompile() + await self.graph.aupdate_state(self.config, initial.values, as_node="generate_title") + saved, events = await self.send("changed", decision("data_update", [call("query_recent_disruptions", {"type": "cancelled"})])) + self.assertEqual(saved.values["operation_receipt"]["outcome"], "unfulfilled") + self.assertEqual(saved.values["dashboard"], initial.values["dashboard"]) + self.assertEqual(self.patches(events), []) + + async def test_backing_result_changed_before_wrapper_never_mounts_layout(self): + self.setup_graph() + original = self.module.assess_results + def changed_after_assessment(state): + output = original(state) + result = next(m for m in self.module.current_results(state) if m.name == "query_recent_disruptions") + output["messages"] = [result.model_copy(deep=True, update={"content": "[]"})] + return output + from langchain_core.runnables import RunnableLambda + self.module.graph.builder.nodes["assess_results"].runnable = RunnableLambda(changed_after_assessment) + self.recompile() + saved, events = await self.initial() + self.assertEqual(saved.values["operation_receipt"]["outcome"], "unfulfilled") + self.assertEqual(saved.values["dashboard"], {}) + self.assertIsNone(saved.values["_owned_layout"]) + self.assertEqual(self.patches(events), []) + for kind, value in events: + if kind == "values": + self.assertFalse(any(isinstance(m, AIMessage) and m.content.startswith("{") for m in value["messages"])) + self.assertFalse(any(isinstance(m, ToolMessage) and m.content == "rendered" for m in value["messages"])) + + async def test_direct_chronological_latest_results_and_slot_ownership(self): + self.setup_graph() + calls = [call("query_recent_disruptions", {"limit": 1}, "first"), + call("query_recent_disruptions", {"type": "cancelled"}, "last")] + query = next(t for t in self.module._DATA_TOOLS if t.name == "query_recent_disruptions") + results = [ToolMessage(id="result-" + c["id"], name=c["name"], tool_call_id=c["id"], + content=json.dumps(query.invoke(c["args"]))) for c in calls] + slots = self.module.latest_data_slots(results, calls) + self.assertEqual([r["flight_number"] for r in slots["recent_disruptions"]], ["AA456", "UA204", "UA640"]) + reversed_slots = self.module.latest_data_slots(list(reversed(results)), calls) + self.assertEqual(len(reversed_slots["recent_disruptions"]), 1) + slots["recent_disruptions"][0]["type"] = "changed" + self.assertEqual(json.loads(results[-1].content)[0]["type"], "cancelled") + + async def test_exact_submitted_human_snapshot_is_owned_and_enforced(self): + self.setup_graph() + human = HumanMessage(id="exact", content="Original question") + saved, _ = await self.send("exact", decision("unsupported"), messages=[human]) + self.assertEqual(saved.values["operation_receipt"].get("submitted_human"), human.model_dump()) + snapshot = deepcopy(saved.values["_submitted_human"]) + changed = deepcopy(saved.values) + changed["messages"] = changed["messages"][:-1] + changed["_decision"] = {"kind": "unsupported", "calls": [], "interpretation": ""} + self.assertEqual(self.module.admit_operation(changed)["_operation"]["kind"], "unsupported") + changed["messages"][-1].content = "Replaced question" + rejected = self.module.admit_operation(changed) + self.assertIn("Submitted human changed", rejected["_reason"]) + saved.values["operation_receipt"]["submitted_human"]["content"] = "Changed receipt" + self.assertEqual(saved.values["_submitted_human"], snapshot) + self.assertEqual((await self.graph.aget_state(self.config)).values["operation_receipt"]["submitted_human"], snapshot) + + async def test_failed_interpretation_preserves_failed_run_without_completion(self): + self.setup_graph() + initial, _ = await self.initial() + self.respond_model.fail = True + with self.assertRaisesRegex(RuntimeError, "Local title failure"): + await self.send("explain", decision("interpret", interpretation="Why?")) + saved = await self.graph.aget_state(self.config) + self.assertEqual(saved.next, ("respond",)) + self.assertIsNone(saved.values["completed_turn_id"]) + self.assertIsNone(saved.values["completed_answer_id"]) + self.assertIsNone(saved.values["completed_message_ids"]) + self.assertEqual(saved.values["dashboard"], initial.values["dashboard"]) + self.assertEqual(saved.values["_owned_layout"], initial.values["_owned_layout"]) + + async def test_compiled_valid_shape_wrong_filter_and_wrong_airline_are_unfulfilled(self): + from langchain_core.tools import StructuredTool + from langgraph.prebuilt import ToolNode + from src.dashboard_tools import RECENT_DISRUPTIONS + for name, args in [("query_recent_disruptions", {"type": "cancelled"}), + ("query_flights_by_airline", {"airlines": ["Delta"]})]: + with self.subTest(name=name): + self.setup_graph() + initial, _ = await self.initial() + def mixed(limit: int = 5, type: str | None = None) -> list: + return deepcopy(RECENT_DISRUPTIONS[:2]) + def airline(airlines: list[str] | None = None) -> list: + return [{"airline": "United", "count": 1}] + controlled = StructuredTool.from_function(mixed if name == "query_recent_disruptions" else airline, + name=name, description="Wrong selection") + self.module._tool_node = ToolNode([controlled]) + saved, events = await self.send("wrong-selection", decision("data_update", [call(name, args)])) + self.assertEqual(saved.values["operation_receipt"]["outcome"], "unfulfilled") + self.assertEqual(saved.values["dashboard"], initial.values["dashboard"]) + self.assertEqual(saved.values["_owned_layout"], initial.values["_owned_layout"]) + self.assertEqual(self.patches(events), []) + self.assertEqual(saved.values["operation_receipt"]["results"][0]["status"], "success") + + async def test_changed_submitted_human_before_wrapper_cannot_publish(self): + self.setup_graph() + original = self.module.assess_results + def changed_human(state): + output = original(state) + human = next(m for m in reversed(state["messages"]) if isinstance(m, HumanMessage)) + output["messages"] = [human.model_copy(deep=True, update={"content": "Changed intent"})] + return output + from langchain_core.runnables import RunnableLambda + self.module.graph.builder.nodes["assess_results"].runnable = RunnableLambda(changed_human) + self.recompile() + saved, events = await self.initial() + self.assertEqual(saved.values["operation_receipt"]["outcome"], "unfulfilled") + self.assertEqual(saved.values["dashboard"], {}) + self.assertIsNone(saved.values["_owned_layout"]) + self.assertIsNone(saved.values["completed_turn_id"]) + self.assertEqual(self.patches(events), []) + for kind, value in events: + if kind == "values": + self.assertFalse(any(isinstance(m, AIMessage) and m.content.startswith("{") for m in value["messages"])) + + async def test_full_initial_five_tool_batch_and_data_only_untouched_slots(self): + self.setup_graph() + marker = 'For "show me the dashboard":' + layout = json.loads(self.module._PROMPT.split(marker)[-1].strip()) + initial, events = await self.send("all-six-types", decision("create", [ + call("render_spec", layout), *[call(t.name) for t in self.module._DATA_TOOLS]])) + self.assertEqual(initial.values["operation_receipt"]["outcome"], "applied") + self.assertEqual(len(initial.values["operation_receipt"]["admitted_calls"]), 5) + self.assertEqual(len(initial.values["operation_receipt"]["results"]), 5) + self.assertEqual(initial.values["dashboard"]["on_time"], {"value": "84.2%", "delta": "+1.4%"}) + self.assertEqual(len(initial.values["dashboard"]["on_time_trend"]), 12) + self.assertEqual(len(initial.values["dashboard"]["flights_by_airline"]), 4) + self.assertEqual(len(initial.values["dashboard"]["recent_disruptions"]), 5) + self.assertEqual(self.patches(events)[0]["/on_time/value"], "84.2%") + saved, events = await self.send("six-months", decision("data_update", [call("query_on_time_trend", {"months": 6})])) + self.assertEqual(len(saved.values["dashboard"]["on_time_trend"]), 6) + for slot in initial.values["dashboard"]: + if slot != "on_time_trend": + self.assertEqual(saved.values["dashboard"][slot], initial.values["dashboard"][slot]) + self.assertEqual(saved.values["_owned_layout"], layout) + self.assertEqual(list(self.patches(events)[0]), ["/on_time_trend"]) + + +class InstalledBindingTests(unittest.TestCase): + def test_real_installed_chatopenai_required_binding(self): + # LocalModel.bind_tools intentionally ignores kwargs; this is the separate real-provider binding proof. + with patch.object(socket.socket, "connect", side_effect=AssertionError("Remote socket forbidden")) as network, patch.dict(os.environ, {"OPENAI_API_KEY": "offline"}): + spec = importlib.util.spec_from_file_location("installed_dashboard_binding", Path(__file__).parents[1] / "src/graph.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + bound = module._llm_with_tools + network.assert_not_called() + self.assertEqual(bound.kwargs["tool_choice"], {"type": "function", "function": {"name": "plan_dashboard_operation"}}) + self.assertIs(bound.kwargs["parallel_tool_calls"], False) + schema = bound.kwargs["tools"][0]["function"] + self.assertIs(schema["strict"], False) + self.assertEqual(schema["parameters"]["properties"]["kind"]["enum"], ["create", "data_update", "restructure", "interpret", "unsupported"]) + self.assertEqual(schema["parameters"]["properties"]["calls"]["type"], "array") + self.assertEqual(len(bound.kwargs["tools"]), 1) + self.assertIn("nostream", bound.bound.tags) if __name__ == "__main__": diff --git a/cockpit/chat/generative-ui/react/e2e/generative-ui.spec.ts b/cockpit/chat/generative-ui/react/e2e/generative-ui.spec.ts index 19e693bdb..261361ef8 100644 --- a/cockpit/chat/generative-ui/react/e2e/generative-ui.spec.ts +++ b/cockpit/chat/generative-ui/react/e2e/generative-ui.spec.ts @@ -42,6 +42,22 @@ test.afterEach(async ({ page, request }) => { actualCompiledGraph: true, networkConnectAttempts: 0, titleMessageCallbacks: 0, + decisionMessageCallbacks: 0, + interpretationMessageCallbacks: 0, + productionSourceSha256: Object.fromEntries( + [ + 'src/graph.py', + 'src/dashboard_tools.py', + 'src/operations.py', + 'src/dashboard_contract.py', + 'prompts/generative-ui.md', + ].map((path) => [ + path, + createHash('sha256') + .update(readFileSync(resolve(__dirname, '../../python/' + path))) + .digest('hex'), + ]) + ), sourceSha256: createHash('sha256') .update(readFileSync(resolve(__dirname, '../../python/src/graph.py'))) .digest('hex'), @@ -103,6 +119,12 @@ test('mount is inert; four confirmed turns retain ownership and share refreshed expect( await table.locator('tbody tr td:nth-child(2)').allTextContents() ).toEqual(['cancelled', 'cancelled', 'cancelled']); + expect( + await table.locator('tbody tr td:first-child').allTextContents() + ).toEqual(['AA456', 'UA204', 'UA640']); + await expect( + page.getByText('Stored 3 cancelled disruption rows (limit 5).') + ).toBeVisible(); await expect(table).toContainText('cancelled'); expect(await requests(request)).toHaveLength(9); await send(page, 'Remove the disruptions table from the structure.'); @@ -173,6 +195,102 @@ test('nonempty parent preserves prose with inspectable raw result and notice', a .first() ).toBeVisible(); expect(await requests(request)).toHaveLength(5); + const proofs = await (await request.get('/__graph-proof')).json(); + expect( + proofs.some( + (p: { op: string; fixtureControls: { prose_parent: boolean } }) => + p.op === 'submit' && p.fixtureControls.prose_parent + ) + ).toBe(true); +}); +for (const decisionFault of ['missing', 'invalid']) + test(`saved ${decisionFault} decision honestly leaves the prior board unchanged`, async ({ + page, + request, + }) => { + await page.goto(base); + await send(page, 'Show dashboard'); + const old = await boards(page).innerText(); + const owner = await boards(page).getAttribute('data-dashboard-spec-owner'); + await configure(request, { decisionFault }); + await send(page, 'Filter to cancelled flights'); + await expect(boards(page)).toHaveCount(1); + expect(await boards(page).innerText()).toBe(old); + expect(await boards(page).getAttribute('data-dashboard-spec-owner')).toBe( + owner + ); + await expect( + page.getByText( + 'No valid dashboard operation was selected. No dashboard changes were made.' + ) + ).toBeVisible(); + await expect(box(page)).toBeEnabled(); + expect( + (await requests(request)).filter((r: { path: string }) => + r.path.endsWith('/runs/stream') + ) + ).toHaveLength(2); + }); +for (const toolFault of ['failKpis', 'malformedTrend']) + test(`real ${toolFault} batch keeps successful peers inspectable but cannot replace the confirmed board`, async ({ + page, + request, + }) => { + await configure(request, { partialDashboard: true }); + await page.goto(base); + await send(page, 'Show dashboard'); + const old = await boards(page).innerText(); + const owner = await boards(page).getAttribute('data-dashboard-spec-owner'); + await configure(request, { partialDashboard: false, [toolFault]: true }); + await box(page).fill('Show another dashboard'); + await box(page).press('Enter'); + await expect(page.getByRole('alert')).toContainText( + 'could not be confirmed' + ); + await expect(boards(page)).toHaveCount(1); + expect(await boards(page).innerText()).toBe(old); + expect(await boards(page).getAttribute('data-dashboard-spec-owner')).toBe( + owner + ); + await expect(box(page)).toBeDisabled(); + await expect( + page.getByText('Start a new conversation to continue.', { exact: true }) + ).toBeVisible(); + await page.getByText('Server tool result', { exact: true }).last().click(); + await expect( + page + .locator('details[open] > pre') + .filter({ hasText: 'on_time_pct' }) + .last() + ).toBeVisible(); + expect( + (await requests(request)).filter((r: { path: string }) => + r.path.endsWith('/runs/stream') + ) + ).toHaveLength(2); + await page.getByRole('button', { name: 'New conversation' }).click(); + await expect(boards(page)).toHaveCount(0); + await expect(box(page)).toBeEnabled(); + }); +test('failed read-only interpretation retains the confirmed board and requires New', async ({ + page, + request, +}) => { + await page.goto(base); + await send(page, 'Show dashboard'); + const old = await boards(page).innerText(); + await configure(request, { failInterpretation: true }); + await box(page).fill('Why is on-time low?'); + await box(page).press('Enter'); + await expect(page.getByRole('alert')).toBeVisible(); + await expect(boards(page)).toHaveCount(1); + expect(await boards(page).innerText()).toBe(old); + await expect(box(page)).toBeDisabled(); + expect( + (await requests(request)).filter((r: { path: string }) => + r.path.endsWith('/runs/stream') + ) + ).toHaveLength(2); }); test('pending update keeps the old board; double submit, Stop and New cannot publish stale state', async ({ page, diff --git a/cockpit/chat/generative-ui/react/src/connection.integration.spec.ts b/cockpit/chat/generative-ui/react/src/connection.integration.spec.ts index 50685a9f9..444628199 100644 --- a/cockpit/chat/generative-ui/react/src/connection.integration.spec.ts +++ b/cockpit/chat/generative-ui/react/src/connection.integration.spec.ts @@ -103,6 +103,76 @@ it('actual nonempty-parent render keeps prose and raw result without mounting a const last = f.app.getSnapshot().toolCalls.at(-1); expect(last?.status === 'complete' && last.result).not.toBe('rendered'); }, 60000); +it.each(['missing', 'invalid'])( + 'actual %s decision confirms an honest saved no-op with the prior dashboard', + async (decisionFault) => { + const f = await fixture(); + expect(await f.app.submit('Show dashboard')).toBe(true); + const prior = f.app.getSnapshot(); + await f.control({ decisionFault }); + expect(await f.app.submit('Filter to cancelled flights')).toBe(true); + const saved = f.app.getSnapshot(); + expect(saved.dashboard).toEqual(prior.dashboard); + expect(saved.surfaces).toEqual(prior.surfaces); + expect(saved.toolCalls).toEqual(prior.toolCalls); + expect(saved.messages.at(-1)?.content).toContain( + 'No dashboard changes were made.' + ); + expect(saved.canSubmit).toBe(true); + expect( + (await f.requests()).filter((r) => r.path.endsWith('/runs/stream')) + ).toHaveLength(2); + }, + 60000 +); +it.each(['failKpis', 'malformedTrend'])( + 'actual %s structural rollback stays unconfirmed with real successful peers', + async (toolFault) => { + const f = await fixture(); + await f.control({ partialDashboard: true }); + expect(await f.app.submit('Show dashboard')).toBe(true); + const prior = f.app.getSnapshot(); + expect(Object.keys(prior.dashboard).sort()).toEqual([ + 'flights_by_airline', + 'recent_disruptions', + ]); + await f.control({ partialDashboard: false, [toolFault]: true }); + expect(await f.app.submit('Show another dashboard')).toBe(false); + const failed = f.app.getSnapshot(); + expect(failed.dashboard).toBe(prior.dashboard); + expect(failed.surfaces).toBe(prior.surfaces); + expect(failed.canSubmit).toBe(false); + const render = failed.toolCalls + .filter((c) => c.name === 'render_spec') + .at(-1); + expect(render?.status === 'complete' && render.result).not.toBe('rendered'); + const peer = failed.toolCalls.find((c) => c.name === 'query_on_time_trend'); + expect(peer?.status).toBe('complete'); + const count = (await f.requests()).length; + expect(await f.app.submit('Try again')).toBe(false); + expect(await f.requests()).toHaveLength(count); + await f.app.newConversation(); + expect(f.app.getSnapshot()).toMatchObject({ + canSubmit: true, + surfaces: [], + dashboard: {}, + }); + }, + 60000 +); +it('actual failed read-only interpretation retains the confirmed dashboard and requires New', async () => { + const f = await fixture(); + expect(await f.app.submit('Show dashboard')).toBe(true); + const prior = f.app.getSnapshot(); + await f.control({ failInterpretation: true }); + expect(await f.app.submit('Why is on-time low?')).toBe(false); + expect(f.app.getSnapshot().dashboard).toBe(prior.dashboard); + expect(f.app.getSnapshot().surfaces).toBe(prior.surfaces); + expect(f.app.getSnapshot().canSubmit).toBe(false); + expect( + (await f.requests()).filter((r) => r.path.endsWith('/runs/stream')) + ).toHaveLength(2); +}, 60000); it('actual pending task rejects confirmation and Stop preserves the confirmed board without retry', async () => { const f = await fixture(); expect(await f.app.submit('Show dashboard')).toBe(true); diff --git a/libs/cockpit-registry/src/lib/content-descriptors.ts b/libs/cockpit-registry/src/lib/content-descriptors.ts index f851ab9e8..0d127271e 100644 --- a/libs/cockpit-registry/src/lib/content-descriptors.ts +++ b/libs/cockpit-registry/src/lib/content-descriptors.ts @@ -830,7 +830,12 @@ const capabilityModuleData: RegisteredCapabilityModule[] = [ 'cockpit/chat/generative-ui/angular/src/app/generative-ui.component.ts', 'cockpit/chat/generative-ui/angular/src/app/app.config.ts', ], - backendAssetPaths: ['cockpit/chat/generative-ui/python/src/graph.py'], + backendAssetPaths: [ + 'cockpit/chat/generative-ui/python/src/graph.py', + 'cockpit/chat/generative-ui/python/src/dashboard_tools.py', + 'cockpit/chat/generative-ui/python/src/operations.py', + 'cockpit/chat/generative-ui/python/src/dashboard_contract.py', + ], runtimeUrl: 'chat/generative-ui', devPort: 4508, }, diff --git a/libs/cockpit-registry/src/lib/frontend-pilot.spec.ts b/libs/cockpit-registry/src/lib/frontend-pilot.spec.ts index ca3853d5d..38d1dcb63 100644 --- a/libs/cockpit-registry/src/lib/frontend-pilot.spec.ts +++ b/libs/cockpit-registry/src/lib/frontend-pilot.spec.ts @@ -19,6 +19,8 @@ it('resolves native Chat Generative UI with its canonical docs and existing back if(!canonical) throw Error('Missing canonical Chat Generative UI'); const react=getFrontendCapabilityDescriptor(canonical.manifestIdentity as CockpitManifestIdentity,'react'); expect(react).toMatchObject({id:'chat-generative-ui-python-react',frontend:'react',runtimeAdapter:'langgraph',runtimeUrl:'chat/generative-ui/react',devPort:4627,docsPath:'/docs/chat/guides/generative-ui',backendAssetPaths:canonical.backendAssetPaths,promptAssetPaths:canonical.promptAssetPaths}); + expect(canonical.backendAssetPaths).toEqual(['graph.py', 'dashboard_tools.py', 'operations.py', 'dashboard_contract.py'].map(file => `cockpit/chat/generative-ui/python/src/${file}`)); + for (const asset of canonical.backendAssetPaths) expect(existsSync(new URL('../../../../' + asset, import.meta.url))).toBe(true); expect(react?.codeAssetPaths).toEqual(['app.tsx','application.ts','connection.ts','dashboard-views.tsx','dashboard-data.ts','dashboard-spec.ts','tool-evidence.ts','observation.ts','terminal.ts','main.tsx','styles.css'].map(file=>`cockpit/chat/generative-ui/react/src/${file}`)); for(const asset of react?.codeAssetPaths ?? [])expect(existsSync(new URL('../../../../'+asset,import.meta.url))).toBe(true); expect(getCockpitFrontends().find(entry=>entry.project==='cockpit-chat-generative-ui-react')).toMatchObject({port:4627,runtimePath:'chat/generative-ui/react'}); diff --git a/scripts/react-cockpit/chat-generative-ui-fixture.mjs b/scripts/react-cockpit/chat-generative-ui-fixture.mjs index d1d9b8286..c675b3dad 100644 --- a/scripts/react-cockpit/chat-generative-ui-fixture.mjs +++ b/scripts/react-cockpit/chat-generative-ui-fixture.mjs @@ -123,6 +123,27 @@ export function createChatGenerativeUiFixture({ .digest('hex'), ]) ); + const productionSourceSha256 = Object.fromEntries( + [ + 'src/graph.py', + 'src/dashboard_tools.py', + 'src/operations.py', + 'src/dashboard_contract.py', + 'prompts/generative-ui.md', + ].map((path) => [ + path, + createHash('sha256') + .update( + readFileSync( + new URL( + '../../cockpit/chat/generative-ui/python/' + path, + import.meta.url + ) + ) + ) + .digest('hex'), + ]) + ); let worker, closed = false, closing, @@ -164,10 +185,17 @@ export function createChatGenerativeUiFixture({ !result.proof?.actualCompiledGraph || result.proof.networkConnectAttempts !== 0 || result.proof.titleMessageCallbacks !== 0 || + result.proof.decisionMessageCallbacks !== 0 || + result.proof.interpretationMessageCallbacks !== 0 || result.proof.op !== value.op || result.proof.threadId !== (value.threadId ?? null) || Object.entries(expected).some( ([key, hash]) => result.proof[key] !== hash + ) || + Object.keys(result.proof.productionSourceSha256 ?? {}).length !== + Object.keys(productionSourceSha256).length || + Object.entries(productionSourceSha256).some( + ([path, hash]) => result.proof.productionSourceSha256?.[path] !== hash ) ) throw Error('Invalid graph proof'); @@ -262,11 +290,32 @@ export function createChatGenerativeUiFixture({ } if (pathname === '/__configure') { Object.assign(options, body); - if ('failTitle' in body || 'failAgent' in body) + if ( + [ + 'failTitle', + 'failAgent', + 'failInterpretation', + 'failMetadata', + 'failWrite', + 'mapped', + 'decisionFault', + 'partialDashboard', + 'failKpis', + 'malformedTrend', + ].some((key) => key in body) + ) await operate({ op: 'configure', failTitle: options.failTitle ?? false, failAgent: options.failAgent ?? false, + failInterpretation: options.failInterpretation ?? false, + failMetadata: options.failMetadata ?? false, + failWrite: options.failWrite ?? false, + mapped: options.mapped ?? false, + decisionFault: options.decisionFault ?? null, + partialDashboard: options.partialDashboard ?? false, + failKpis: options.failKpis ?? false, + malformedTrend: options.malformedTrend ?? false, }); json(response, {}); return true; diff --git a/scripts/react-cockpit/chat-generative-ui-fixture.spec.mjs b/scripts/react-cockpit/chat-generative-ui-fixture.spec.mjs index 366104bb7..7e4f8d771 100644 --- a/scripts/react-cockpit/chat-generative-ui-fixture.spec.mjs +++ b/scripts/react-cockpit/chat-generative-ui-fixture.spec.mjs @@ -6,17 +6,20 @@ import { writeFile, mkdtemp, rm } from 'node:fs/promises'; import { test } from 'node:test'; import { Client } from '@langchain/langgraph-sdk'; import { build } from 'esbuild'; +import { createHash } from 'node:crypto'; +import { readFileSync } from 'node:fs'; +import { spawn } from 'node:child_process'; const { createChatGenerativeUiFixture } = await import( './chat-generative-ui-fixture.mjs' ).catch(() => ({})); -async function fixture(t) { +async function fixture(t, options) { assert.equal( typeof createChatGenerativeUiFixture, 'function', 'Generative UI fixture exists' ); - const handle = createChatGenerativeUiFixture(); + const handle = createChatGenerativeUiFixture(options); const server = createServer(async (req, res) => { if ( !(await handle(req, res, new URL(req.url, 'http://localhost').pathname)) @@ -71,6 +74,241 @@ async function fixture(t) { return { base, client, thread, submit, control }; } +const productionSources = [ + 'src/graph.py', + 'src/dashboard_tools.py', + 'src/operations.py', + 'src/dashboard_contract.py', + 'prompts/generative-ui.md', +]; +function assertReceipt(saved, id) { + const receipt = saved.values.operation_receipt; + assert.equal(receipt.submitted_human_id, id); + assert.deepEqual( + receipt.submitted_human, + saved.values.messages.find((m) => m.id === id) + ); + assert.equal(saved.values.completed_turn_id, id); + assert.deepEqual( + saved.values.completed_message_ids, + saved.values.messages.map((m) => m.id) + ); + assert.equal( + saved.values.completed_answer_id, + saved.values.messages.at(-1).id + ); + const suffix = saved.values.messages.slice( + saved.values.messages.findIndex((m) => m.id === id) + 1 + ); + assert.deepEqual( + receipt.admitted_calls, + suffix.flatMap((m) => m.tool_calls ?? []) + ); + assert.deepEqual( + receipt.results, + suffix + .filter((m) => m.type === 'tool') + .map(({ id, tool_call_id, name, status, content }) => ({ + id, + tool_call_id, + name, + status, + content, + })) + ); + return receipt; +} +async function assertProofs(f) { + const proofs = await (await fetch(f.base + '/__graph-proof')).json(); + assert(proofs.length > 0); + for (const proof of proofs) { + assert.equal(proof.networkConnectAttempts, 0); + assert.equal(proof.titleMessageCallbacks, 0); + assert.equal(proof.decisionMessageCallbacks, 0); + assert.equal(proof.interpretationMessageCallbacks, 0); + assert.deepEqual( + proof.productionSourceSha256, + Object.fromEntries( + productionSources.map((path) => [ + path, + createHash('sha256') + .update( + readFileSync( + new URL( + '../../cockpit/chat/generative-ui/python/' + path, + import.meta.url + ) + ) + ) + .digest('hex'), + ]) + ) + ); + } +} + +for (const decisionFault of ['missing', 'invalid']) + test( + `actual SDK saves honest ${decisionFault} decision with zero dashboard calls`, + { timeout: 60000 }, + async (t) => { + const f = await fixture(t); + await f.submit('Show dashboard'); + const prior = await f.client.threads.getState(f.thread); + assert.equal(prior.values.operation_receipt.outcome, 'applied'); + assert.equal(Object.keys(prior.values.dashboard).length, 7); + await f.control({ decisionFault }); + const turn = await f.submit(' Filter to cancelled flights '); + const saved = await f.client.threads.getState(f.thread); + const receipt = assertReceipt(saved, turn.id); + assert.equal(receipt.outcome, 'rejected'); + assert.deepEqual(receipt.admitted_calls, []); + assert.deepEqual(receipt.results, []); + assert.deepEqual(saved.values.dashboard, prior.values.dashboard); + assert.deepEqual(saved.values._owned_layout, prior.values._owned_layout); + assert.deepEqual(saved.values._render_owner, prior.values._render_owner); + assert.deepEqual( + saved.values.messages + .slice(prior.values.messages.length) + .flatMap((m) => m.tool_calls ?? []), + [] + ); + assert(!turn.events.some((e) => e.event === 'custom')); + assert.match( + saved.values.messages.at(-1).content, + /No dashboard changes were made/ + ); + assert.equal( + (await f.client.runs.get(f.thread, turn.run)).status, + 'success' + ); + assert.deepEqual( + (await f.client.threads.getHistory(f.thread, { limit: 100 }))[0], + saved + ); + assert.deepEqual( + await f.client.threads.getState(f.thread, prior.checkpoint), + prior + ); + await assertProofs(f); + } + ); + +for (const toolFault of ['failKpis', 'malformedTrend']) + test( + `actual SDK ${toolFault} structural batch rolls back all publication while retaining real peer results`, + { timeout: 60000 }, + async (t) => { + const f = await fixture(t); + await f.control({ partialDashboard: true, mapped: true }); + await f.submit('Show dashboard'); + const prior = await f.client.threads.getState(f.thread); + assert.deepEqual(Object.keys(prior.values.dashboard).sort(), [ + 'flights_by_airline', + 'recent_disruptions', + ]); + await f.control({ partialDashboard: false, [toolFault]: true }); + const turn = await f.submit('Show another dashboard'); + const saved = await f.client.threads.getState(f.thread); + const receipt = assertReceipt(saved, turn.id); + assert.equal(receipt.kind, 'restructure'); + assert.equal(receipt.outcome, 'unfulfilled'); + assert.deepEqual( + receipt.admitted_calls.map((c) => c.name), + ['render_spec', 'query_airline_kpis', 'query_on_time_trend'] + ); + assert.deepEqual(receipt.validated_slots, {}); + assert.equal(receipt.render_owner, null); + assert.equal( + receipt.results.find((r) => r.name === 'query_airline_kpis').status, + toolFault === 'failKpis' ? 'error' : 'success' + ); + const trend = receipt.results.find( + (r) => r.name === 'query_on_time_trend' + ); + assert.equal(trend.status, 'success'); + assert.equal( + JSON.parse(trend.content).length, + toolFault === 'failKpis' ? 12 : 1 + ); + for (const result of receipt.results) + assert.equal( + result.tool_call_id, + receipt.admitted_calls.find((c) => c.name === result.name).id + ); + assert.notEqual( + receipt.results.find((r) => r.name === 'render_spec').content, + 'rendered' + ); + assert(!turn.events.some((e) => e.event === 'custom')); + assert.deepEqual(saved.values.dashboard, prior.values.dashboard); + assert.deepEqual(saved.values._owned_layout, prior.values._owned_layout); + assert.deepEqual(saved.values._render_owner, prior.values._render_owner); + assert.match( + saved.values.messages.at(-1).content, + /No dashboard changes were made/ + ); + assert.deepEqual(saved.checkpoint.checkpoint_map, { + '': saved.checkpoint.checkpoint_id, + }); + assert.deepEqual( + await f.client.threads.getState(f.thread, saved.checkpoint), + saved + ); + await assertProofs(f); + } + ); + +for (const field of [ + ...productionSources, + 'titleMessageCallbacks', + 'decisionMessageCallbacks', + 'interpretationMessageCallbacks', +]) + test(`fixture receiver refuses altered ${field} evidence on every RPC`, async (t) => { + const hash = (path) => + createHash('sha256') + .update( + readFileSync( + new URL( + '../../cockpit/chat/generative-ui/python/' + path, + import.meta.url + ) + ) + ) + .digest('hex'); + const proof = { + actualCompiledGraph: true, + networkConnectAttempts: 0, + titleMessageCallbacks: 0, + decisionMessageCallbacks: 0, + interpretationMessageCallbacks: 0, + sourceSha256: hash('src/graph.py'), + lockSha256: hash('uv.lock'), + productionSourceSha256: Object.fromEntries( + productionSources.map((path) => [path, hash(path)]) + ), + }; + if (productionSources.includes(field)) + proof.productionSourceSha256[field] = '0'.repeat(64); + else proof[field] = 1; + const f = await fixture(t, { + spawnWorker: () => + spawn( + process.execPath, + [ + '-e', + `const p=${JSON.stringify( + proof + )};require('node:readline').createInterface({input:process.stdin}).on('line',line=>{const q=JSON.parse(line);process.stdout.write(JSON.stringify({requestId:q.requestId,state:{},proof:{...p,op:q.op,threadId:q.threadId??null}})+'\\n')})`, + ], + { stdio: ['pipe', 'pipe', 'pipe'] } + ), + }); + await assert.rejects(f.client.threads.getState(f.thread)); + assert.deepEqual(await (await fetch(f.base + '/__graph-proof')).json(), []); + }); + test( 'installed SDK consumes actual graph initial, filter, structural and prose turns', { timeout: 60000 }, @@ -121,6 +359,14 @@ test( ]) ); let saved = await f.client.threads.getState(f.thread); + const initialReceipt = assertReceipt(saved, initial.id); + assert.deepEqual(initialReceipt.validated_slots, saved.values.dashboard); + assert.equal(initialReceipt.admitted_calls.length, 5); + assert.deepEqual(initialReceipt.render_owner, { + parent_id: parent.id, + call_id: raw.tool_call_id, + result_id: raw.id, + }); assert.equal(Object.keys(saved.values.dashboard).length, 7); assert.equal(saved.values.dashboard.on_time.value, '84.2%'); assert.equal(saved.values.completed_turn_id, initial.id); @@ -138,6 +384,33 @@ test( const original = structuredClone(saved); const filtered = await f.submit('Filter to cancelled flights only'); saved = await f.client.threads.getState(f.thread); + const filterReceipt = assertReceipt(saved, filtered.id); + assert.deepEqual(filterReceipt.validated_slots, { + recent_disruptions: saved.values.dashboard.recent_disruptions, + }); + assert.equal(filterReceipt.render_owner, null); + assert.deepEqual(saved.values._owned_layout, original.values._owned_layout); + assert.deepEqual(saved.values._render_owner, original.values._render_owner); + assert.deepEqual( + saved.values.dashboard.recent_disruptions.map((r) => r.flight_number), + ['AA456', 'UA204', 'UA640'] + ); + assert.deepEqual( + filtered.events.filter((e) => e.event === 'custom').map((e) => e.data), + [ + { + name: 'state_update', + data: { + '/recent_disruptions': + filterReceipt.validated_slots.recent_disruptions, + }, + }, + ] + ); + assert.equal( + saved.values.messages.at(-1).content, + 'Stored 3 cancelled disruption rows (limit 5).' + ); assert.deepEqual( saved.values.messages.slice(0, original.values.messages.length), original.values.messages @@ -386,6 +659,11 @@ test( m.tool_calls?.some((c) => c.name === 'render_spec') ); assert.equal(parent.content, 'Here is the layout.'); + assert.equal(saved.values.operation_receipt.outcome, 'unfulfilled'); + assert.deepEqual(saved.values.operation_receipt.validated_slots, {}); + assert.deepEqual(saved.values.dashboard, {}); + assert.equal(saved.values._owned_layout, null); + assert.equal(saved.values.operation_receipt.admitted_calls.length, 1); assert.equal( JSON.parse( saved.values.messages.find((m) => m.name === 'render_spec').content @@ -401,6 +679,12 @@ test( p.titleMessageCallbacks === 0 ) ); + assert( + proofs.some( + (p) => p.op === 'submit' && p.fixtureControls.prose_parent === true + ) + ); + await assertProofs(f); assert( proofs.every( (p) => @@ -492,3 +776,36 @@ test( assert.deepEqual(history[0].tasks, saved.tasks); } ); + +test( + 'actual interpretation provider failure clears completion after emit and saves a task error', + { timeout: 60000 }, + async (t) => { + const f = await fixture(t); + await f.submit('Show dashboard'); + const prior = await f.client.threads.getState(f.thread); + await f.control({ failInterpretation: true }); + const turn = await f.submit('Why is on-time low?'); + const saved = await f.client.threads.getState(f.thread); + assert.equal((await f.client.runs.get(f.thread, turn.run)).status, 'error'); + assert( + saved.tasks.some( + (task) => + task.name === 'respond' && task.error.includes('Local title failure') + ) + ); + assert.equal(saved.values.completed_turn_id, null); + assert.equal(saved.values.completed_answer_id, null); + assert.equal(saved.values.completed_message_ids, null); + assert.equal(saved.values.operation_receipt.outcome, 'interpret'); + assert.deepEqual(saved.values.dashboard, prior.values.dashboard); + assert.deepEqual(saved.values._owned_layout, prior.values._owned_layout); + assert.deepEqual(saved.values._render_owner, prior.values._render_owner); + assert(!turn.events.some((e) => e.event === 'custom')); + assert.deepEqual( + (await f.client.threads.getHistory(f.thread, { limit: 100 }))[0], + saved + ); + await assertProofs(f); + } +); diff --git a/scripts/react-cockpit/chat-generative-ui-wire.py b/scripts/react-cockpit/chat-generative-ui-wire.py index a648b8864..7f5a60f72 100644 --- a/scripts/react-cockpit/chat-generative-ui-wire.py +++ b/scripts/react-cockpit/chat-generative-ui-wire.py @@ -20,6 +20,12 @@ from test_streaming import LocalModel, call from langchain_core.messages import AIMessage from copy import deepcopy +from langchain_core.runnables import RunnableLambda +from langchain_core.tools import StructuredTool +from langgraph.prebuilt import ToolNode +from src.dashboard_contract import SLOT_TO_TOOL, bindings, resolve + +PRODUCTION_SOURCES = ("src/graph.py", "src/dashboard_tools.py", "src/operations.py", "src/dashboard_contract.py", "prompts/generative-ui.md") class Metadata: def __init__(self): @@ -70,12 +76,17 @@ def checkpoint(state, mapped=False): async def main(): client = AuthoredClient() - options = {"fail_title": False, "mapped": False} + options = {"fail_title": False, "mapped": False, "decisionFault": None, + "partialDashboard": False, "failKpis": False, "malformedTrend": False, + "prose_parent": False} + callbacks = {"titleMessageCallbacks": 0, "decisionMessageCallbacks": 0, "interpretationMessageCallbacks": 0} agent_model, respond_model = LocalModel(), LocalModel() layout = json.loads((PROJECT / "prompts/generative-ui.md").read_text().split('For "show me the dashboard":')[1].strip()) def model_factory(**kwargs): if kwargs.get("streaming"): - return agent_model if kwargs["model"] == "gpt-5" else respond_model + model = agent_model if kwargs["model"] == "gpt-5" else respond_model + model.tags = kwargs.get("tags", []) + return model return LocalModel(responses=[AIMessage(id="title", content="Local dashboard title")], tags=kwargs.get("tags", []), fail=options["fail_title"]) source = PROJECT / "src/graph.py" with patch.object(socket.socket, "connect", side_effect=AssertionError("Remote socket forbidden")) as connect, patch( @@ -85,6 +96,14 @@ def model_factory(**kwargs): spec = importlib.util.spec_from_file_location("authored_generative_ui_wire", source) module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) + # Explicit test-only node intervention proves the production wrapper's + # nonempty-parent fallback. Normal decisions never supply parent prose. + def fixture_admit(state): + output = module.admit_operation(state) + if options["prose_parent"] and output.get("messages"): + output["messages"][0] = output["messages"][0].model_copy(update={"content": "Here is the layout."}) + return output + module.graph.builder.nodes["admit"].runnable = RunnableLambda(fixture_admit) graph = module.graph.builder.compile(checkpointer=InMemorySaver()) for line in sys.stdin: request = json.loads(line) @@ -103,33 +122,73 @@ def model_factory(**kwargs): client.threads.fail_write = request.get("failWrite", False) options["fail_title"] = request.get("failTitle", False) options["mapped"] = request.get("mapped", options["mapped"]) + for key in ("decisionFault", "partialDashboard", "failKpis", "malformedTrend"): + options[key] = request.get(key, options[key]) + respond_model.fail = request.get("failInterpretation", respond_model.fail) result = {} elif request["op"] == "submit": identity = request["messages"][0]["id"] text = request["messages"][0]["content"].lower() + prior = graph.get_state(config).values + accepted = prior.get("_owned_layout") spec_layout = deepcopy(layout) + options["prose_parent"] = "prose render" in text + if options["prose_parent"] and not accepted: + for element in spec_layout["elements"].values(): + for key, value in element.get("props", {}).items(): + if isinstance(value, dict) and "$state" in value: + element["props"][key] = None + elif options["partialDashboard"] and not accepted: + for element in spec_layout["elements"].values(): + for key, value in element.get("props", {}).items(): + if isinstance(value, dict) and value.get("$state", "").split("/")[1] in ("on_time", "flights_today", "avg_delay", "load_factor", "on_time_trend"): + element["props"][key] = None if "cancelled" in text: - batch = [call("query_recent_disruptions", {"type": "cancelled"}, identity + "-filter")] + kind, batch = "data_update", [call("query_recent_disruptions", {"type": "cancelled"})] elif "remove" in text or "structure" in text: - spec_layout["elements"]["root"]["children"].remove("table_section") - del spec_layout["elements"]["table_section"] - batch = [call("render_spec", spec_layout, identity + "-render")] + spec_layout = deepcopy(accepted or spec_layout) + if "table_section" in spec_layout["elements"]: + spec_layout["elements"]["root"]["children"].remove("table_section") + del spec_layout["elements"]["table_section"] + kind, batch = "restructure" if accepted else "create", [call("render_spec", spec_layout)] elif "why" in text or "interpret" in text: - batch = [] + kind, batch = "interpret", [] else: - batch = [call("render_spec", spec_layout, identity + "-render")] - batch += [call(tool.name, identity=identity + "-" + tool.name) for tool in module._DATA_TOOLS] - agent_model.responses = ([AIMessage(id=identity + "-agent", content="Here is the layout." if "prose render" in text else "", tool_calls=batch)] if batch else []) + [AIMessage(id=identity + "-done", content="Ready.")] + kind, batch = "restructure" if accepted else "create", [call("render_spec", spec_layout)] + if batch and batch[0]["name"] == "render_spec": + prior_data = prior.get("dashboard", {}) if accepted else {} + necessary = {SLOT_TO_TOOL[pointer.split("/")[1]] for pointer in bindings(spec_layout) if not resolve(pointer, prior_data)} + batch += [call(tool.name) for tool in module._DATA_TOOLS if tool.name in necessary] + selected = call("plan_dashboard_operation", {"kind": kind, "calls": [{"name": c["name"], "args": c["args"]} for c in batch], "interpretation": text if kind == "interpret" else ""}) + if options["decisionFault"] == "invalid": + selected["args"]["calls"] = [{"name": "query_recent_disruptions", "args": {"type": "invented"}}] + agent_model.responses = [AIMessage(id=identity + "-decision", content="", tool_calls=[] if options["decisionFault"] == "missing" else [selected])] agent_model.cursor = 0 - respond_model.responses = [AIMessage(id=identity + "-answer", content="Dashboard updated." if batch else "On-time performance needs operational context to explain.")] + respond_model.responses = [AIMessage(id=identity + "-interpretation", content="On-time performance needs operational context to explain.")] respond_model.cursor = 0 + def failed_kpis(): + raise RuntimeError("Explicit local KPI tool failure") + def malformed_trend(months=12): + return [{"month": "Invalid", "on_time_pct": 101}] + tools = [module.render_spec, *module._DATA_TOOLS] + if options["failKpis"]: + tools = [StructuredTool.from_function(failed_kpis, name="query_airline_kpis", description="Test-only throwing KPI tool") if tool.name == "query_airline_kpis" else tool for tool in tools] + if options["malformedTrend"]: + tools = [StructuredTool.from_function(malformed_trend, name="query_on_time_trend", description="Test-only malformed trend tool") if tool.name == "query_on_time_trend" else tool for tool in tools] + module._tool_node = ToolNode(tools, handle_tool_errors="The dashboard tool failed.") events = [] terminal_config = config failure = False try: async for kind, event in graph.astream({"messages": request["messages"]}, config, stream_mode=["messages", "values", "updates", "custom", "checkpoints"]): - if kind == "messages" and event[1].get("langgraph_node") == "generate_title": - raise AssertionError("Title callbacks leaked into answer stream") + if kind == "messages": + node, message = event[1].get("langgraph_node"), event[0] + key = ("titleMessageCallbacks" if node == "generate_title" else + "decisionMessageCallbacks" if node == "decide" else + "interpretationMessageCallbacks" if message.id == identity + "-interpretation" else None) + if key: + callbacks[key] += 1 + raise AssertionError("Private model callbacks leaked into answer stream") if kind == "checkpoints": if options["mapped"]: position = event["config"]["configurable"] @@ -148,7 +207,9 @@ def model_factory(**kwargs): "op": request["op"], "threadId": request.get("threadId"), "sourceSha256": hashlib.sha256(source.read_bytes()).hexdigest(), "lockSha256": hashlib.sha256((PROJECT / "uv.lock").read_bytes()).hexdigest(), - "networkConnectAttempts": connect.call_count, "titleMessageCallbacks": 0, + "productionSourceSha256": {path: hashlib.sha256((PROJECT / path).read_bytes()).hexdigest() for path in PRODUCTION_SOURCES}, + "networkConnectAttempts": connect.call_count, **callbacks, + "fixtureControls": {key: options[key] for key in ("decisionFault", "partialDashboard", "failKpis", "malformedTrend", "prose_parent")}, "httpCheckpointRepresentation": "root-map" if options["mapped"] else "native", } result["requestId"] = request["requestId"] diff --git a/scripts/react-parity/baseline.json b/scripts/react-parity/baseline.json index 57a929de3..f05cce68e 100644 --- a/scripts/react-parity/baseline.json +++ b/scripts/react-parity/baseline.json @@ -1,10 +1,9 @@ { "schemaVersion": 1, - "baselineHead": "cfc4811a5331e1e4fde8115938641cafb9263ef5", + "baselineHead": "b5ba906aa73768003dd1697c435a96b67c010d88", "sourceState": { "modified": [ "apps/website/content/docs/chat/guides/generative-ui.mdx", - "libs/cockpit-registry/src/lib/capability-registry.ts", "libs/cockpit-registry/src/lib/content-descriptors.ts" ], "untracked": [] @@ -2626,7 +2625,7 @@ "id": "doc:apps/website/content/docs/chat/guides/generative-ui.mdx", "kind": "doc", "path": "apps/website/content/docs/chat/guides/generative-ui.mdx", - "sha256": "0a806da93db7eb535b25b35193c7ad136d0696fb9acca5754f5a4d7516262631" + "sha256": "04da58bebc5e16e46f00418f09e924071ba43861ff0a33bd083e82373a983f59" }, { "id": "doc:apps/website/content/docs/chat/guides/layout-modes.mdx", @@ -13471,7 +13470,7 @@ "id": "source:libs/cockpit-registry/src/lib/content-descriptors.ts", "kind": "source", "path": "libs/cockpit-registry/src/lib/content-descriptors.ts", - "sha256": "96c22b1ddb34bcadb69810e18803f41301b857548da8370e131709359a0482ba" + "sha256": "cc4bbfd31fc7d1bdec187b37d2007557aa3a2de1af6a4f55eea29732b2ef4b84" }, { "id": "source:libs/cockpit-registry/src/lib/docs-links.ts", diff --git a/scripts/react-parity/verify-middleware-package.mjs b/scripts/react-parity/verify-middleware-package.mjs index 9d65a764a..d3370356d 100644 --- a/scripts/react-parity/verify-middleware-package.mjs +++ b/scripts/react-parity/verify-middleware-package.mjs @@ -1,4 +1,5 @@ import { spawnSync } from 'node:child_process'; +import assert from 'node:assert/strict'; import { cpSync, existsSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, realpathSync, rmSync, writeFileSync } from 'node:fs'; import { isBuiltin } from 'node:module'; import { tmpdir } from 'node:os'; @@ -6,6 +7,7 @@ import { dirname, join, relative, resolve, sep } from 'node:path'; import { fileURLToPath } from 'node:url'; import { satisfies } from 'semver'; import ts from 'typescript'; +import { lockedVendorGraph, vendorOverrides } from './langgraph-candidate-package.mjs'; export function runMiddlewareConsumer(command, args, cwd) { const result = spawnSync(command, args, { @@ -27,7 +29,27 @@ export function lockedMiddlewareManifest(manifest, lock) { if (!satisfies(pinned, range)) throw new Error(`${name}@${pinned} does not satisfy middleware peer range ${range}`); return [name, pinned]; })); - return { private: true, type: 'module', dependencies, devDependencies: Object.fromEntries(['typescript', '@types/node'].map((name) => [name, version(name)])) }; + const seeds = Object.keys(dependencies); + const vendors = lockedVendorGraph(lock, seeds); + const overrides = vendorOverrides(vendors, seeds); + // A qualified nested selector can miss a newly published compatible version. + // Pin every unique provider globally, retaining owner overrides for duplicates. + for (const vendor of vendors) { + const versions = new Set(vendors.filter(({ name }) => name === vendor.name).map(({ version }) => version)); + if (versions.size !== 1) continue; + const key = `${vendor.name}@${vendor.version}`; + overrides[vendor.name] = vendorOverrides(vendors, [key])[key]; + } + return { private: true, type: 'module', dependencies, overrides, devDependencies: Object.fromEntries(['typescript', '@types/node'].map((name) => [name, version(name)])) }; +} + +export function assertMiddlewareVendorGraph(manifest, rootLock, installedLock) { + const seeds = Object.keys(manifest.peerDependencies ?? {}); + assert.deepEqual( + lockedVendorGraph(installedLock, seeds), + lockedVendorGraph(rootLock, seeds), + 'Installed middleware provider edges preserve the reviewed locked graph' + ); } export function assertConsumerFiles(consumer, files) { @@ -100,7 +122,8 @@ export async function verifyMiddlewarePackage(root = process.cwd()) { const errors = auditMiddlewarePackage(artifact); if (errors.length) throw new Error(errors.join('\n')); const packageManifest = JSON.parse(readFileSync(join(artifact, 'package.json'), 'utf8')); - const manifest = lockedMiddlewareManifest(packageManifest, JSON.parse(readFileSync(join(root, 'package-lock.json'), 'utf8'))); + const rootLock = JSON.parse(readFileSync(join(root, 'package-lock.json'), 'utf8')); + const manifest = lockedMiddlewareManifest(packageManifest, rootLock); manifest.dependencies['@threadplane/middleware'] = `file:${tarball}`; const consumer = join(temporary, 'consumer'); cpSync(join(root, 'fixtures/react-parity/consumers/middleware'), consumer, { recursive: true }); @@ -108,6 +131,7 @@ export async function verifyMiddlewarePackage(root = process.cwd()) { console.log(`Middleware consumer toolchain: ${JSON.stringify(manifest)}`); console.log(runMiddlewareConsumer('npm', ['install', '--ignore-scripts', '--no-audit', '--no-fund'], consumer)); const installedLock = JSON.parse(readFileSync(join(consumer, 'package-lock.json'), 'utf8')); + assertMiddlewareVendorGraph(packageManifest, rootLock, installedLock); for (const [path, value] of Object.entries(installedLock.packages)) { if (!path) continue; const name = path.split('node_modules/').at(-1); diff --git a/scripts/react-parity/verify-middleware-package.spec.mjs b/scripts/react-parity/verify-middleware-package.spec.mjs index 4d1a29474..da1ad4a65 100644 --- a/scripts/react-parity/verify-middleware-package.spec.mjs +++ b/scripts/react-parity/verify-middleware-package.spec.mjs @@ -8,6 +8,88 @@ import * as verifier from './verify-middleware-package.mjs'; const manifest = () => ({ name: '@threadplane/middleware', type: 'module', exports: { './langgraph': { types: './index.d.ts', default: './index.js' } }, peerDependencies: { '@langchain/core': '^1.0.0', '@langchain/langgraph': '^1.0.0' } }); const lock = { packages: Object.fromEntries(Object.entries({ '@langchain/core': '1.2.9', '@langchain/langgraph': '1.4.13', typescript: '5.9.3', '@types/node': '25.6.0' }).map(([name, version]) => [`node_modules/${name}`, { version }])) }; +function providerLock() { + const result = structuredClone(lock); + result.packages['node_modules/@langchain/core'].dependencies = { langsmith: '>=0.5.0 <1.0.0' }; + result.packages['node_modules/@langchain/langgraph'].dependencies = { '@langchain/core': '^1.0.0', '@langchain/langgraph-sdk': '^1.0.0' }; + result.packages['node_modules/@langchain/langgraph-sdk'] = { version: '1.2.0', dependencies: { langsmith: '^0.5.0' } }; + result.packages['node_modules/langsmith'] = { version: '0.5.10', dependencies: { 'uuid': '^10.0.0' } }; + result.packages['node_modules/uuid'] = { version: '10.0.0' }; + return result; +} + +test('pins the full provider closure, including broad LangSmith transitive ranges', () => { + const result = verifier.lockedMiddlewareManifest(manifest(), providerLock()); + assert.ok(result.overrides, 'The isolated consumer must pin provider transitives'); + assert.deepEqual(result.overrides['@langchain/core@1.2.9'], { + '.': '1.2.9', 'langsmith@0.5.10': { '.': '0.5.10', 'uuid@10.0.0': '10.0.0' }, + }); + assert.deepEqual(result.overrides['@langchain/langgraph@1.4.13']['@langchain/langgraph-sdk@1.2.0'], { + '.': '1.2.0', 'langsmith@0.5.10': { '.': '0.5.10', 'uuid@10.0.0': '10.0.0' }, + }); +}); + +test('rejects missing or incompatible locked transitive provider edges before installing', () => { + const missing = providerLock(); + delete missing.packages['node_modules/uuid']; + assert.throws(() => verifier.lockedMiddlewareManifest(manifest(), missing), /Missing locked dependency uuid/); + const incompatible = providerLock(); + incompatible.packages['node_modules/uuid'].version = '11.0.0'; + assert.throws(() => verifier.lockedMiddlewareManifest(manifest(), incompatible), /must satisfy \^10.0.0/); +}); + +test('pins unique transitive providers globally so fresh registry versions cannot escape nested selectors', () => { + const result = verifier.lockedMiddlewareManifest(manifest(), providerLock()); + assert.deepEqual(result.overrides.langsmith, { '.': '0.5.10', 'uuid@10.0.0': '10.0.0' }); + assert.equal(result.overrides.uuid, '10.0.0'); +}); + +test('preserves incompatible provider versions under their owning dependency subtrees', () => { + const multiple = providerLock(); + multiple.packages['node_modules/@langchain/core'].dependencies['p-queue'] = '^6.0.0'; + multiple.packages['node_modules/@langchain/langgraph-sdk'].dependencies['p-queue'] = '^9.0.0'; + multiple.packages['node_modules/p-queue'] = { version: '6.6.2' }; + multiple.packages['node_modules/@langchain/langgraph-sdk/node_modules/p-queue'] = { version: '9.1.0' }; + const result = verifier.lockedMiddlewareManifest(manifest(), multiple); + assert.equal(Object.hasOwn(result.overrides, 'p-queue'), false); + assert.equal(result.overrides['@langchain/core@1.2.9']['p-queue@6.6.2'], '6.6.2'); + assert.equal(result.overrides['@langchain/langgraph@1.4.13']['@langchain/langgraph-sdk@1.2.0']['p-queue@9.1.0'], '9.1.0'); + verifier.assertMiddlewareVendorGraph(manifest(), multiple, multiple); + const flattened = structuredClone(multiple); + delete flattened.packages['node_modules/@langchain/langgraph-sdk/node_modules/p-queue']; + assert.throws(() => verifier.assertMiddlewareVendorGraph(manifest(), multiple, flattened), /must satisfy \^9.0.0/); +}); + +test('globally pins one distinct provider version repeated at multiple owner paths', () => { + const repeated = providerLock(); + repeated.packages['node_modules/@langchain/core'].dependencies.child = '^1.0.0'; + repeated.packages['node_modules/@langchain/langgraph-sdk'].dependencies.child = '^1.0.0'; + repeated.packages['node_modules/@langchain/core/node_modules/child'] = { version: '1.0.0' }; + repeated.packages['node_modules/@langchain/langgraph-sdk/node_modules/child'] = { version: '1.0.0' }; + const result = verifier.lockedMiddlewareManifest(manifest(), repeated); + assert.equal(result.overrides.child, '1.0.0'); + assert.equal(result.overrides['@langchain/core@1.2.9']['child@1.0.0'], '1.0.0'); + assert.equal(result.overrides['@langchain/langgraph@1.4.13']['@langchain/langgraph-sdk@1.2.0']['child@1.0.0'], '1.0.0'); + verifier.assertMiddlewareVendorGraph(manifest(), repeated, repeated); + const drift = structuredClone(repeated); + drift.packages['node_modules/@langchain/langgraph-sdk/node_modules/child'].version = '1.0.1'; + assert.throws(() => verifier.assertMiddlewareVendorGraph(manifest(), repeated, drift), /preserve the reviewed locked graph/); +}); + +test('independently rejects installed provider drift, missing vendors, and invalid edges', () => { + assert.equal(typeof verifier.assertMiddlewareVendorGraph, 'function'); + verifier.assertMiddlewareVendorGraph(manifest(), providerLock(), providerLock()); + const drift = providerLock(); + drift.packages['node_modules/langsmith'].version = '0.5.11'; + assert.throws(() => verifier.assertMiddlewareVendorGraph(manifest(), providerLock(), drift), /preserve the reviewed locked graph/); + const missing = providerLock(); + delete missing.packages['node_modules/uuid']; + assert.throws(() => verifier.assertMiddlewareVendorGraph(manifest(), providerLock(), missing), /Missing locked dependency uuid/); + const invalid = providerLock(); + invalid.packages['node_modules/langsmith'].version = '1.0.0'; + assert.throws(() => verifier.assertMiddlewareVendorGraph(manifest(), providerLock(), invalid), /must satisfy/); +}); + test('pins declared peers and the compiler from the root lock and rejects unsupported peer ranges', () => { assert.equal(typeof verifier.lockedMiddlewareManifest, 'function'); const result = verifier.lockedMiddlewareManifest(manifest(), lock);