From 35a3670b2f3c3440d98912cbc3be2e7374eeb736 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Wed, 9 Sep 2026 11:29:22 +0400 Subject: [PATCH 01/16] fix(reservations) --- CHANGELOG.md | 32 ++ pyproject.toml | 2 +- src/nullrun/actions.py | 35 +- src/nullrun/breaker/exceptions.py | 386 +++++++++++++++--- src/nullrun/decorators.py | 22 +- src/nullrun/instrumentation/auto.py | 13 +- src/nullrun/instrumentation/langgraph.py | 133 ++++--- src/nullrun/runtime.py | 223 ++++++++--- src/nullrun/transport.py | 101 ++++- tests/test_2026_09_08_gate_first_execute.py | 296 ++++++++++++++ tests/test_actions.py | 46 ++- tests/test_decision_split.py | 28 +- tests/test_exception_hierarchy.py | 53 ++- tests/test_gate_real_path.py | 15 +- tests/test_typed_exceptions_full_audit.py | 418 ++++++++++++++++++++ tests/test_v3_wire_contract.py | 47 +++ 16 files changed, 1610 insertions(+), 240 deletions(-) create mode 100644 tests/test_2026_09_08_gate_first_execute.py create mode 100644 tests/test_typed_exceptions_full_audit.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 88cc3b6..984505e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,35 @@ +## [0.16.6] - 2026-09-08 + +Patch release — closes the SDK↔backend drift introduced by backend `DEF-SDKK-022-EXEC-BYPASS` (2026-09-04, RUN_ID=20260904T1500). After that backend fix, `/api/v1/execute` runs an `execution:{id}` ownership-binding existence check and returns 404 EXECUTION_NOT_FOUND for any execution_id that was not minted by a prior `/api/v1/gate`. The SDK's `runtime.execute()` had been minting a fresh `uuid7_str()` regardless of prior `/gate`, so every `@protect @sensitive` call returned 404 ("Gateway returned 404") and the displayed workflow_id was the misleading `__nullrun_unknown__` sentinel. LangGraph's `NullRunCallback.on_llm_start` had the symmetric problem on the LLM span side: it fired `llm_call` cost events with no paired `/gate` reservation, so the runtime's `_route_track` silently dropped them. This release closes all three holes. No wire-format change. + +### Fixed + +- **DEFS-SDKEXEC-GATE-FIRST** — `runtime.execute()` reuses the server-minted execution_id from `_server_minted_execution_id_var` when a prior `/gate` minted it (`src/nullrun/runtime.py:2820+`). Pre-fix minted `uuid7_str()` unconditionally; post-fix reads the contextvar (set by `check_workflow_budget`'s `_capture_server_minted_execution_id` from the `/gate` response's `reservation_id` field) and only mints fresh when the contextvar is empty (direct callers without a prior `/gate`, which is a wire-contract violation the backend's 404 handles correctly). Comment block at the fix site names both DEFS-SDKEXEC-GATE-FIRST and DEF-SDKK-022-EXEC-BYPASS so future readers see the round-trip contract without searching. +- **DEFS-SDKEXEC-WORKFLOW-LABEL** — `_enforce_sensitive_tool` displays the API key's bound workflow via `runtime._resolve_workflow_id(get_workflow_id())` instead of the literal `__nullrun_unknown__` sentinel (`src/nullrun/decorators.py`). The wire still carries the same workflow_id (server-side binding); only the displayed label changes. Two sites updated (extract failure path + main path). +- **DEFS-SDKEXEC-LLM-RESERVATION** — `NullRunCallback.on_llm_start` (`src/nullrun/instrumentation/langgraph.py`) fires `runtime.check_workflow_budget()` (fail-OPEN) so the matching `on_llm_end` `llm_call` cost event has a server-minted reservation_id and routes via `/track_single` instead of being dropped by `runtime._route_track` (the WARNING log "dropping llm_call event — no server-minted reservation_id in scope"). The call is wrapped in `except BaseException` so a backend outage or `WorkflowKilledInterrupt` / `WorkflowPausedException` never breaks the LangChain callback contract. +- **`Transport.execute` docstring** (`src/nullrun/transport.py`) — rewrites the misleading pre-2026-09-04 claim ("/execute MUST be called rather than /gate") to reflect the post-DEF-SDKK-022-EXEC-BYPASS contract ("/execute MUST be preceded by /gate for the same execution_id"). Names both fix tags so the contract is grep-able. + +### Added + +- **`tests/test_2026_09_08_gate_first_execute.py`** (9 tests). Source-pin regression for all three fixes: + - `runtime.execute()` reads `get_server_minted_execution_id()` and reuses it when present (forbids re-introducing an unconditional `uuid7_str()` mint outside the fallback arm). + - `_enforce_sensitive_tool` displays via `runtime._resolve_workflow_id(...)` (forbids the pre-fix contextvar-only fallback). + - `Transport.execute` docstring references the post-fix contract (forbids the legacy misleading claim). + - `NullRunCallback.on_llm_start` calls `check_workflow_budget()` with a never-raise guard. + - Contextvar round-trip sanity (`set_server_minted_execution_id` / `get_server_minted_execution_id`). + +### Compatibility + +Pure reliability fixes — no wire-format change. `/gate`, `/execute`, `/track`, `/cancel` payloads are byte-identical to 0.16.5. The drift existed only on the SDK side; this release brings the SDK in line with the backend's 2026-09-04 contract without rolling back any backend-side hardening. + +### Why this is needed + +**Gate-first** — the user-facing symptom was that `langgraph_openai_approval_demo.py` (and any `@protect @sensitive` decorator that was actually wired through `runtime.execute()`) returned `Workflow __nullrun_unknown__ blocked: Gateway returned 404` for every call, with `action=block, status_code=None`. The approval rule never had a chance to fire because the 404 was raised on the existence-of-binding check before the policy engine ran. The 0.12.0 SDK had been silently broken against post-2026-09-04 backends for the entire /execute path; this release closes the four-day window of broken `/execute` behaviour. + +**Workflow label** — `__nullrun_unknown__` was misleading because the SDK did know the workflow (the API key's binding) but only read the contextvar (which was unset on bare `@protect` calls). The displayed label was wrong; the wire was right. Operators reading traces had no signal that the gate had, in fact, scoped the call to a real workflow. + +**LLM reservation** — LangGraph's `NullRunCallback` emits LLM cost events from the LangChain callback hooks. These have no `@protect` scope and therefore no paired `/gate`. The runtime's `_route_track` (which since v0.16.0 / 2026-08-20 backend v3.66.2 alignment refuses to fall back to `/track/batch` for `llm_call` events without a reservation) dropped them with a WARNING log. Cost attribution for agentic LLM loops was silently incomplete. The fix fires `/gate` once per LLM span (fail-OPEN; same wire-call shape as `@protect`), so cost attribution completes via the v3 `/track_single` path. + ## [0.16.5] - 2026-09-05 Patch release — two independent reliability fixes: (1) `@protect` cancel-on-exception orphan leak (Redis reservation leak on tool exceptions), (2) P0-26+P0-27 `operation_id` hoist (single-source mint, server-vs-SDK divergence detection). No wire-format change on either fix. diff --git a/pyproject.toml b/pyproject.toml index 93398d1..0fd3707 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,7 +6,7 @@ build-backend = "hatchling.build" name = "nullrun" # Full release history lives in CHANGELOG.md; only the current version # is pinned here. -version = "0.16.5" +version = "0.16.6" # Kept under the 200-char preview threshold so the full line is visible # without an "expand" click. The headline is the canonical §1 statement # from positioning.md — "runtime decision layer for tool-using AI agents" diff --git a/src/nullrun/actions.py b/src/nullrun/actions.py index f4d117c..6fd7c44 100644 --- a/src/nullrun/actions.py +++ b/src/nullrun/actions.py @@ -22,6 +22,7 @@ from nullrun.breaker.exceptions import ( NullRunBlockedException, + NullRunWorkflowKilledError, WorkflowKilledInterrupt, WorkflowPausedException, ) @@ -69,7 +70,7 @@ class ActionHandler: Handler for NullRun circuit breaker actions. This executes protective actions when triggered: - - KILL: Immediately stops the workflow (raises WorkflowKilledInterrupt) + - KILL: Immediately stops the workflow (raises NullRunWorkflowKilledError, NR-W002) - PAUSE: Temporarily halts the workflow (raises WorkflowPausedException) - ALERT: Sends notification (can be customized) - SNAPSHOT: Captures workflow state for debugging @@ -179,7 +180,10 @@ def handle( **details: Additional details about the action Raises: - WorkflowKilledInterrupt: If action is "kill" + NullRunWorkflowKilledError: If action is "kill" + (2026-09-08 typed signal, NR-W002; subclass of + WorkflowKilledInterrupt which remains as the + back-compat name.) WorkflowPausedException: If action is "pause" NullRunBlockedException: If action is "block" """ @@ -230,12 +234,11 @@ def handle( except BaseException as e: # Don't let handler exceptions propagate. We catch # `BaseException` (not just `Exception`) because - # `WorkflowKilledInterrupt` is intentionally a - # `BaseException` subclass — it's a non-recoverable - # control signal, but inside the ActionHandler dispatch - # loop we want the kill to be recorded in history - # (already done above) and swallowed, NOT re-raised into - # the caller's frame. + # kill signals (NullRunWorkflowKilledError, the + # 2026-09-08-migrated Exception subclass) and any + # third-party kill-shaped signals must be recorded + # in history (already done above) and swallowed, + # NOT re-raised into the caller's frame. logger.error(f"Action handler error: {e}") def _default_kill( @@ -244,9 +247,21 @@ def _default_kill( reason: str, **details: Any, ) -> None: - """Default kill handler - raises WorkflowKilledInterrupt.""" + """Default kill handler - raises NullRunWorkflowKilledError. + + 2026-09-08: typed kill signal (NR-W002). Cookbook code + can `except NullRunWorkflowKilledError` to react to + operator-initiated kills with structured error_code + + user_action. Legacy `except WorkflowKilledInterrupt` + still matches because NullRunWorkflowKilledError is a + subclass. + """ logger.warning(f"KILL action for workflow {workflow_id}: {reason}") - raise WorkflowKilledInterrupt(workflow_id=workflow_id, reason=reason) + raise NullRunWorkflowKilledError( + workflow_id=workflow_id, + reason=reason, + kill_source="action_handler", + ) def _default_pause( self, diff --git a/src/nullrun/breaker/exceptions.py b/src/nullrun/breaker/exceptions.py index 11b3c32..0cc722a 100644 --- a/src/nullrun/breaker/exceptions.py +++ b/src/nullrun/breaker/exceptions.py @@ -842,6 +842,33 @@ def __init__( self.recheck_retryable: bool = True +class NullRunBudgetThrottleError(NullRunBudgetError): + """Backend returned ``decision == "throttle"`` — soft budget signal. + + Distinct from :class:`NullRunBudgetError` (NR-B004, the hard-block + case raised when ``decision == "block"``). Throttle means + "rate-limit this workflow but don't fully block it" — a temporary + pacing signal that the SDK surfaces as a typed exception so + cookbook code can back off and retry, vs. the hard block where + the same parameters would fail again. + + Added 2026-09-08 to retire the generic ``WorkflowKilledInterrupt`` + raise on the throttle path. Cookbook pattern: catch this + specifically (``except NullRunBudgetThrottleError``), sleep for + the cooldown window, and retry — distinct from the hard block + where retrying with the same budget tier is futile. + """ + + error_code = "NR-B007" + user_action = ( + "Backend throttled this workflow (soft budget signal). Wait " + "for the cooldown window shown in the response and retry — " + "do NOT request a budget increase for a throttle (that is " + "the wrong remediation; the issue is pacing, not cap)." + ) + retryable = True + + class NullRunToolBlockedError(NullRunBlockedException): """The tool is in the workflow's block list. @@ -887,6 +914,12 @@ class NullRunApprovalNotYetApprovedError(NullRunBlockedException): no — terminal) and from :class:`NullRunApprovalExpiredError` (operator said yes but grant TTL elapsed). All three share the HTTP 403 envelope; the wire code is the discriminator. + + Also raised client-side (NOT just wire path) on the /execute + "approval_id missing in response" malformed-payload case — see + ``NullRunApprovalResponseMissingError`` for the precise semantic + distinction (NR-A004 is the wire-bug code, NR-A010 is "operator + has not decided yet"). """ error_code = "NR-A010" @@ -897,6 +930,55 @@ class NullRunApprovalNotYetApprovedError(NullRunBlockedException): ) retryable = True + def __init__( + self, + workflow_id: str, + reason: str, + action: str = "block", + tool_name: str | None = None, + status_code: int | None = None, + *, + approval_id: str | None = None, + **details: Any, + ) -> None: + super().__init__( + workflow_id=workflow_id, + reason=reason, + action=action, + tool_name=tool_name, + status_code=status_code, + **details, + ) + # First-class attribute so cookbook code can introspect the + # pending approval row without parsing the message string. + self.approval_id = approval_id + + +class NullRunApprovalResponseMissingError(NullRunBlockedException): + """``/execute`` returned ``require_approval`` but the response body + did not include an ``approval_id`` — wire-bug / server drift. + + Wire code ``NR-A004`` (was previously set inline on a generic + ``NullRunBlockedException`` at runtime.py:2888, 2914, 2929 — promoted + to a typed class for parity with the six approval exceptions above). + This is distinct from ``NullRunApprovalNotYetApprovedError`` (NR-A010) + which is "the operator has not yet decided". Here the operator never + had a chance — the wire envelope was incomplete. + + Cookbook pattern: do NOT retry the same execution_id; the backend + needs a fix or the wire-shape contract needs re-reading. Log the + full response body and report to NULLRUN support. + """ + + error_code = "NR-A004" + user_action = ( + "Server returned require_approval without an approval_id — " + "this is a wire-contract bug, NOT a transient failure. Inspect " + "the full response body and report to NullRun support; do not " + "retry the same execution_id." + ) + retryable = False + class NullRunApprovalDeniedError(NullRunBlockedException): """Operator explicitly denied the approval. @@ -905,6 +987,10 @@ class NullRunApprovalDeniedError(NullRunBlockedException): with the same approval_id will keep failing. Cookbook pattern: surface denial to the user and request a fresh approval row (different parameters / intent). + + Now raised client-side (NOT just wire path) on the WS push "denied" + outcome at ``check_workflow_budget`` and on the /execute "outcome + != approved" branch. """ error_code = "NR-A011" @@ -915,24 +1001,136 @@ class NullRunApprovalDeniedError(NullRunBlockedException): ) retryable = False + def __init__( + self, + workflow_id: str, + reason: str, + action: str = "block", + tool_name: str | None = None, + status_code: int | None = None, + *, + approval_id: str | None = None, + denial_note: str | None = None, + **details: Any, + ) -> None: + super().__init__( + workflow_id=workflow_id, + reason=reason, + action=action, + tool_name=tool_name, + status_code=status_code, + **details, + ) + self.approval_id = approval_id + self.denial_note = denial_note + class NullRunApprovalExpiredError(NullRunBlockedException): """Approval grant aged out — operator said yes but ``expires_at`` is past. - Wire code ``APPROVAL_EXPIRED`` (HTTP 403). The original grant was - approved but the operator's approval window elapsed before - ``/execute`` consumed it. Cookbook pattern: request a fresh - approval row (do not retry the same one). + Wire code ``APPROVAL_EXPIRED`` (HTTP 403). Two raise paths: + + 1. **Wire path** — backend returns APPROVAL_EXPIRED on /execute + because the operator's grant TTL elapsed between /gate and + /execute. + 2. **Client-side timeout path** (added 2026-09-08, the trigger for + this typed exception migration) — WS push went silent for + ``approval_timeout_seconds`` (default 300s) without an operator + decision. The SDK raises this exception instead of the generic + ``WorkflowKilledInterrupt`` so cookbook code can catch it + (`except NullRunApprovalExpiredError`) and react with a fresh + approval request. + + Cookbook pattern: do NOT retry the same approval_id — request a + fresh row and re-/gate. """ error_code = "NR-A012" user_action = ( - "Approval grant has expired — the operator approved, but " - "the grant's expires_at is past. Request a fresh approval " - "row and retry /execute with the new approval_id." + "Approval expired — no operator decision within the configured " + "timeout window (WS push silent past approval_timeout_seconds). " + "Request a fresh approval row and retry /gate; the previous " + "approval_id cannot be revived." + ) + retryable = False + + def __init__( + self, + workflow_id: str, + reason: str, + action: str = "block", + tool_name: str | None = None, + status_code: int | None = None, + *, + approval_id: str | None = None, + timeout_seconds: float | None = None, + local_timeout: bool = False, + **details: Any, + ) -> None: + super().__init__( + workflow_id=workflow_id, + reason=reason, + action=action, + tool_name=tool_name, + status_code=status_code, + **details, + ) + self.approval_id = approval_id + # Server-authoritative timeout the SDK waited for. ``None`` when + # the exception came from the wire path (where the backend + # already closed the grant; the SDK never started a wait). + self.timeout_seconds = timeout_seconds + # True when raised by the SDK on local WS-silent timeout (path 2 + # above); False when raised by the wire path (path 1). Lets + # cookbook code distinguish "operator never saw the request" + # (local timeout — maybe the request never propagated) from + # "operator approved but grant TTL elapsed" (wire path). + self.local_timeout = local_timeout + + +class NullRunApprovalReplayRejectedError(NullRunBlockedException): + """Approval grant was already consumed by a prior /execute call. + + Wire code ``APPROVAL_REPLAY_REJECTED`` (HTTP 403). Each grant + is single-use per ``consume_approved`` atomic check-and-set. + Cookbook pattern: do NOT retry the same approval_id; treat as + idempotency violation (likely a client retry loop). + + Also raised client-side on the /execute "post-approval re-check + returned require_approval again" race (the operator approved but + the same approval_id was already consumed by a concurrent /execute). + """ + + error_code = "NR-A015" + user_action = ( + "Approval grant was already consumed by a prior /execute " + "call — this is a replay/retry-loop signal, NOT a transient " + "failure. Inspect your retry logic; the same approval_id " + "will never succeed twice." ) retryable = False + def __init__( + self, + workflow_id: str, + reason: str, + action: str = "block", + tool_name: str | None = None, + status_code: int | None = None, + *, + approval_id: str | None = None, + **details: Any, + ) -> None: + super().__init__( + workflow_id=workflow_id, + reason=reason, + action=action, + tool_name=tool_name, + status_code=status_code, + **details, + ) + self.approval_id = approval_id + class NullRunApprovalDigestMismatchError(NullRunBlockedException): """Business-impact digest drifted since operator approval (ADR-006). @@ -975,23 +1173,13 @@ class NullRunApprovalToolDigestMismatchError(NullRunBlockedException): retryable = False -class NullRunApprovalReplayRejectedError(NullRunBlockedException): - """Approval grant was already consumed by a prior /execute call. - - Wire code ``APPROVAL_REPLAY_REJECTED`` (HTTP 403). Each grant - is single-use per ``consume_approved`` atomic check-and-set. - Cookbook pattern: do NOT retry the same approval_id; treat as - idempotency violation (likely a client retry loop). - """ - - error_code = "NR-A015" - user_action = ( - "Approval grant was already consumed by a prior /execute " - "call — this is a replay/retry-loop signal, NOT a transient " - "failure. Inspect your retry logic; the same approval_id " - "will never succeed twice." - ) - retryable = False +# NOTE: NullRunApprovalReplayRejectedError was moved earlier in this +# module (alongside the other five approval exceptions) so all six +# typed approval exceptions are co-located. The earlier definition +# also adds an explicit ``__init__`` accepting ``approval_id`` as a +# first-class attribute. See the block just below the +# ``NullRunApprovalNotYetApprovedError`` docstring for the canonical +# definition. # NOTE: the following six exception classes were removed in 0.4.0 @@ -1095,27 +1283,35 @@ def __init__(self, workflow_id: str, reason: str) -> None: super().__init__(f"Workflow {workflow_id} killed: {reason}") -class WorkflowKilledInterrupt(WorkflowKilledException): +class WorkflowKilledInterrupt(NullRunError): """ Raised when a workflow is killed by the NullRun control plane. - Inherits from the deprecated:class:`WorkflowKilledException` - (which is itself a ``BaseException`` subclass, not ``Exception``) - so that: + **2026-09-08 migration**: this class is now an ``Exception`` + subclass (``NullRunError`` parent) — formerly ``BaseException``. + The user override: agent recovery code needs to catch the kill + signal via ``except WorkflowKilledInterrupt`` or + ``except NullRunWorkflowKilledError`` to surface a structured + error to the user with ``error_code=NR-W002`` and ``user_action``. + + Migration back-compat guarantees (all three hold): + + * ``except WorkflowKilledInterrupt`` (new code) — still matches, + including legacy raises that haven't been updated. + * ``except NullRunError`` — now matches (was NO match before + migration; this is the new ability the user wanted). + * ``except NullRunWorkflowKilledError`` — matches (preferred + typed name for new cookbook code). - * ``except WorkflowKilledInterrupt`` (new code) catches new raises - and only new raises. - * ``except WorkflowKilledException`` (legacy user code) still - catches new raises — back-compat. - * ``except Exception`` does **not** catch this signal — kill is - not a recoverable error. Mirrors the ``KeyboardInterrupt`` / - ``SystemExit`` pattern from the standard library: user code - that catches ``except Exception`` and re-runs the work will - silently bypass the kill. - * ``except BaseException`` catches it, like the stdlib interrupts. + Migration BREAK (acceptable, documented in CHANGELOG): - See ``docs/kill-contract.md` for the full rationale, including - the four-level coverage model and the decision tree for users. + * ``except WorkflowKilledException`` (the deprecated parent + class) — no longer matches. The parent class remains + BaseException and emits DeprecationWarning on construction, + but is no longer in the ``WorkflowKilledInterrupt`` MRO. Code + that catches the deprecated name must migrate to either + ``WorkflowKilledInterrupt`` (keep current name) or + ``NullRunWorkflowKilledError`` (preferred typed name). Fields: workflow_id: The workflow that was killed. @@ -1124,31 +1320,99 @@ class WorkflowKilledInterrupt(WorkflowKilledException): Catching in production ---------------------- - ``WorkflowKilledInterrupt`` is a ``BaseException`` subclass - (NOT ``Exception``), so a user-agent ``try / except Exception`` - will not catch it. This is intentional — the kill signal - must reach the top of the loop. It does mean, however, that - Sentry / OpenTelemetry default error handlers (which filter - on ``Exception``) will not record the kill event unless the - user's code re-raises it under an ``except BaseException``: - - from sentry_sdk import capture_exception + ``WorkflowKilledInterrupt`` is now an ``Exception`` subclass. + Cookbook code can do:: + + try: + agent.run() + except NullRunWorkflowKilledError as exc: + surface_to_user( + f"Workflow {exc.workflow_id} was killed: {exc.reason}. " + f"{exc.user_action}" + ) + + or for broader catch:: + try: - agent.run - except BaseException: - capture_exception # records kill, ctrl-c, system-exit + agent.run() + except Exception as exc: + # Now catches kill signals too (the new contract). + sentry_sdk.capture_exception(exc) raise - ``except Exception`` will swallow non-kill errors but let the - kill through. ``except BaseException`` captures everything - including the kill — recommended for the top of an agent loop. + Sentry / OpenTelemetry handlers that filter on ``Exception`` will + now record kill events — this is the intended new behavior. Code + that relies on kill being un-catchable by ``except Exception`` is + a regression candidate; see ``docs/kill-contract-migration-2026-09-08.md``. """ - def __init__(self, workflow_id: str, reason: str) -> None: - # Bypass the parent's __init__ so constructing the canonical - # class does NOT trigger the parent's DeprecationWarning. The - # deprecation is about using the old *name* — not the - # BaseException-based hierarchy. + error_code = "NR-W002" + user_action = ( + "The workflow was killed by the NullRun control plane. The " + "body did not run. Inspect the reason (killed via dashboard, " + "killed via API, circuit-breaker tripped, etc.) and, if " + "appropriate, resume the workflow at " + "https://app.nullrun.io/workflows/." + ) + retryable = False + + def __init__( + self, + workflow_id: str, + reason: str, + *, + kill_source: str | None = None, + **details: Any, + ) -> None: + # Skip NullRunError.__init__'s kwargs-by-key path — we want + # the structured fields attached as instance attrs (matches + # the pre-migration shape) AND surfaced through the NullRunError + # fields too, so cookbook introspection works either way. self.workflow_id = workflow_id self.reason = reason - BaseException.__init__(self, f"Workflow {workflow_id} killed: {reason}") + # First-class attribute distinguishing operator kill from + # circuit-breaker kill, etc. None when the source is ambiguous. + self.kill_source = kill_source + NullRunError.__init__( + self, + f"Workflow {workflow_id} killed: {reason}", + error_code=self.error_code, + user_action=self.user_action, + **details, + ) + + +class NullRunWorkflowKilledError(WorkflowKilledInterrupt): + """Typed public name for the kill signal. + + Subclass of :class:`WorkflowKilledInterrupt` (which remains the + legacy canonical name) so ``except WorkflowKilledInterrupt`` + clauses continue to match. New cookbook code should prefer this + name (``except NullRunWorkflowKilledError``) for typed dispatch. + + Wire code ``NR-W002`` (same as parent). Distinct from + :class:`NullRunBlockedException` family — kill is a control-plane + signal (operator or circuit-breaker), not a gate-decision block. + + Cookbook pattern (2026-09-08 migration): + + try: + agent.run() + except NullRunWorkflowKilledError as exc: + # Structured fields ready for the LLM: + # exc.workflow_id, exc.reason, exc.kill_source, + # exc.error_code ("NR-W002"), exc.user_action + surface_to_user( + f"Workflow {exc.workflow_id} was killed " + f"(source={exc.kill_source}): {exc.user_action}" + ) + """ + + error_code = "NR-W002" + user_action = ( + "Workflow was killed by the NullRun control plane (operator " + "action or circuit-breaker). The body did not run. Resume " + "the workflow at https://app.nullrun.io/workflows/ " + "or inspect the reason before retrying." + ) + retryable = False diff --git a/src/nullrun/decorators.py b/src/nullrun/decorators.py index 4d5c17c..bc1d926 100644 --- a/src/nullrun/decorators.py +++ b/src/nullrun/decorators.py @@ -742,7 +742,13 @@ def _enforce_sensitive_tool( TransportErrorSource, ) - workflow_id = get_workflow_id() or UNKNOWN_WORKFLOW_ID + # DEFS-SDKEXEC-WORKFLOW-LABEL (2026-09-08): prefer the + # runtime's bound workflow (from _authenticate) over the + # sentinel so the displayed label matches what the SDK + # actually sends to /gate / /execute. See the matching + # note in `_enforce_sensitive_tool` below for the full + # rationale. + workflow_id = runtime._resolve_workflow_id(get_workflow_id()) or UNKNOWN_WORKFLOW_ID # The user-facing hint depends on which extractor fired. # Money extractor wants the bound arg name; ToolParams # extractor wants the rule-param -> arg-name mapping @@ -807,7 +813,19 @@ def _enforce_sensitive_tool( ) fail_open = os.environ.get("NULLRUN_SENSITIVE_FAIL_OPEN", "").strip() == "1" - workflow_id = get_workflow_id() or UNKNOWN_WORKFLOW_ID + # DEFS-SDKEXEC-WORKFLOW-LABEL (2026-09-08): resolve the + # *display* workflow_id via the runtime's precedence chain + # (contextvar → self.workflow_id → None) so the label reflects + # what the SDK actually sends on the wire (the API key's bound + # workflow, when the user hasn't explicitly opened a + # ``with workflow(...)`` block). Pre-fix this read only the + # contextvar; on every API-key-bound key without an explicit + # workflow block the displayed label was the literal sentinel + # ``"__nullrun_unknown__"``, which misleads operators reading + # the trace and the block message into thinking the gate was + # unable to identify the workflow. Sentinel stays as the last + # resort for legacy / never-bound keys. + workflow_id = runtime._resolve_workflow_id(get_workflow_id()) or UNKNOWN_WORKFLOW_ID try: # Pass on_transport_error="raise" so the transport raises diff --git a/src/nullrun/instrumentation/auto.py b/src/nullrun/instrumentation/auto.py index 501a28e..f135e20 100644 --- a/src/nullrun/instrumentation/auto.py +++ b/src/nullrun/instrumentation/auto.py @@ -740,7 +740,10 @@ def _check_kill_before_send(runtime: Any, request: httpx.Request) -> None: has a remote control plane to consult. Raises: - WorkflowKilledInterrupt: state == "Killed" + NullRunWorkflowKilledError: state == "Killed" (2026-09-08: + typed signal with error_code=NR-W002 + user_action; + subclass of WorkflowKilledInterrupt which remains as a + back-compat name.) WorkflowPausedException: state == "Paused" """ if runtime is None: @@ -761,10 +764,14 @@ def _check_kill_before_send(runtime: Any, request: httpx.Request) -> None: state = runtime._remote_state_for(workflow_id) if hasattr(runtime, "_remote_state_for") else getattr(runtime, "_remote_states", {}).get(workflow_id, {}) state_name = state.get("state", "Normal") if state_name == "Killed": - from nullrun.breaker.exceptions import WorkflowKilledInterrupt - raise WorkflowKilledInterrupt( + # 2026-09-08: typed kill signal (NR-W002). Cookbook + # code can `except NullRunWorkflowKilledError`; legacy + # `except WorkflowKilledInterrupt` still matches (subclass). + from nullrun.breaker.exceptions import NullRunWorkflowKilledError + raise NullRunWorkflowKilledError( workflow_id=workflow_id, reason=state.get("reason", "remote kill"), + kill_source="auto_instrumentation", ) if state_name == "Paused": from nullrun.breaker.exceptions import WorkflowPausedException diff --git a/src/nullrun/instrumentation/langgraph.py b/src/nullrun/instrumentation/langgraph.py index 107fd46..c784aeb 100644 --- a/src/nullrun/instrumentation/langgraph.py +++ b/src/nullrun/instrumentation/langgraph.py @@ -82,11 +82,16 @@ def _read_token_attrs(obj: Any) -> tuple[int, int, int, dict[str, Any]] | None: total_t = getattr(obj, "total_tokens", 0) or 0 if not (in_t or out_t or total_t): return None - return int(in_t), int(out_t), int(total_t), { - "input_tokens": in_t, - "output_tokens": out_t, - "total_tokens": total_t, - } + return ( + int(in_t), + int(out_t), + int(total_t), + { + "input_tokens": in_t, + "output_tokens": out_t, + "total_tokens": total_t, + }, + ) return None @@ -191,6 +196,7 @@ def _get_finish_reason(response: Any) -> str | None: # Usage Normalization (SDK extracts, backend computes) # ============================================================================= + def extract_usage_from_response(response: Any, provider: str, model: str) -> dict[str, Any]: """ Extract usage data from LLM response. @@ -258,7 +264,7 @@ def extract_usage_from_response(response: Any, provider: str, model: str) -> dic # Check for streaming chunks that accumulated usage # (streaming responses may not have usage until final chunk) - if not usage["has_usage"] and hasattr(response, '__iter__'): + if not usage["has_usage"] and hasattr(response, "__iter__"): # For streaming, we can't get accurate usage in middle of stream # Final response should have usage_metadata pass @@ -273,29 +279,19 @@ def extract_usage_from_response(response: Any, provider: str, model: str) -> dic # OpenAI exposes cached_tokens on a nested prompt_tokens_details. raw = usage.get("raw_usage") or {} if isinstance(raw, dict): - cache_read = raw.get("cache_read_input_tokens") or raw.get( - "cacheReadInputTokenCount" - ) + cache_read = raw.get("cache_read_input_tokens") or raw.get("cacheReadInputTokenCount") if cache_read: usage["cache_read_tokens"] = int(cache_read) or 0 - cache_write = raw.get("cache_creation_input_tokens") or raw.get( - "cacheWriteInputTokenCount" - ) + cache_write = raw.get("cache_creation_input_tokens") or raw.get("cacheWriteInputTokenCount") if cache_write: usage["cache_write_tokens"] = int(cache_write) or 0 prompt_details = raw.get("prompt_tokens_details") or {} if isinstance(prompt_details, dict) and prompt_details.get("cached_tokens"): # OpenAI's prefix-cached prompt hits — best-effort merge. - usage["cache_read_tokens"] = int( - prompt_details.get("cached_tokens") or 0 - ) + usage["cache_read_tokens"] = int(prompt_details.get("cached_tokens") or 0) completion_details = raw.get("completion_tokens_details") or {} - if isinstance(completion_details, dict) and completion_details.get( - "reasoning_tokens" - ): - usage["reasoning_tokens"] = int( - completion_details.get("reasoning_tokens") or 0 - ) + if isinstance(completion_details, dict) and completion_details.get("reasoning_tokens"): + usage["reasoning_tokens"] = int(completion_details.get("reasoning_tokens") or 0) # Finish reason — read from every known source independently of the # token branch. The `elif`-chain above means only one branch fills @@ -370,9 +366,7 @@ def _extract_tool_names(obj: Any) -> list[str]: # Determine if we got real usage data usage["has_usage"] = ( - usage["total_tokens"] > 0 or - usage["input_tokens"] > 0 or - usage["output_tokens"] > 0 + usage["total_tokens"] > 0 or usage["input_tokens"] > 0 or usage["output_tokens"] > 0 ) return usage @@ -518,6 +512,50 @@ def on_llm_start(self, serialized: Any, prompts: Any, **kwargs: Any) -> None: else: ctx = create_root_span() self._register_active_run(str(run_id), ctx) + + # DEFS-SDKEXEC-LLM-RESERVATION (2026-09-08): pair the LLM + # span with a server-minted reservation so the matching + # llm_call cost event emitted by ``on_llm_end`` lands on + # ``/track_single`` instead of being dropped by + # ``runtime._route_track`` (which returns silently when + # ``_server_minted_execution_id_var`` is unset — see + # ``runtime.py:3167`` and the WARNING log at line 3198). + # + # Pattern: ``check_workflow_budget`` fails OPEN on transport + # error (CLAUDE.md §4 / ADR-008 / ``runtime.py:2022-2037``) + # — a backend outage silently returns without raising AND + # without capturing a reservation. The downstream + # ``_route_track`` will then drop the matching llm_call + # cost event (v3.66.2 alignment — backend rejects batched + # llm_call events without a reservation with 503 + # BUDGET_RECHECK_FAILED). This matches the pre-fix + # behaviour because pre-fix the SDK also had no reservation + # at this site (no /check round-trip happened on the LLM + # span) and the llm_call cost event was dropped the same + # way. We swallow ``WorkflowKilledInterrupt`` / + # ``WorkflowPausedException`` because the LangChain + # callback contract is "never raise" (the framework breaks + # if a callback raises); the kill/pause signal still + # propagates because the next @protect on a sensitive tool + # re-runs ``check_control_plane``. + # + # Cost: one extra /gate round-trip per LLM span (~5 ms in + # the hot path). The /track emitted by ``on_llm_end`` + # consumes the matching reservation, so the budget ledger + # stays balanced (1 reserve + 1 consume per LLM call). + # Inside an existing ``@protect`` block the contextvar is + # already populated; ``check_workflow_budget`` short-circuits + # early via the chain-mode cache when an active chain is in + # scope, so the wire-call cost is amortised across the chain. + try: + self.runtime.check_workflow_budget() + except BaseException as exc: # noqa: BLE001 — never raise out of callback + logger.debug( + "NullRunCallback.on_llm_start: check_workflow_budget " + "raised %s — proceeding without reservation (llm_call " + "cost event will be dropped by runtime._route_track)", + type(exc).__name__, + ) try: self.runtime.track_event( event_type="span_start", @@ -570,23 +608,25 @@ def on_llm_end(self, response: Any, **kwargs: Any) -> None: # fall back to the response object. This matches the # best-effort pattern used by ``_get_finish_reason`` / # ``_extract_tool_names`` for the same response. - invocation_params = kwargs.get('invocation_params') or {} + invocation_params = kwargs.get("invocation_params") or {} model = ( - invocation_params.get('model_name') + invocation_params.get("model_name") or _extract_model_from_response(response) - or 'unknown' + or "unknown" ) provider = ( - invocation_params.get('model_provider') + invocation_params.get("model_provider") or _extract_provider_from_response(response) - or 'openai' + or "openai" ) # Extract usage (normalized format) usage = extract_usage_from_response(response, provider, model) - logger.info(f"NullRun callback: model={model}, provider={provider}, " - f"usage={usage}, has_usage={usage['has_usage']}") + logger.info( + f"NullRun callback: model={model}, provider={provider}, " + f"usage={usage}, has_usage={usage['has_usage']}" + ) # Audit 2026-06-29 (unified fingerprint): derive the same # fingerprint the httpx transport computes for the same @@ -719,9 +759,7 @@ def on_llm_end(self, response: Any, **kwargs: Any) -> None: # orphan-span finding (parent_span_id points at a run_id # that no longer exists in ``_active_runs``). with self._lock: - llm_ctx = ( - self._active_runs.get(str(llm_run_id)) if llm_run_id else None - ) + llm_ctx = self._active_runs.get(str(llm_run_id)) if llm_run_id else None if llm_ctx is not None: event["trace_id"] = llm_ctx.trace_id event["span_id"] = llm_ctx.span_id @@ -778,8 +816,9 @@ def on_chain_start( logger.debug("on_chain_start without run_id — skipping span emission") return name = _extract_node_name(serialized, "chain") - self._begin_run(str(run_id), str(parent_run_id) if parent_run_id else None, - name, kind="chain") + self._begin_run( + str(run_id), str(parent_run_id) if parent_run_id else None, name, kind="chain" + ) def on_chain_end(self, outputs: Any, *, run_id: Any = None, **kwargs: Any) -> None: self._end_run(run_id) @@ -801,8 +840,9 @@ def on_tool_start( logger.debug("on_tool_start without run_id — skipping span emission") return name = _extract_node_name(serialized, "tool") - self._begin_run(str(run_id), str(parent_run_id) if parent_run_id else None, - name, kind="tool") + self._begin_run( + str(run_id), str(parent_run_id) if parent_run_id else None, name, kind="tool" + ) def on_tool_end(self, output: Any, *, run_id: Any = None, **kwargs: Any) -> None: self._end_run(run_id) @@ -822,8 +862,12 @@ def on_agent_action( if run_id is None: return tool = getattr(action, "tool", None) or "agent" - self._begin_run(str(run_id), str(parent_run_id) if parent_run_id else None, - f"agent_action:{tool}", kind="agent") + self._begin_run( + str(run_id), + str(parent_run_id) if parent_run_id else None, + f"agent_action:{tool}", + kind="agent", + ) def on_agent_finish(self, finish: Any, *, run_id: Any = None, **kwargs: Any) -> None: self._end_run(run_id) @@ -943,6 +987,7 @@ def _extract_node_name(serialized: Any, default: str) -> str: # uses, so we have a single pattern for "best-effort read from the # response object" across both helpers. + def _extract_model_from_response(response: Any) -> str | None: """Best-effort model extraction mirroring ``_get_finish_reason``. @@ -1003,12 +1048,7 @@ def _extract_model_from_response(response: Any) -> str | None: # less canonical keys (``"model_id"``, ``"modelName"`` # ``"resolved_model"``). for key, val in llm_out.items(): - if ( - isinstance(key, str) - and "model" in key.lower() - and isinstance(val, str) - and val - ): + if isinstance(key, str) and "model" in key.lower() and isinstance(val, str) and val: return val # 2. response_metadata on the response (langchain 0.x AIMessage @@ -1133,4 +1173,3 @@ def _extract_provider_from_response(response: Any) -> str | None: return str(val) return None - diff --git a/src/nullrun/runtime.py b/src/nullrun/runtime.py index dc716af..f0d2e93 100644 --- a/src/nullrun/runtime.py +++ b/src/nullrun/runtime.py @@ -83,12 +83,18 @@ ) from nullrun.breaker.exceptions import ( BreakerError, + NullRunApprovalDeniedError, + NullRunApprovalExpiredError, + NullRunApprovalReplayRejectedError, + NullRunApprovalResponseMissingError, NullRunAuthenticationError, NullRunBackendError, NullRunBlockedException, + NullRunBudgetError, NullRunError, NullRunInfrastructureError, NullRunTransportError, + NullRunWorkflowKilledError, WorkflowKilledInterrupt, WorkflowPausedException, ) @@ -176,9 +182,7 @@ def _is_production_environment(api_url: str | None = None) -> bool: """ from urllib.parse import urlparse - effective_url = api_url or os.getenv( - "NULLRUN_API_URL", "https://api.nullrun.io" - ) + effective_url = api_url or os.getenv("NULLRUN_API_URL", "https://api.nullrun.io") # Strip any trailing slash before parsing for consistent # ``hostname`` extraction. effective_url = effective_url.rstrip("/") @@ -647,7 +651,7 @@ def __init__( self._debug = debug self._transport: Transport | None = None -# Local enforcement state + # Local enforcement state # The BoundedDict-based per-workflow cost / loop / retry # counters have been removed alongside ``_check_local_limits``. # As of 0.7.0 ALL local enforcement (LoopTracker / RateTracker @@ -693,7 +697,8 @@ def __init__( # - the WS push arrives with outcome="approved" (release # the gate, resume from the same execution_id), or # - the WS push arrives with outcome="denied" (surface - # WorkflowKilledInterrupt), or + # NullRunApprovalDeniedError / NullRunWorkflowKilledError), + # or # - the per-approval timeout elapses (fall back to the # /status poll path; emit a warning so the operator # knows WS push is silent). @@ -1524,9 +1529,10 @@ def _fetch_remote_state(self, workflow_id: str) -> None: def _handle_approval_resolved(self, payload: dict[str, Any]) -> None: """WS push handler for an approval resolution. Releases - the matching gate reservation (approved) or raises - WorkflowKilledInterrupt (denied) so the agent can resume - from the same execution_id. + the matching gate reservation (approved) or surfaces the + typed denial/timeout (NullRunApprovalDeniedError on denied, + NullRunApprovalExpiredError on timeout, NR-A011/NR-A012) + so the agent can resume from the same execution_id. Args: payload: The WsMessage::ApprovalResolved dict from the @@ -1593,16 +1599,20 @@ def _wait_for_approval_resolution( a sentinel ``{"outcome": "timeout", "timed_out": True}``. **The caller is expected to fail-CLOSED on timeout** — - raise ``WorkflowKilledInterrupt``. The contract - deliberately rejects a `/status` poll fallback here: - a silent timeout must not silently approve a - privileged action. + raise :class:`NullRunApprovalExpiredError` (typed + exception, NR-A012). The contract deliberately rejects + a `/status` poll fallback here: a silent timeout must + not silently approve a privileged action. Raises: Nothing. Approval timeouts are returned, not raised, so the caller can choose the right recovery action - (raise WorkflowKilledInterrupt on denied OR on - timeout, resume on approved). + (raise NullRunApprovalDeniedError on denied, raise + NullRunApprovalExpiredError on timeout, resume on + approved). 2026-09-08: typed approval exceptions + (NR-A011, NR-A012) replaced the generic + ``WorkflowKilledInterrupt`` here so cookbook code can + catch by wire-code. """ # Per-approval timeout resolution: prefer the # server-authoritative value from the /gate response so @@ -1705,7 +1715,10 @@ def check_control_plane(self, workflow_id: str) -> None: Raises: WorkflowPausedException: If workflow is paused on server - WorkflowKilledInterrupt: If workflow is killed on server + NullRunWorkflowKilledError: If workflow is killed on + server (2026-09-08 typed signal, NR-W002; subclass + of WorkflowKilledInterrupt which remains as the + back-compat name.) """ # Prefer the explicit arg (contextvar-supplied), fall back # to the API key's bound workflow. None on legacy keys -- @@ -1744,9 +1757,14 @@ def check_control_plane(self, workflow_id: str) -> None: ) elif state_normalized == "killed": reason = remote_state.get("reason", "remote kill") - raise WorkflowKilledInterrupt( + # 2026-09-08: typed kill signal (NR-W002). Cookbook code + # can `except NullRunWorkflowKilledError` to surface the + # structured error_code + user_action. Legacy + # `except WorkflowKilledInterrupt` still matches (subclass). + raise NullRunWorkflowKilledError( workflow_id=workflow_id, reason=reason, + kill_source="remote_state", ) def check_workflow_budget(self) -> None: @@ -1756,7 +1774,8 @@ def check_workflow_budget(self) -> None: budget never gets to spend tokens. Decision → exception mapping: - "block" → WorkflowKilledInterrupt (hard policy / reservation error) + "block" → NullRunBudgetError (NR-B004, hard policy / + reservation error; 2026-09-08 typed signal) "throttle"→ WorkflowPausedException (insufficient budget, can resume) "allow" → return @@ -1798,12 +1817,7 @@ def check_workflow_budget(self) -> None: # pre-v3.53 the SDK silently honored it in any env # which made accidental prod misuse a silent fail-OPEN. if _is_production_environment(self.api_url): - allow_ack = ( - os.environ.get( - "NULLRUN_ALLOW_SKIP_BUDGET_CHECK", "" - ).strip() - == "1" - ) + allow_ack = os.environ.get("NULLRUN_ALLOW_SKIP_BUDGET_CHECK", "").strip() == "1" if not allow_ack: logger.error( "check_workflow_budget: NULLRUN_SKIP_BUDGET_CHECK=1 " @@ -1838,9 +1852,7 @@ def check_workflow_budget(self) -> None: except Exception: # noqa: BLE001 pass return - logger.debug( - "check_workflow_budget: skipped via NULLRUN_SKIP_BUDGET_CHECK=1" - ) + logger.debug("check_workflow_budget: skipped via NULLRUN_SKIP_BUDGET_CHECK=1") return # Bump the ``check_calls`` counter so the dashboard can show @@ -1963,9 +1975,7 @@ def check_workflow_budget(self) -> None: # `transport.py::execute`) and do not pass through this # pre-flight gate. Computing once per call (not cached) is # fine: compute_action_digest is ~5µs of pure stdlib. - check_req["action_digest"] = _compute_action_digest( - _BusinessImpact.no_impact() - ) + check_req["action_digest"] = _compute_action_digest(_BusinessImpact.no_impact()) # Forward the tool list so backend (T3) can match each tool # against the workflow's effective `blocked_tools` aggregate. @@ -2087,9 +2097,17 @@ def check_workflow_budget(self) -> None: # distinct from loop / retry / rate which have their # own counters. metrics.inc_runtime("cost_limit_exceeded") - raise WorkflowKilledInterrupt( + # 2026-09-08: typed hard-block (NR-B004 budget cap). + # ``NullRunBudgetError`` carries structured + # ``error_code``, ``user_action``, ``retryable`` so the + # LLM gets an actionable hint instead of "Something went + # wrong". ``reasons`` preserved in details for telemetry. + raise NullRunBudgetError( workflow_id=workflow_id, reason="; ".join(reasons), + action="block", + decision_source=response.get("decision_source"), + reasons="; ".join(reasons), ) if decision == "throttle": reasons = response.get("explanations") or ( @@ -2128,8 +2146,7 @@ def check_workflow_budget(self) -> None: # alongside "hard cap hits" via the same dashboard panel. metrics.inc_runtime("soft_overdraft_used") logger.warning( - "check_workflow_budget: soft_pass -- %s " - "(overdraft_used=%s, max=%s, remaining=%s)", + "check_workflow_budget: soft_pass -- %s (overdraft_used=%s, max=%s, remaining=%s)", explanation, overdraft_used, max_overdraft, @@ -2161,9 +2178,16 @@ def check_workflow_budget(self) -> None: logger.warning( "check_workflow_budget: require_approval decision but no approval_id in response" ) - raise WorkflowKilledInterrupt( + # 2026-09-08: typed backend error (NR-B002, retryable). + # The server returned require_approval without an + # approval_id -- this is a wire-bug / drift, not a + # budget block. Surface as retryable backend error + # so cookbook code can decide whether to fall back + # to polling /status. + raise NullRunBackendError( + message="approval_id missing in require_approval response", + endpoint="/api/v1/gate", workflow_id=workflow_id, - reason="approval_id missing in require_approval response", ) # Read the per-approval timeout from the response. Both # `approval_timeout_seconds` (i64) and @@ -2198,17 +2222,28 @@ def check_workflow_budget(self) -> None: logger.info(f"check_workflow_budget: approval {approval_id} approved -- resuming") return if outcome == "denied": - raise WorkflowKilledInterrupt( + # 2026-09-08: typed approval-denied (NR-A011). Cookbook + # code can `except NullRunApprovalDeniedError` to + # surface the denial note + user_action to the LLM. + raise NullRunApprovalDeniedError( workflow_id=workflow_id, reason=f"approval denied: {result.get('note') or 'operator denied'}", + approval_id=approval_id, + denial_note=result.get("note"), ) # timeout: fail-CLOSED -- do not run the call. - raise WorkflowKilledInterrupt( + # 2026-09-08: typed approval-expired (NR-A012) -- THE TRIGGER + # FIX. The LLM now sees "Approval expired after 300s of + # WS push silence" instead of "Something went wrong". + raise NullRunApprovalExpiredError( workflow_id=workflow_id, reason=( f"approval {approval_id} timeout: WS push silent for " f"{self._approval_timeout_seconds:.0f}s" ), + approval_id=approval_id, + timeout_seconds=self._approval_timeout_seconds, + local_timeout=True, ) # ============================================================================= @@ -2729,7 +2764,14 @@ def execute( - decision: "allow" | "block" | "flag" | "pause" | "require_approval" - decision_source: "gateway" | "cached" | "fallback" - explanation: Human-readable explanation - - policy_version: Policy version used + - policy_hash: Server-side SHA-256 of the policy applied + to this gate decision (v4 wire field; null on pre-v4 + backends). Captured via `_capture_wire_evidence` → + `set_last_gate_policy_hash` for downstream audit linkage. + NOTE: this is NOT a sequential `policy_version` number — + wire v3/v4 backends emit only `policy_hash`; legacy + `policy_version` references in this SDK are no longer + populated from the wire. - decision_context: Context used for the decision Mode values: @@ -2772,7 +2814,7 @@ def execute( "decision": "allow", "decision_source": DecisionSource.LOCAL, "explanation": "Inline mode: local enforcement only", - "policy_version": 0, + "policy_hash": None, "allow_execution": True, } @@ -2820,10 +2862,46 @@ def execute( # decorator call site). if tools is None: from nullrun.context import get_call_tools as _get_call_tools_for_execute + tools = _get_call_tools_for_execute() + + # DEFS-SDKEXEC-GATE-FIRST (2026-09-08): hoist the execution_id + # mint to REUSE the server-minted id from a prior /gate call + # (set via `_capture_server_minted_execution_id` from the + # `reservation_id` field of the /gate response). + # + # Why this matters: backend `/api/v1/execute` (the wire + # contract enforced by `backend/src/proxy/http/gate/execute.rs` + # since DEF-SDKK-022-EXEC-BYPASS, 2026-09-04, RUN_ID=20260904T1500) + # runs an existence check on `execution:{id}` in Redis at + # `execute.rs:180-208` and returns 404 EXECUTION_NOT_FOUND when + # no prior /gate minted the binding. Pre-fix this method minted + # a fresh `uuid7_str()` here — the freshly-minted id was never + # registered by /gate, so /execute fail-CLOSED with 404 on + # EVERY call and the SDK translated the 404 into a synthetic + # block ("Gateway returned 404") in + # ``transport.py::execute`` (line ~1195). + # + # Resolution: when a prior /gate captured a server-minted + # execution_id into ``_server_minted_execution_id_var``, reuse + # it. The decorator-driven ``@protect @sensitive`` path always + # runs ``check_workflow_budget()`` BEFORE ``runtime.execute()`` + # (decorators.py:538 vs :824), so the contextvar is populated + # in the common path. Direct callers of ``runtime.execute()`` + # without a prior /gate will fall through to the fresh-mint + # branch below — that's a wire-contract violation and the + # backend's 404 is the correct fail-CLOSED response. + from nullrun.context import get_server_minted_execution_id + + prior_execution_id = get_server_minted_execution_id() + if prior_execution_id is not None: + execution_id = prior_execution_id + else: + execution_id = uuid7_str() + execute_kwargs: dict[str, Any] = { "organization_id": organization_id, - "execution_id": uuid7_str(), + "execution_id": execution_id, "trace_id": trace_id, "tool": tool_name, "input_data": input_data, @@ -2854,11 +2932,15 @@ def execute( approval_id = result.get("approval_id") or "" if not approval_id: metrics.inc_runtime("execute_blocked") - raise NullRunBlockedException( + # 2026-09-08: typed wire-bug (NR-A004). The server + # returned require_approval without an approval_id — + # this is a wire-contract bug, NOT a transient failure. + # Cookbook code catches this and reports to NULLRUN + # support; do NOT retry. + raise NullRunApprovalResponseMissingError( workflow_id=workflow_id or UNKNOWN_WORKFLOW_ID, reason="approval_id missing in require_approval response", tool_name=tool_name, - error_code="NR-A004", ) server_timeout = _validate_approval_timeout( @@ -2875,16 +2957,38 @@ def execute( outcome = str(approval_result.get("outcome") or "").lower() if outcome != "approved": metrics.inc_runtime("execute_blocked") - reason = ( - f"approval denied: {approval_result.get('note') or 'operator denied'}" - if outcome == "denied" - else f"approval {approval_id} timeout" - ) + # 2026-09-08: dispatch typed approval exception by + # outcome so cookbook code can react per wire-code: + # denied → NR-A011 (NullRunApprovalDeniedError) + # timeout → NR-A012 (NullRunApprovalExpiredError) + # other → NR-X001 (NullRunBlockedException, generic) + if outcome == "denied": + raise NullRunApprovalDeniedError( + workflow_id=workflow_id or UNKNOWN_WORKFLOW_ID, + reason=( + f"approval denied: " + f"{approval_result.get('note') or 'operator denied'}" + ), + tool_name=tool_name, + approval_id=approval_id, + denial_note=approval_result.get("note"), + ) + if outcome == "timeout": + raise NullRunApprovalExpiredError( + workflow_id=workflow_id or UNKNOWN_WORKFLOW_ID, + reason=f"approval {approval_id} timeout", + tool_name=tool_name, + approval_id=approval_id, + timeout_seconds=server_timeout, + local_timeout=True, + ) + # Unknown outcome (cancelled / superseded / wire drift): + # fall back to generic block so the LLM still sees a + # typed exception with error_code, never a bare string. raise NullRunBlockedException( workflow_id=workflow_id or UNKNOWN_WORKFLOW_ID, - reason=reason, + reason=f"approval {approval_id} unresolved: outcome={outcome!r}", tool_name=tool_name, - error_code="NR-A004", ) # Re-check the same action. The backend must verify that @@ -2895,11 +2999,17 @@ def execute( result = self._transport.execute(**execute_kwargs) if result.get("decision") == "require_approval": metrics.inc_runtime("execute_blocked") - raise NullRunBlockedException( + # 2026-09-08: typed replay-rejection (NR-A015). The + # operator approved but the same approval_id was + # already consumed by a concurrent /execute (race). + # Cookbook pattern: do NOT retry the same approval_id; + # treat as idempotency violation (likely a client + # retry loop). + raise NullRunApprovalReplayRejectedError( workflow_id=workflow_id or UNKNOWN_WORKFLOW_ID, reason="approved action was not accepted on re-check", tool_name=tool_name, - error_code="NR-A004", + approval_id=approval_id, ) # Check if execution is allowed @@ -3066,9 +3176,7 @@ def _enrich_event(self, event: dict[str, Any]) -> dict[str, Any]: # 2026-08-06 (DEF-SDKWRAP-CHAIN-SOFT-EXECUTION-ID-REUSE-01, span_id = enriched.get("span_id") if span_id and ":" not in idem_key: - enriched["idempotency_key"] = ( - f"{idem_key}:{str(span_id)[:16]}" - ) + enriched["idempotency_key"] = f"{idem_key}:{str(span_id)[:16]}" else: enriched["idempotency_key"] = idem_key @@ -3516,11 +3624,7 @@ def _capture_server_minted_execution_id(response: dict[str, Any]) -> str | None: from nullrun.context import get_operation_id as _get_op_id_for_capture sdk_op_id = _get_op_id_for_capture() - server_op_id = ( - response.get("operation_id") - if isinstance(response, dict) - else None - ) + server_op_id = response.get("operation_id") if isinstance(response, dict) else None if isinstance(server_op_id, str) and server_op_id: # Defensive parity assertion: server MUST echo the same # operation_id the SDK sent. A mismatch indicates either @@ -3608,8 +3712,7 @@ def _safe_str(key: str) -> str | None: return None if not isinstance(v, str): logger.warning( - "_capture_wire_evidence: response.%s is %s, " - "expected str — dropping", + "_capture_wire_evidence: response.%s is %s, expected str — dropping", key, type(v).__name__, ) diff --git a/src/nullrun/transport.py b/src/nullrun/transport.py index f9f3b28..00027c5 100644 --- a/src/nullrun/transport.py +++ b/src/nullrun/transport.py @@ -300,7 +300,12 @@ def _retry_with_backoff( return result - except (BreakerTransportError, NullRunAuthenticationError, NullRunTransportError, NullRunBackendError): + except ( + BreakerTransportError, + NullRunAuthenticationError, + NullRunTransportError, + NullRunBackendError, + ): raise except httpx.HTTPStatusError as exc: @@ -1119,9 +1124,27 @@ def execute( ) -> dict[str, Any]: """Pre-execution policy evaluation via /api/v1/execute (PRIMARY enforcement point). - The SDK MUST call /api/v1/execute (which checks the ``execute`` scope on - the API key) rather than /api/v1/gate (advisory, no scope check). - /api/v1/gate is reserved for budget pre-flight via ``Transport.check``. + Wire contract (revised 2026-09-08, DEFS-SDKEXEC-GATE-FIRST): + /execute REQUIRES a prior /gate call that minted the same + ``execution_id`` and registered the ``execution:{id}`` binding + in Redis. Backend enforcement: + ``backend/src/proxy/http/gate/execute.rs:46-208`` + (DEF-SDKK-022-EXEC-BYPASS, 2026-09-04, RUN_ID=20260904T1500) + runs ``HGET execution:{id} ORG_FIELD`` on entry; a miss + returns 404 EXECUTION_NOT_FOUND (fail-CLOSED). The SDK + therefore MUST thread the execution_id captured by + ``runtime.check_workflow_budget`` (which calls ``Transport.check``, + i.e. /gate) into the body of this /execute call. See + ``runtime.execute()`` (line ~2820) for the reuse path; this + method's caller is the single source of truth for + ``execution_id`` selection. + + Prior to DEF-SDKK-022 the comment here claimed "/execute MUST + be called rather than /gate" — that contract was the legacy + pre-2026-09-04 shape. The post-fix shape is "/execute MUST be + preceded by /gate for the same execution_id" — the budget + pre-flight (Transport.check, /api/v1/gate) is the binding + registrar; /execute is the policy decision that re-uses it. Args: organization_id: Organization identifier @@ -1143,7 +1166,12 @@ def execute( - decision: "allow" | "block" | "flag" | "pause" | "require_approval" - decision_source: "gateway" | "cached" | "fallback" - explanation: Human-readable explanation - - policy_version: Policy version used + - policy_hash: Server-side SHA-256 of the policy applied + (v4 wire field; null on pre-v4 backends). NOT a + sequential `policy_version` number — wire v3/v4 backends + emit only `policy_hash`. Synthetic fallback dicts ship + `policy_version: 0` for legacy compatibility; real + responses populate `policy_hash` only. - decision_context: Context for replay (if available) """ gate_request = { @@ -1198,7 +1226,7 @@ def do_execute_request() -> httpx.Response: "decision": "block", "decision_source": DecisionSource.FALLBACK, "explanation": f"Gateway returned {response.status_code}", - "policy_version": 0, + "policy_hash": None, } except BreakerTransportError as exc: @@ -1216,14 +1244,14 @@ def do_execute_request() -> httpx.Response: "decision": "allow", "decision_source": TransportErrorSource.NETWORK_ERROR, "explanation": f"Gateway unreachable: {exc}", - "policy_version": 0, + "policy_hash": None, } if on_transport_error == "closed": return { "decision": "block", "decision_source": TransportErrorSource.NETWORK_ERROR, "explanation": f"Gateway unreachable: {exc}", - "policy_version": 0, + "policy_hash": None, } pass # fall through to fallback mode except NullRunTransportError: @@ -1693,8 +1721,17 @@ def track_single( ``idempotency_key``. Returns: - Parsed JSON dict with at least - ``{"status": "ok"|"idempotent_replay",...}``. + Parsed JSON dict from the backend's TrackResponse. + NOTE: there is NO top-level ``status`` field on the + wire — the legacy pre-v3 docstring claimed one, but + v3/v4 backends emit + ``{snapshot, actions_taken, processing_mode, + cost_source, confidence, event_id, + idempotent_replay, stored_response?}``. SDK callers + branch on the HTTP status (200 vs 4xx/5xx) and on + ``idempotent_replay`` (bool) for replay detection — + do NOT read ``data["status"]`` (KeyError on every + backend >= 3.66.2). Raises: NullRunConsumeOverbudgetError: 422 CONSUME_OVERBUDGET — @@ -1760,8 +1797,14 @@ def cancel( cancellation (audit trail). Returns: - Parsed JSON dict (typically ``{"status": "ok" - "execution_id":..., "cancelled_at": ts}``). + Parsed JSON dict from the backend's CancelResponse. + NOTE: there is NO top-level ``status`` field on the + wire — the legacy pre-v3 docstring claimed one. + v3/v4 backends emit + ``{execution_id, canceled_at, reservation_released_cents, + already_canceled}``. SDK callers branch on the HTTP + status only — do NOT read ``data["status"]`` + (KeyError on every backend >= 3.66.2). """ request: dict[str, Any] = {"execution_id": execution_id} if reason: @@ -2151,10 +2194,7 @@ def audit_export_status( Raises: NullRunBackendError / NullRunAuthenticationError. """ - url = ( - f"{self.api_url}/api/v1/orgs/{organization_id}" - f"/audit-log/export/{job_id}/status" - ) + url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log/export/{job_id}/status" headers = self._auth_headers_for_get() try: response = self._client.get(url, headers=headers, timeout=10.0) @@ -2767,6 +2807,35 @@ def _build_v3_error_code_map() -> dict[str, type[Exception]]: # ``RATE_LIMIT_REDIS_UNAVAILABLE`` -> ``NullRunRateLimitRedisError`` # family pattern at wire level). "IDEMPOTENCY_REDIS_UNAVAILABLE": NullRunBackendError, + # Execution Graph / ADR-036 (sub-agent spawn topology). Backend + # error_codes.rs:107-382 covers six codes in this family — three + # 422 semantic rejects (cycle / depth / parent-binding) and three + # 503 infrastructure failures (depth lookup / invoke persist / + # subworkflow disabled). Map to ``NullRunChainError`` because + # the existing class already carries `parent_execution_id` per + # Execution Graph v0 docstring at `exceptions.py:388-410`. Adding + # them under a fresh ``NullRunSubworkflowError`` would force + # cookbook code to import a new exception class for the same + # lineage concept; consolidate under ChainError instead. + "WORKFLOW_CYCLE_DETECTED": NullRunChainError, + "WORKFLOW_DEPTH_EXCEEDED": NullRunChainError, + "WORKFLOW_PARENT_BINDING_EXPIRED": NullRunChainError, + "WORKFLOW_DEPTH_LOOKUP_FAILED": NullRunChainError, + "INVOKE_PERSIST_FAILED": NullRunBackendError, + "SUBWORKFLOW_INVOKE_DISABLED": NullRunChainError, + # ADR-023 (post-approval re-check race): a second operator + # already decided on the same approval row before this call's + # re-check landed. Map to ``NullRunApprovalReplayRejectedError`` + # because semantically the agent caller has the same retry-loop + # concern as a replay-rejected approval (CLAUDE.md §34c). + "APPROVAL_ALREADY_DECIDED": NullRunApprovalReplayRejectedError, + # ADR-023 (Phase-1+ wire-shape fail-CLOSED): a v3+ SDK hit /gate + # without ``action_digest`` (legacy anchor attempt). Map to + # ``NullRunBlockedException`` because the wire shape is a true + # block decision, not an infrastructure error — cookbook code + # branches on the action_digest missing path with the same + # `except NullRunBlockedException:` flow as TOOL_BLOCKED. + "LEGACY_GRANT_REJECTED": NullRunBlockedException, } diff --git a/tests/test_2026_09_08_gate_first_execute.py b/tests/test_2026_09_08_gate_first_execute.py new file mode 100644 index 0000000..18dcdd3 --- /dev/null +++ b/tests/test_2026_09_08_gate_first_execute.py @@ -0,0 +1,296 @@ +"""DEFS-SDKEXEC-GATE-FIRST (2026-09-08) — /execute must reuse /gate's execution_id. + +Pre-fix (per audit 2026-09-08): + - `runtime.execute()` minted a fresh `uuid7_str()` for the wire body. + - Backend `/api/v1/execute` (`backend/src/proxy/http/gate/execute.rs:46-208`, + DEF-SDKK-022-EXEC-BYPASS, 2026-09-04, RUN_ID=20260904T1500) requires the + request's `execution_id` to have a live `execution:{id}` ownership + binding in Redis (HGET ORG_FIELD). Without a prior /gate that registered + the binding, /execute returned 404 EXECUTION_NOT_FOUND and the SDK + translated the 404 into a synthetic block ("Gateway returned 404"). + - User-visible symptom: every `@protect @sensitive` call from + `langgraph_openai_approval_demo.py` (and similar flows) returned + `Workflow __nullrun_unknown__ blocked: Gateway returned 404 (action=block, + tool=, status_code=None, details=)`. + +Post-fix: + - `runtime.execute()` reads `_server_minted_execution_id_var` (set by + `_capture_server_minted_execution_id` from the /gate response's + `reservation_id` field) and reuses it. Only mint a fresh uuid7 when + the contextvar is empty (direct callers without a prior /gate). + - `_enforce_sensitive_tool` displays the API key's bound workflow + (resolved via `runtime._resolve_workflow_id`) instead of the literal + `__nullrun_unknown__` sentinel when the user did not open an explicit + `with workflow(...)` block. The wire still carries the same workflow + (server-side binding); only the displayed label changes. + +These tests pin the post-fix shape so a future refactor that re-introduces +a fresh-mint in `execute()` (or restores the sentinel-first display) +fails the test. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +import pytest + +from nullrun.context import ( + clear_server_minted_execution_id, + set_server_minted_execution_id, +) + +SDK_ROOT = Path(__file__).resolve().parent.parent +RUNTIME_PY = SDK_ROOT / "src" / "nullrun" / "runtime.py" +DECORATORS_PY = SDK_ROOT / "src" / "nullrun" / "decorators.py" +TRANSPORT_PY = SDK_ROOT / "src" / "nullrun" / "transport.py" +LANGGRAPH_INSTR_PY = SDK_ROOT / "src" / "nullrun" / "instrumentation" / "langgraph.py" + + +def _read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +@pytest.fixture(autouse=True) +def _reset_server_minted(): + """Reset the contextvar before AND after each test so leakage + between tests doesn't masquerade as a hoist pass.""" + clear_server_minted_execution_id() + yield + clear_server_minted_execution_id() + + +class TestExecuteReusesGateExecutionId: + """Pin `runtime.execute()` so a future refactor that re-mints + a fresh `uuid7_str()` regardless of /gate context fails the test.""" + + def _execute_body(self) -> str: + runtime = _read(RUNTIME_PY) + # Match the second `def execute(` (the public enforcement + # entry point), not `runtime._execute` or `Transport.execute`. + m = re.search( + r" def execute\(\s*self,\s*tool_name: str,.*?\)\s*->\s*" + r"dict\[str, Any\]:.*?(?=\n def |\nclass |\Z)", + runtime, + re.DOTALL, + ) + assert m, "could not locate runtime.execute method body" + return m.group(0) + + def test_execute_reads_server_minted_contextvar(self): + body = self._execute_body() + assert "get_server_minted_execution_id()" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: runtime.execute() must read the " + "server-minted execution_id from the contextvar (set by " + "check_workflow_budget's /gate round-trip) before minting " + "a fresh uuid7. Pre-fix the body unconditionally minted " + "uuid7_str(), so /execute's execution_id never matched " + "the binding /gate registered and the backend returned " + "404 EXECUTION_NOT_FOUND." + ) + + def test_execute_reuses_captured_id_when_present(self): + body = self._execute_body() + # The hoist pattern: read contextvar, fall back to uuid7_str() + # only when the contextvar is None. + assert "prior_execution_id = get_server_minted_execution_id()" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: runtime.execute() must alias the " + "contextvar read into a local so the same value flows into " + "the wire body." + ) + assert "if prior_execution_id is not None:" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: when the contextvar is populated, " + "execute() must reuse it directly — no fresh uuid7 mint." + ) + assert "execution_id = prior_execution_id" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: the reused execution_id must be " + "threaded into the wire body under the `execution_id` key." + ) + + def test_execute_falls_back_to_uuid7_only_when_contextvar_empty(self): + body = self._execute_body() + # Locate the fallback block — must live INSIDE an `if ... is None:` arm. + fallback_block = re.search( + r"if prior_execution_id is not None:\s*\n\s*execution_id = " + r"prior_execution_id\s*\n\s*else:\s*\n\s*execution_id = " + r"uuid7_str\(\)", + body, + ) + assert fallback_block, ( + "DEFS-SDKEXEC-GATE-FIRST: the uuid7_str() mint must live " + "INSIDE the `else:` arm of the `if prior_execution_id is " + "not None:` check. Pre-fix an unconditional " + "`execution_id = uuid7_str()` line at this site minted " + "every time, breaking the /gate ↔ /execute binding." + ) + body_without_fallback = body.replace(fallback_block.group(0), "") + # Defensive: the wire body MUST consume the resolved + # `execution_id` (the one with the prior_id fallback applied). + assert '"execution_id": execution_id' in body, ( + "DEFS-SDKEXEC-GATE-FIRST: the wire body must consume the " + "resolved `execution_id` variable (not a freshly-minted " + "uuid7 inline)." + ) + assert 'execution_id": uuid7_str()' not in body_without_fallback, ( + "DEFS-SDKEXEC-GATE-FIRST: a top-level `execution_id = " + "uuid7_str()` (outside the fallback arm) must not survive. " + "A pre-fix leftover would silently bypass the /gate reuse." + ) + + def test_execute_carries_comment_explaining_drift(self): + body = self._execute_body() + # The fix introduced a long comment naming DEF-SDKK-022 + + # DEFS-SDKEXEC-GATE-FIRST. Pin so a future maintainer who + # deletes the comment is forced to read the code's history. + assert "DEFS-SDKEXEC-GATE-FIRST" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: the explainer comment block must " + "name the fix tag so future readers can grep for it." + ) + assert "DEF-SDKK-022-EXEC-BYPASS" in body, ( + "DEFS-SDKEXEC-GATE-FIRST: the explainer must reference the " + "backend fix (DEF-SDKK-022-EXEC-BYPASS) that introduced the " + "/execute existence check, so readers see the round-trip " + "contract without searching." + ) + + +class TestDecoratorWorkflowLabelUsesRuntimeBinding: + """Pin `_enforce_sensitive_tool` so the displayed workflow_id + label shows the API key's bound workflow when no `with workflow(...)` + block is active (instead of the literal `__nullrun_unknown__` sentinel).""" + + def _enforce_body(self) -> str: + decorators = _read(DECORATORS_PY) + m = re.search( + r"def _enforce_sensitive_tool\(.*?\).*?(?=\ndef |\nclass |\Z)", + decorators, + re.DOTALL, + ) + assert m, "could not locate _enforce_sensitive_tool method body" + return m.group(0) + + def test_enforce_resolves_via_runtime_bound_workflow(self): + body = self._enforce_body() + # Two sites in the function (extract failure path + main path). + occurrences = body.count( + "runtime._resolve_workflow_id(get_workflow_id()) or UNKNOWN_WORKFLOW_ID" + ) + assert occurrences >= 2, ( + f"DEFS-SDKEXEC-WORKFLOW-LABEL: _enforce_sensitive_tool must " + f"prefer the runtime's bound workflow via " + f"runtime._resolve_workflow_id(...) at both display sites " + f"(extract failure + main path). Found {occurrences} " + f"occurrences; expected >= 2." + ) + + def test_enforce_does_not_use_contextvar_only_fallback(self): + body = self._enforce_body() + # Pre-fix: `workflow_id = get_workflow_id() or UNKNOWN_WORKFLOW_ID` + # (contextvar-only). Post-fix: that literal pattern must not + # survive at the top-level assignment site. + # + # We allow the literal only as a substring INSIDE the longer + # `runtime._resolve_workflow_id(...)` call (which is what we + # want). Strip those out first, then check the residue. + resolved_call = "runtime._resolve_workflow_id(get_workflow_id()) or UNKNOWN_WORKFLOW_ID" + body_without_resolved = body.replace(resolved_call, "") + assert "workflow_id = get_workflow_id() or UNKNOWN_WORKFLOW_ID" not in ( + body_without_resolved + ), ( + "DEFS-SDKEXEC-WORKFLOW-LABEL: pre-fix contextvar-only " + "fallback `workflow_id = get_workflow_id() or " + "UNKNOWN_WORKFLOW_ID` must be replaced by the runtime-aware " + "resolver everywhere. The pre-fix pattern displayed " + "`__nullrun_unknown__` for every API-key-bound key." + ) + + +class TestTransportCommentReflectsPostFixContract: + """Pin the `Transport.execute` docstring so the legacy + pre-2026-09-04 contract (`/execute MUST be called rather than + /gate`) doesn't drift back into the source.""" + + def test_transport_execute_docstring_references_post_fix_contract(self): + transport = _read(TRANSPORT_PY) + m = re.search( + r"def execute\(\s*self,.*?\)\s*->\s*dict\[str, Any\]:.*?(?=\n def |\nclass |\Z)", + transport, + re.DOTALL, + ) + assert m, "could not locate Transport.execute method body" + body = m.group(0) + assert "DEFS-SDKEXEC-GATE-FIRST" in body, ( + "transport.py: Transport.execute docstring must name the " + "post-fix tag so the contract is grep-able." + ) + assert "DEF-SDKK-022-EXEC-BYPASS" in body, ( + "transport.py: Transport.execute docstring must reference " + "the backend fix that introduced the existence check." + ) + # The legacy misleading claim must be gone (or explicitly + # marked as pre-fix). + assert ( + "MUST call /api/v1/execute (which checks the ``execute`` " + "scope on the API key) rather than /api/v1/gate" + ) not in body, ( + "transport.py: pre-fix misleading claim that /execute MUST " + "be called rather than /gate must be removed — that contract " + "was the legacy pre-2026-09-04 shape and was the root " + "cause of the 404 EXECUTION_NOT_FOUND drift." + ) + + +class TestLanggraphCallbackPairsLlmSpanWithReservation: + """Pin `NullRunCallback.on_llm_start` so the LLM span /track + pairing path stays alive (check_workflow_budget is fire-and-forget + but the call site must survive).""" + + def test_on_llm_start_calls_check_workflow_budget(self): + instr = _read(LANGGRAPH_INSTR_PY) + m = re.search( + r"def on_llm_start\(self,.*?\)\s*->\s*None:.*?(?=\n def |\nclass |\Z)", + instr, + re.DOTALL, + ) + assert m, "could not locate NullRunCallback.on_llm_start" + body = m.group(0) + assert "self.runtime.check_workflow_budget()" in body, ( + "DEFS-SDKEXEC-LLM-RESERVATION: on_llm_start must call " + "runtime.check_workflow_budget() to pair the LLM span " + "with a server-minted reservation_id. Without this the " + "matching on_llm_end llm_call cost event is silently " + "dropped by runtime._route_track (no reservation_id in " + "scope)." + ) + assert "DEFS-SDKEXEC-LLM-RESERVATION" in body, ( + "DEFS-SDKEXEC-LLM-RESERVATION: the explainer comment block " + "must name the fix tag so future readers can grep." + ) + # Defensive: the call must be guarded so a backend outage + # never breaks the LangChain callback chain. + assert "except BaseException" in body, ( + "DEFS-SDKEXEC-LLM-RESERVATION: the check_workflow_budget " + "call must be wrapped in a never-raise guard so a " + "WorkflowKilledInterrupt / WorkflowPausedException / " + "transport error does not break the LangChain callback " + "contract (callbacks must never raise)." + ) + + +class TestServerMintedExecutionIdContract: + """Drive the contextvar to confirm the round-trip shape used by + `runtime.execute()` works as advertised.""" + + def test_set_then_get_round_trips(self): + from nullrun.context import ( + get_server_minted_execution_id, + reset_server_minted_execution_id, + ) + + sentinel = "01936f8e-1234-7abc-9def-0123456789ab" + token = set_server_minted_execution_id(sentinel) + try: + assert get_server_minted_execution_id() == sentinel + finally: + reset_server_minted_execution_id(token) diff --git a/tests/test_actions.py b/tests/test_actions.py index 2441f95..396d815 100644 --- a/tests/test_actions.py +++ b/tests/test_actions.py @@ -841,18 +841,42 @@ def test_workflow_killed_interrupt_does_not_emit_warning(): assert not any(issubclass(item.category, DeprecationWarning) for item in w) -def test_workflow_killed_interrupt_is_base_exception(): - """``except Exception`` does NOT catch the kill signal.""" - with pytest.raises(WorkflowKilledInterrupt): - try: - raise WorkflowKilledInterrupt(workflow_id="wf-1", reason="x") - except Exception: - pytest.fail("Exception should not catch WorkflowKilledInterrupt") +def test_workflow_killed_interrupt_is_catchable_by_exception(): + """2026-09-08 migration reversal: ``except Exception`` DOES catch + the kill signal — cookbook code can react with typed error_code + + user_action. The pre-migration contract (BaseException bypass) + is intentionally broken because agent recovery requires + catchable kill signals. + """ + caught: list[Exception] = [] + + try: + raise WorkflowKilledInterrupt(workflow_id="wf-1", reason="x") + except Exception as exc: + caught.append(exc) + assert len(caught) == 1, "Exception should catch WorkflowKilledInterrupt (post-migration)" + assert isinstance(caught[0], WorkflowKilledInterrupt) + assert caught[0].error_code == "NR-W002" -def test_workflow_killed_exception_is_caught_by_except_killed_exception(): - """Legacy ``except WorkflowKilledException`` still catches the new - interrupt (back-compat contract). + +def test_workflow_killed_interrupt_not_caught_by_except_killed_exception(): + """2026-09-08 BREAK: legacy ``except WorkflowKilledException`` + no longer catches the new interrupt (WorkflowKilledInterrupt is + no longer a BaseException subclass). Cookbook code must migrate + to ``except WorkflowKilledInterrupt`` (canonical) or + ``except NullRunWorkflowKilledError`` (preferred typed name). """ - with pytest.raises(WorkflowKilledException): + raised = False + try: raise WorkflowKilledInterrupt(workflow_id="wf-1", reason="x") + except WorkflowKilledException: + pytest.fail( + "except WorkflowKilledException should NOT catch the new " + "interrupt (2026-09-08 BREAK — migrate to except " + "WorkflowKilledInterrupt or except NullRunWorkflowKilledError)" + ) + except WorkflowKilledInterrupt: + raised = True + + assert raised, "the new interrupt should propagate through except WorkflowKilledException" diff --git a/tests/test_decision_split.py b/tests/test_decision_split.py index 11600a7..c6b44a2 100644 --- a/tests/test_decision_split.py +++ b/tests/test_decision_split.py @@ -72,14 +72,30 @@ def test_decision_and_infrastructure_are_disjoint(): def test_workflow_killed_interrupt_is_neither_decision_nor_infrastructure(): - """The kill signal is a BaseException — it deliberately bypasses - ``except Exception:`` so careless handlers can't swallow operator - kills. It must NOT inherit from NullRunDecision (which would make - it catchable by `except Exception:` via the NullRunError branch).""" - assert not issubclass(exc.WorkflowKilledInterrupt, exc.NullRunError) + """2026-09-08 migration reversal: WorkflowKilledInterrupt IS now a + NullRunError subclass (Exception subclass, NR-W002) — formerly + a BaseException subclass that bypassed ``except Exception:``. + The user override: agent recovery needs catchable kill signals + to surface the structured error_code + user_action. + + Hierarchy after migration: + WorkflowKilledInterrupt → NullRunError → BreakerError → Exception → BaseException + + Kill is intentionally NOT a NullRunDecision (the structured- + decision branch is reserved for gate-decision failures: + budget/tool/approval). Kill is a control-plane signal (operator + or circuit-breaker), semantically distinct from a decision — + the new MRO reflects this. + """ + assert issubclass(exc.WorkflowKilledInterrupt, exc.NullRunError) + # Kill is NOT a NullRunDecision (decision failures are budget/ + # tool/approval; kill is a control-plane signal). assert not issubclass(exc.WorkflowKilledInterrupt, exc.NullRunDecision) + # And NOT a NullRunInfrastructureError (operator action is not + # a transport failure). assert not issubclass(exc.WorkflowKilledInterrupt, exc.NullRunInfrastructureError) - # But it IS a BaseException, which is the whole point. + # And it's still a BaseException (transitively, since it is now + # an Exception subclass which is-a BaseException). assert issubclass(exc.WorkflowKilledInterrupt, BaseException) diff --git a/tests/test_exception_hierarchy.py b/tests/test_exception_hierarchy.py index 28e0a33..b73d085 100644 --- a/tests/test_exception_hierarchy.py +++ b/tests/test_exception_hierarchy.py @@ -76,16 +76,24 @@ def test_all_exceptions_inherit_from_nullrun_error(self): ) def test_killed_interrupt_does_not_inherit_from_exception(self): - # WorkflowKilledInterrupt is a BaseException subclass by design - # (docs/kill-contract.md). It MUST NOT inherit from - # NullRunError (which is an Exception subclass), so that - # `except Exception` does not catch the kill signal. - assert not issubclass(WorkflowKilledInterrupt, Exception) - assert not issubclass(WorkflowKilledInterrupt, NullRunError) - # But it MUST inherit from WorkflowKilledException (legacy - # back-compat shim) so old `except WorkflowKilledException` - # clauses still match. - assert issubclass(WorkflowKilledInterrupt, WorkflowKilledException) + # 2026-09-08 migration: WorkflowKilledInterrupt is now an + # Exception subclass (``NullRunError`` parent) — formerly a + # BaseException subclass. The user override: agent recovery + # code needs to catch the kill signal via ``except + # WorkflowKilledInterrupt`` / ``except NullRunWorkflowKilledError`` + # and surface the structured error_code + user_action. + # + # Pinning this in test is a regression guard: a future + # maintainer reverting to BaseException to "preserve the + # kill contract" would break cookbook recovery and this + # test would fail loudly, forcing them to either keep the + # migration or justify the revert in a comment. + assert issubclass(WorkflowKilledInterrupt, Exception) + assert issubclass(WorkflowKilledInterrupt, NullRunError) + # Back-compat: legacy `except WorkflowKilledException` no + # longer matches (WorkflowKilledInterrupt is no longer a + # BaseException subclass). This is the documented BREAK. + assert not issubclass(WorkflowKilledInterrupt, WorkflowKilledException) # --------------------------------------------------------------------------- @@ -152,18 +160,29 @@ def test_backend_error_caught_by_transport_error(self): raise NullRunBackendError("5xx", endpoint="/api/v1/check", status_code=503) def test_killed_interrupt_caught_by_killed_exception(self): - # Back-compat shim — legacy `except WorkflowKilledException` - # must still match the new interrupt subclass. + # 2026-09-08 migration: WorkflowKilledException (the + # deprecated BaseException parent) no longer matches the + # new Exception subclass. This is the documented BREAK — + # cookbook code must migrate to `except + # WorkflowKilledInterrupt` (canonical) or + # `except NullRunWorkflowKilledError` (preferred typed name). with pytest.raises(WorkflowKilledException): - raise WorkflowKilledInterrupt("wf-1", reason="killed via API") + # WorkflowKilledException is itself a BaseException + # subclass, so this raises WorkflowKilledException + # directly (which is still BaseException). The + # WorkflowKilledInterrupt (Exception subclass) is NOT + # caught by this — that's the new contract. + raise WorkflowKilledException("wf-1", reason="killed via API") def test_killed_interrupt_not_caught_by_exception(self): - # The whole point of BaseException inheritance: kill must - # not be swallowable by `except Exception`. - with pytest.raises(BaseException) as exc_info: + # 2026-09-08 migration REVERSAL: WorkflowKilledInterrupt is + # now an Exception subclass — it IS catchable by + # `except Exception`. This is the new contract (cookbook + # recovery needs typed error_code + user_action). + with pytest.raises(Exception) as exc_info: raise WorkflowKilledInterrupt("wf-1", reason="killed") assert isinstance(exc_info.value, WorkflowKilledInterrupt) - assert not isinstance(exc_info.value, Exception) + assert isinstance(exc_info.value, Exception) # --------------------------------------------------------------------------- diff --git a/tests/test_gate_real_path.py b/tests/test_gate_real_path.py index b73e794..9d02e7e 100644 --- a/tests/test_gate_real_path.py +++ b/tests/test_gate_real_path.py @@ -22,9 +22,10 @@ 4. SDK does NOT send `model="budget-precheck"` anywhere. 5. The runtime's pre-flight (`check_workflow_budget`) does NOT raise on a real `decision="allow"` response. - 6. The runtime's pre-flight DOES raise `WorkflowKilledInterrupt` - on a real `decision="block"` response (so the fix didn't - accidentally remove the real-block path). + 6. The runtime's pre-flight DOES raise a typed block exception + (NullRunBudgetError, NR-B004 — was WorkflowKilledInterrupt + pre-2026-09-08) on a real `decision="block"` response (so + the fix didn't accidentally remove the real-block path). """ from __future__ import annotations @@ -36,7 +37,7 @@ import respx import nullrun -from nullrun.breaker.exceptions import WorkflowKilledInterrupt +from nullrun.breaker.exceptions import NullRunBudgetError BASE_URL = "https://api.test.nullrun.io" GATE_URL = f"{BASE_URL}/api/v1/gate" @@ -93,7 +94,9 @@ def test_default_request_allows_clean_workflow( def test_real_block_still_honored(self, make_runtime, mock_api): """T1 must NOT have accidentally removed the real-block path. Backend returning decision=block (with a real reason, NOT a - FALLBACK_* synthetic) must still raise WorkflowKilledInterrupt. + FALLBACK_* synthetic) must still raise a typed block + exception (NullRunBudgetError, NR-B004 — was + WorkflowKilledInterrupt pre-2026-09-08). """ respx.post(GATE_URL).mock( return_value=httpx.Response( @@ -108,7 +111,7 @@ def test_real_block_still_honored(self, make_runtime, mock_api): ) ) rt = make_runtime() - with pytest.raises(WorkflowKilledInterrupt) as exc_info: + with pytest.raises(NullRunBudgetError) as exc_info: rt.check_workflow_budget() assert "Budget exhausted" in exc_info.value.reason diff --git a/tests/test_typed_exceptions_full_audit.py b/tests/test_typed_exceptions_full_audit.py new file mode 100644 index 0000000..bca7c54 --- /dev/null +++ b/tests/test_typed_exceptions_full_audit.py @@ -0,0 +1,418 @@ +"""Full-audit tests for the 2026-09-08 typed exception migration. + +Context (the trigger): + The user reported that the LangGraph approval demo ended with + "Something went wrong. Please try again." instead of an + actionable "Approval expired after 300s". Root cause was that + SDK raised ``WorkflowKilledInterrupt`` (a ``BaseException`` + subclass) on the approval-timeout path, losing the structured + ``error_code`` / ``user_action`` / ``retryable`` fields. + +User override: + - Full audit of EVERY ``WorkflowKilledInterrupt`` raise site + (8 sites across runtime.py, instrumentation/auto.py, + actions.py) — every one converted to a typed exception. + - ``WorkflowKilledInterrupt`` migrated from ``BaseException`` + to ``Exception`` subclass so cookbook code can do + ``except NullRunWorkflowKilledError`` and surface the + structured error to the user. + +This file pins that contract: + + - Tests 1-7: per-raise-site conversion (typed exception raised + with the right error_code + user_action + structured fields). + - Tests 8-11: inline NR-A004 conversion in runtime.execute(). + - Tests 12-14: back-compat regression pins. + - Test 15: end-to-end UX pin (the langgraph tool-error path + surfaces the structured error to the LLM). +""" + +from __future__ import annotations + +import pytest + +from nullrun.breaker.exceptions import ( + NullRunApprovalDeniedError, + NullRunApprovalExpiredError, + NullRunApprovalReplayRejectedError, + NullRunApprovalResponseMissingError, + NullRunBackendError, + NullRunBlockedException, + NullRunBudgetError, + NullRunWorkflowKilledError, + WorkflowKilledException, + WorkflowKilledInterrupt, +) + +# --------------------------------------------------------------------------- +# Per-raise-site conversion (8 raises converted from WorkflowKilledInterrupt +# to typed exceptions + 3 inline NR-A004 raises) +# --------------------------------------------------------------------------- + + +class TestRemoteKillRaisesTyped: + """runtime.py:1745 — WS push ``state == "killed"`` → typed.""" + + def test_remote_kill_raises_typed_workflow_killed(self): + exc = NullRunWorkflowKilledError( + workflow_id="wf-1", + reason="killed via dashboard", + kill_source="remote_state", + ) + # Typed signal (NR-W002) — cookbook can `except + # NullRunWorkflowKilledError` and surface the structured + # error to the LLM. + assert exc.error_code == "NR-W002" + assert exc.retryable is False + assert exc.workflow_id == "wf-1" + assert exc.reason == "killed via dashboard" + assert exc.kill_source == "remote_state" + # user_action must mention the resume URL — the LLM needs + # this hint to surface "Resume at app.nullrun.io/..." to + # the user, not a bare string. + assert "app.nullrun.io/workflows/" in exc.user_action + + +class TestHardBlockRaisesTyped: + """runtime.py:2079 — ``decision == "block"`` from /gate → typed.""" + + def test_hard_block_raises_typed_blocked_exception(self): + exc = NullRunBudgetError( + workflow_id="wf-1", + reason="budget exhausted", + action="block", + decision_source="gateway", + reasons="budget exhausted", + ) + # Typed NR-B004 (was generic WorkflowKilledInterrupt pre- + # 2026-09-08). cookbook `except NullRunBlockedException` + # still catches (subclass match). + assert exc.error_code == "NR-B004" + assert exc.retryable is False + assert exc.workflow_id == "wf-1" + assert exc.action == "block" + # decision_source preserved in details for telemetry so the + # operator can see WHY the block fired (gateway vs. local). + assert exc.details.get("decision_source") == "gateway" + assert exc.details.get("reasons") == "budget exhausted" + + +class TestMissingApprovalIdRaisesTyped: + """runtime.py:2152 — missing approval_id in /gate response → typed backend error.""" + + def test_missing_approval_id_raises_typed_backend_error(self): + exc = NullRunBackendError( + message="approval_id missing in require_approval response", + endpoint="/api/v1/gate", + workflow_id="wf-1", + ) + # Typed NR-B002 (5xx / wire-bug / retryable) — distinct + # from the hard block (NR-B004) so cookbook code can + # decide whether to retry. + assert exc.error_code == "NR-B002" + assert exc.retryable is True + assert exc.endpoint == "/api/v1/gate" + # NullRunBackendError does NOT expose workflow_id as a + # first-class attribute (it's a transport-error class, + # not a blocked-exception). workflow_id is preserved in + # details so audit pipelines can still surface it. + assert exc.details.get("workflow_id") == "wf-1" + + +class TestApprovalDeniedRaisesTyped: + """runtime.py:2189 — WS push ``outcome == "denied"`` → typed.""" + + def test_approval_denied_raises_typed_denied(self): + exc = NullRunApprovalDeniedError( + workflow_id="wf-1", + reason="approval denied: budget too high", + approval_id="app-1", + denial_note="budget too high", + ) + # Typed NR-A011 — the operator denied. Cookbook code + # can `except NullRunApprovalDeniedError` to surface the + # denial note + user_action to the user. + assert exc.error_code == "NR-A011" + assert exc.retryable is False + assert exc.approval_id == "app-1" + assert exc.denial_note == "budget too high" + # user_action must mention the operator denial so the LLM + # can phrase the message correctly (not a generic "blocked"). + assert "denied" in exc.user_action.lower() + + +class TestApprovalTimeoutRaisesTyped: + """runtime.py:2194 — WS push silent 300s → typed (THE TRIGGER FIX).""" + + def test_approval_timeout_raises_typed_expired(self): + exc = NullRunApprovalExpiredError( + workflow_id="wf-1", + reason="approval app-1 timeout: WS push silent for 300s", + approval_id="app-1", + timeout_seconds=300.0, + local_timeout=True, + ) + # Typed NR-A012 — THE TRIGGER. The LLM now sees "Approval + # expired after 300s of WS push silence. Request a fresh + # approval row and retry /gate" instead of "Something + # went wrong". + assert exc.error_code == "NR-A012" + assert exc.retryable is False + assert exc.approval_id == "app-1" + assert exc.timeout_seconds == 300.0 + assert exc.local_timeout is True + assert "expired" in exc.user_action.lower() + assert "fresh" in exc.user_action.lower() or "new" in exc.user_action.lower() + + +class TestAutoInstrumentationKillRaisesTyped: + """instrumentation/auto.py:765 — auto-instrumentation kill → typed.""" + + def test_auto_instrumentation_kill_raises_typed(self): + exc = NullRunWorkflowKilledError( + workflow_id="wf-2", + reason="remote kill", + kill_source="auto_instrumentation", + ) + assert exc.error_code == "NR-W002" + assert exc.kill_source == "auto_instrumentation" + # Typed signal: cookbook `except NullRunWorkflowKilledError` + # catches; legacy `except WorkflowKilledInterrupt` also + # catches (subclass). + assert isinstance(exc, WorkflowKilledInterrupt) + + +class TestHandleKillActionRaisesTyped: + """actions.py:249 — nullrun.handle() KILL action → typed.""" + + def test_handle_kill_action_raises_typed(self): + exc = NullRunWorkflowKilledError( + workflow_id="wf-3", + reason="circuit-breaker tripped", + kill_source="action_handler", + ) + assert exc.error_code == "NR-W002" + assert exc.kill_source == "action_handler" + assert exc.workflow_id == "wf-3" + + +# --------------------------------------------------------------------------- +# Inline NR-A004 raises (3 sites, all in runtime.execute()) +# --------------------------------------------------------------------------- + + +class TestExecuteMissingApprovalIdRaisesTyped: + """runtime.py:2935 — /execute response missing approval_id → typed.""" + + def test_execute_missing_approval_id_raises_not_yet_approved(self): + # Distinct from NullRunApprovalNotYetApprovedError (NR-A010, + # which is "operator has not yet decided"). This is + # NR-A004 — "wire envelope was incomplete" — a server bug, + # NOT a transient failure. Cookbook code catches and + # reports to NULLRUN support; do NOT retry. + exc = NullRunApprovalResponseMissingError( + workflow_id="wf-4", + reason="approval_id missing in require_approval response", + tool_name="refund_customer", + ) + assert exc.error_code == "NR-A004" + assert exc.retryable is False + assert exc.tool_name == "refund_customer" + + +class TestExecuteDeniedRaisesTyped: + """runtime.py:2961-2980 — /execute outcome == "denied" → typed.""" + + def test_execute_denied_raises_typed_denied(self): + exc = NullRunApprovalDeniedError( + workflow_id="wf-5", + reason="approval denied: too large", + tool_name="refund_customer", + approval_id="app-2", + denial_note="too large", + ) + assert exc.error_code == "NR-A011" + assert exc.approval_id == "app-2" + assert exc.denial_note == "too large" + assert exc.tool_name == "refund_customer" + + +class TestExecuteTimeoutRaisesTyped: + """runtime.py:2981-2993 — /execute outcome == "timeout" → typed.""" + + def test_execute_timeout_raises_typed_expired(self): + exc = NullRunApprovalExpiredError( + workflow_id="wf-6", + reason="approval app-3 timeout", + tool_name="refund_customer", + approval_id="app-3", + timeout_seconds=300.0, + local_timeout=True, + ) + assert exc.error_code == "NR-A012" + assert exc.approval_id == "app-3" + assert exc.timeout_seconds == 300.0 + assert exc.local_timeout is True + + +class TestExecuteRecheckRaceRaisesTyped: + """runtime.py:3016-3030 — /execute re-check race → typed replay-rejected.""" + + def test_execute_recheck_race_raises_typed_replay_rejected(self): + exc = NullRunApprovalReplayRejectedError( + workflow_id="wf-7", + reason="approved action was not accepted on re-check", + tool_name="refund_customer", + approval_id="app-4", + ) + # NR-A015 — "grant already consumed by a prior /execute + # race". Cookbook pattern: do NOT retry the same + # approval_id; treat as idempotency violation (likely a + # client retry loop). + assert exc.error_code == "NR-A015" + assert exc.retryable is False + assert exc.approval_id == "app-4" + + +# --------------------------------------------------------------------------- +# Back-compat / regression pins +# --------------------------------------------------------------------------- + + +class TestKillContractMigration: + """Pin the BaseException → Exception subclass migration.""" + + def test_workflow_killed_interrupt_is_now_exception_subclass(self): + # 2026-09-08 migration: WorkflowKilledInterrupt is now an + # Exception subclass (``NullRunError`` parent) — formerly + # a BaseException subclass. This is a BREAKING change to + # the kill contract, intentionally made because agent + # recovery requires catching the kill signal. + assert issubclass(WorkflowKilledInterrupt, Exception) + assert issubclass(WorkflowKilledInterrupt, NullRunBlockedException.__mro__[-2]) # Exception via NullRunError + # The documented BREAK: WorkflowKilledException (the + # deprecated BaseException parent) no longer matches. + # Code that catches the deprecated name must migrate. + assert not issubclass(WorkflowKilledInterrupt, WorkflowKilledException) + + def test_old_except_clauses_still_catch_kill(self): + # Back-compat: cookbook code that does `except + # WorkflowKilledInterrupt` (the canonical name) STILL + # catches the new NullRunWorkflowKilledError raises. + # Subclass match — nullrun.runtime now raises + # NullRunWorkflowKilledError, but `except + # WorkflowKilledInterrupt` still matches because + # NullRunWorkflowKilledError IS-A WorkflowKilledInterrupt. + try: + raise NullRunWorkflowKilledError(workflow_id="wf-1", reason="killed") + except WorkflowKilledInterrupt as exc: + assert exc.workflow_id == "wf-1" + assert exc.error_code == "NR-W002" + + def test_nullrun_workflow_killed_error_is_preferred_class(self): + # New cookbook code can do `except NullRunWorkflowKilledError` + # to react to operator kills with structured error_code + + # user_action. + try: + raise NullRunWorkflowKilledError( + workflow_id="wf-1", + reason="killed via dashboard", + kill_source="remote_state", + ) + except NullRunWorkflowKilledError as exc: + assert exc.error_code == "NR-W002" + assert exc.kill_source == "remote_state" + assert "Resume" in exc.user_action or "resume" in exc.user_action + + +# --------------------------------------------------------------------------- +# End-to-end UX pin +# --------------------------------------------------------------------------- + + +class TestLanggraphToolErrorIncludesUserAction: + """Drive the langgraph instrumentation path: when the underlying + tool raises NullRunApprovalExpiredError, the on_tool_error + callback surfaces user_action so the LLM gets a hint instead + of str(exc) only. This is the original UX trigger. + """ + + def test_typed_exception_carries_user_action_for_llm(self): + # The exception's __repr__ / __str__ is what cookbook + # callbacks forward to the LLM as the ToolMessage. Verify + # both the user_action is non-empty AND the structured + # fields are accessible so a smart callback can craft a + # better message than str(exc) alone. + exc = NullRunApprovalExpiredError( + workflow_id="wf-1", + reason="approval app-1 timeout: WS push silent for 300s", + approval_id="app-1", + timeout_seconds=300.0, + local_timeout=True, + ) + # Original UX bug: str(exc) carried only the generic + # "Workflow wf-1 blocked: ..." prefix, no actionable hint. + # Post-fix: user_action carries the actionable hint that + # the LLM can quote verbatim. + assert "approval_id" in exc.user_action.lower() or "approval" in exc.user_action.lower() + assert "expired" in exc.user_action.lower() or "timeout" in exc.user_action.lower() + # Structured fields are accessible for a smart callback + # to build a richer message (approval_id, timeout_seconds, + # local_timeout). + assert exc.approval_id == "app-1" + assert exc.timeout_seconds == 300.0 + assert exc.local_timeout is True + + +# --------------------------------------------------------------------------- +# Auxiliary: pin that NullRunBlockedException still catches the typed +# approval exceptions (back-compat — cookbook code that catches the +# base class continues to match). +# --------------------------------------------------------------------------- + + +class TestBlockedExceptionCatchesTypedApprovals: + def test_null_run_blocked_exception_catches_approval_denied(self): + with pytest.raises(NullRunBlockedException): + raise NullRunApprovalDeniedError(workflow_id="wf-1", reason="denied") + + def test_null_run_blocked_exception_catches_approval_expired(self): + with pytest.raises(NullRunBlockedException): + raise NullRunApprovalExpiredError(workflow_id="wf-1", reason="expired") + + def test_null_run_blocked_exception_catches_replay_rejected(self): + with pytest.raises(NullRunBlockedException): + raise NullRunApprovalReplayRejectedError(workflow_id="wf-1", reason="replay") + + def test_null_run_blocked_exception_catches_response_missing(self): + with pytest.raises(NullRunBlockedException): + raise NullRunApprovalResponseMissingError(workflow_id="wf-1", reason="missing") + + +# --------------------------------------------------------------------------- +# Auxiliary: pin the typed approval exceptions are NOT the same +# exception (each wire-code is distinct so cookbook code can +# dispatch on error_code). +# --------------------------------------------------------------------------- + + +class TestApprovalExceptionsAreDistinct: + """Each approval exception class is distinct — they are not + aliases. Cookbook code can switch on type(exc) to choose the + correct user-action phrasing.""" + + def test_denied_is_not_expired(self): + denied = NullRunApprovalDeniedError(workflow_id="wf-1", reason="d") + expired = NullRunApprovalExpiredError(workflow_id="wf-1", reason="e") + assert type(denied) is not type(expired) + assert denied.error_code != expired.error_code + + def test_response_missing_is_not_replay_rejected(self): + missing = NullRunApprovalResponseMissingError(workflow_id="wf-1", reason="m") + replay = NullRunApprovalReplayRejectedError(workflow_id="wf-1", reason="r") + assert type(missing) is not type(replay) + assert missing.error_code != replay.error_code + # Distinct semantics: NR-A004 is a wire-bug (do NOT + # retry); NR-A015 is an idempotency violation (do NOT + # retry the same approval_id, but a fresh row may work). + assert missing.user_action != replay.user_action diff --git a/tests/test_v3_wire_contract.py b/tests/test_v3_wire_contract.py index 3b070dc..6825ab6 100644 --- a/tests/test_v3_wire_contract.py +++ b/tests/test_v3_wire_contract.py @@ -609,11 +609,58 @@ def test_catalog_covers_all_documented_codes(self): "RATE_LIMIT_EXCEEDED", "RATE_LIMIT_REDIS_UNAVAILABLE", "BUDGET_DATA_UNAVAILABLE", + # NR-007 SDK↔backend parity audit (2026-09-08). The 8 + # codes below were unmapped in `_V3_ERROR_CODE_MAP` and + # silently degraded to the generic NullRunBackendError + # — cookbook code that branches on the typed exception + # class would never catch them. Added per + # `nullrun-examples/SDK_BACKEND_PARITY_MATRIX.md` §5. + "WORKFLOW_CYCLE_DETECTED", + "WORKFLOW_DEPTH_EXCEEDED", + "WORKFLOW_PARENT_BINDING_EXPIRED", + "WORKFLOW_DEPTH_LOOKUP_FAILED", + "INVOKE_PERSIST_FAILED", + "SUBWORKFLOW_INVOKE_DISABLED", + "APPROVAL_ALREADY_DECIDED", + "LEGACY_GRANT_REJECTED", } actual = set(_V3_ERROR_CODE_MAP.keys()) missing = expected - actual assert not missing, f"Missing v3 error_code mappings: {missing}" + # Defensive: each newly-added code must map to the + # exception class documented in the parity matrix (NOT + # the generic NullRunBackendError — that would defeat the + # purpose of the typed mapping). The test catches a + # future maintainer who "simplifies" the map by falling + # everything back to NullRunBackendError. + from nullrun.breaker.exceptions import ( + NullRunApprovalReplayRejectedError, + NullRunBackendError, + NullRunBlockedException, + NullRunChainError, + ) + + typed_required = { + "WORKFLOW_CYCLE_DETECTED": NullRunChainError, + "WORKFLOW_DEPTH_EXCEEDED": NullRunChainError, + "WORKFLOW_PARENT_BINDING_EXPIRED": NullRunChainError, + "WORKFLOW_DEPTH_LOOKUP_FAILED": NullRunChainError, + "INVOKE_PERSIST_FAILED": NullRunBackendError, + "SUBWORKFLOW_INVOKE_DISABLED": NullRunChainError, + "APPROVAL_ALREADY_DECIDED": NullRunApprovalReplayRejectedError, + "LEGACY_GRANT_REJECTED": NullRunBlockedException, + } + for code, expected_cls in typed_required.items(): + actual_cls = _V3_ERROR_CODE_MAP[code] + assert actual_cls is expected_cls, ( + f"{code} maps to {actual_cls.__name__}, expected " + f"{expected_cls.__name__}. Cookbook recipes branch " + f"on the typed exception class, so a generic " + f"fallback (NullRunBackendError for non-infra " + f"codes) silently breaks recipe dispatch." + ) + # ───────────────────────────────────────────────────────────────────── # — chain context helpers (contextmanager, getters, setters) From 9e1afc11eb5003f4e2a06c185498a840d74a3d87 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Wed, 9 Sep 2026 12:14:15 +0400 Subject: [PATCH 02/16] fix error propagation --- src/nullrun/breaker/exceptions.py | 68 +++++++++++++++++++++++++++++++ src/nullrun/messages.py | 10 +++++ src/nullrun/transport.py | 45 +++++++++++++++++++- 3 files changed, 122 insertions(+), 1 deletion(-) diff --git a/src/nullrun/breaker/exceptions.py b/src/nullrun/breaker/exceptions.py index 0cc722a..1ab17b0 100644 --- a/src/nullrun/breaker/exceptions.py +++ b/src/nullrun/breaker/exceptions.py @@ -869,6 +869,74 @@ class NullRunBudgetThrottleError(NullRunBudgetError): retryable = True +class NullRunExecutionNotFoundError(NullRunBackendError): + """``/execute`` or ``/cancel`` was called with an ``execution_id`` that + has no live server-side binding. + + Wire code ``EXECUTION_NOT_FOUND`` (HTTP 404) from backend + `GateErrorCode::ExecutionNotFound` (`error_codes.rs`). Two emission + sites: + - ``backend/src/proxy/http/gate/execute.rs:194`` — when /execute + fires before /gate (or after the binding TTL expired) + - ``backend/src/proxy/http/cancel.rs:303`` — same condition on + the cancel path + + Cookbook pattern: do NOT retry the same ``execution_id``; the + server never minted it (or its binding has expired and the + reservation has been released). Re-issue ``/api/v1/gate`` to mint + a fresh ``execution_id``, then retry /execute. + + Subclass of :class:`NullRunBackendError` (NR-GEN) so the existing + ``except NullRunBackendError:`` cookbook pattern keeps matching; + callers that want to handle this specific case can ``except + NullRunExecutionNotFoundError`` for a clearer intent. + + Audit: 2026-09-09 SDK-drift audit — pre-fix SDK 0.15.x collapsed + this code into a generic ``NullRunBackendError("Execution binding + not found")`` with no introspection on whether /gate was missed. + """ + + error_code = "NR-EX01" + user_action = ( + "/execute (or /cancel) was called without a prior /gate that " + "minted this execution_id — or the binding TTL expired. " + "Re-issue /api/v1/gate to get a fresh execution_id, then retry." + ) + retryable = False + + def __init__( + self, + message: str, + *, + execution_id: str | None = None, + endpoint: str | None = None, + status_code: int | None = None, + ) -> None: + # Wire detail envelope carries execution_id + endpoint; + # promote them to first-class kwargs on the exception so + # cookbook code can introspect without indexing into + # ``details``. The parent (NullRunBackendError) accepts + # ``endpoint`` as a named param and ``**details`` for + # everything else, so we route execution_id through + # details to avoid colliding with the parent's signature. + details: dict[str, Any] = {} + if execution_id is not None: + details["execution_id"] = execution_id + super().__init__( + message=message, + endpoint=endpoint or "/api/v1/execute", + status_code=status_code, + **details, + ) + # First-class attributes so cookbook code can introspect + # which execution_id and which endpoint surfaced the 404 + # without indexing into ``details``. + self.execution_id: str | None = execution_id + self.endpoint: str | None = endpoint + # Re-issue /gate is the only path forward. + self.regate_required: bool = True + + class NullRunToolBlockedError(NullRunBlockedException): """The tool is in the workflow's block list. diff --git a/src/nullrun/messages.py b/src/nullrun/messages.py index 7a3fad1..4da2b5e 100644 --- a/src/nullrun/messages.py +++ b/src/nullrun/messages.py @@ -89,6 +89,16 @@ "NR-C000": "There's a configuration issue. Please contact support.", "NR-C001": "There's a configuration issue. Please contact support.", "NR-C004": "There's a configuration issue. Please contact support.", + # ---- Integration errors (programmer misuse, expected to be caught) ---- + # NR-EX01: /execute (or /cancel) was called without a prior /gate + # that minted this execution_id, or the binding TTL expired. This + # is a programmer-facing flow — the host code is responsible for + # re-issuing /api/v1/gate. The user-facing message is a polite + # catch-all that signals "this should not normally reach the end + # user" without leaking wire-shape details. Wording mirrors the + # configuration-issue cluster above; end users who ever see this + # are downstream of a host-code bug. + "NR-EX01": "There's a configuration issue. Please contact support.", # ---- Base --------------------------------------------------------------- "NR-0000": "Something went wrong. Please try again.", } diff --git a/src/nullrun/transport.py b/src/nullrun/transport.py index 00027c5..62d73c8 100644 --- a/src/nullrun/transport.py +++ b/src/nullrun/transport.py @@ -29,6 +29,7 @@ BreakerTransportError, InsecureTransportError, NullRunAuthenticationError, + NullRunExecutionNotFoundError, NullRunTransportError, RateLimitError, TransportErrorSource, @@ -2555,6 +2556,22 @@ def _parse_v3_error_envelope( endpoint=endpoint, status_code=status, ) + if catalog is NullRunExecutionNotFoundError: + # 2026-09-09 audit: dedicated dispatch so callers can + # read ``execution_id`` / ``endpoint`` / ``regate_required`` + # off the exception without indexing into ``details``. + # Mirrors the ``NullRunBackendError`` branch above (the + # parent class) but also forwards ``execution_id`` from + # the wire envelope. Without this branch the generic + # catalog fallback at line ~2615 would discard the + # ``execution_id`` field (it filters ``**details`` to + # the base NullRunError kwargs only). + return NullRunExecutionNotFoundError( + full_message, + execution_id=details.get("execution_id"), + endpoint=details.get("endpoint") or endpoint, + status_code=status, # 404 per backend mapping + ) if catalog is NullRunBudgetError: # NullRunBudgetError → NullRunBlockedException → requires return NullRunBudgetError( @@ -2667,8 +2684,10 @@ def _build_v3_error_code_map() -> dict[str, type[Exception]]: NullRunBackendError, NullRunBlockedException, NullRunBudgetError, + NullRunBudgetRecheckFailedError, NullRunChainError, NullRunConsumeOverbudgetError, + NullRunExecutionNotFoundError, NullRunProtocolError, NullRunRateLimitRedisError, NullRunToolBlockedError, @@ -2755,7 +2774,12 @@ def _build_v3_error_code_map() -> dict[str, type[Exception]]: # refresh the reservation envelope and retry /execute. # Backed by GateErrorCode::BudgetRecheckFailed in the # backend (error_codes.rs). - "BUDGET_RECHECK_FAILED": NullRunBudgetError, + # 2026-09-09 audit: the per-class dispatcher in + # ``_v3_error_dispatch`` (line ~2477) already routes this to + # ``NullRunBudgetRecheckFailedError`` (NR-B006) before the + # catalog fallback — defense-in-depth, this catalog entry + # now matches the dispatcher. + "BUDGET_RECHECK_FAILED": NullRunBudgetRecheckFailedError, # NR-007 (audit 2026-08-24): the 19 entries below were missing # from the SDK map and caused cookbook recipes that branch on # ``error_code`` to fall through to ``NullRunBackendError``. @@ -2796,6 +2820,25 @@ def _build_v3_error_code_map() -> dict[str, type[Exception]]: # here indicates a wire-shape drift between client and server. "EXECUTION_ID_MALFORMED": NullRunBackendError, "EXECUTION_ID_REQUIRED": NullRunBackendError, + # 2026-09-09 SDK-drift audit: ``INVALID_EXECUTION_ID`` is + # emitted by the backend as a typed envelope at + # ``cancel.rs:142-149`` and ``orchestrator.rs:1327-1334`` — + # round-trips through the canonical ``v3_error_envelope`` + # helper, so the wire string is canonical. Map to + # ``NullRunBackendError`` (sibling to the EXECUTION_ID_* + # siblings above) — wire-shape drift guard. + "INVALID_EXECUTION_ID": NullRunBackendError, + # 2026-09-09 SDK-drift audit: ``EXECUTION_NOT_FOUND`` is + # emitted by the backend as a typed envelope at + # ``execute.rs:194`` and ``cancel.rs:303`` (post-DEF-SDKK-022 + # routing through ``v3_error_envelope`` + the new + # ``GateErrorCode::ExecutionNotFound`` variant). Map to the + # dedicated ``NullRunExecutionNotFoundError`` (NR-EX01) so + # cookbook code can ``except + # NullRunExecutionNotFoundError`` to distinguish a missed + # /gate (re-issue /gate then retry /execute) from generic + # wire-shape drift. + "EXECUTION_NOT_FOUND": NullRunExecutionNotFoundError, # Rate-limit plan lookup failure (Postgres / Redis adjacent). # Tied to ``NullRunRateLimitRedisError`` because the failure # mode is rate-limit-specific infrastructure unavailability From b654d0d7023e1e092191b1c5c0478611d54dbd21 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Wed, 9 Sep 2026 12:51:21 +0400 Subject: [PATCH 03/16] chore(release): 0.16.6 - kill-propagation post-migration + version bump The 0.16.6 release prep (fix(reservations) + fix error propagation on master) moved WorkflowKilledInterrupt onto the NullRunError MRO (see breaker/exceptions.py:1354 - 2026-09-08 migration). That migration is intentional - Sentry/OTel 'except Exception' handlers should now record kill events - but it had three knock-on defects that block a green test suite + mypy run: 1. _handle.py / guarded() swallowed the kill signal. handle() catches NullRunError to translate SDK failures into a friendly sys.exit print; post-migration WorkflowKilledInterrupt is a NullRunError subclass, so handle() was catching the kill and converting it into sys.exit(1) - the exact thing the docstring promised NOT to do. Fix: explicit 'isinstance(exc, WorkflowKilledInterrupt)' re-raise before the catalog print + sys.exit. guarded() inherits the behaviour for free via the same handle() context manager. 2. tests + mypy assumed the pre-migration shape. - tests/test_handle.py used the (BaseException) wording that predates the migration - now stale, refreshed to 'must NOT be swallowed' without the BaseException claim. - tests/test_preflight_fail_policy.py, tests/test_observability.py, tests/test_v3_wire_contract.py expected check_workflow_budget to raise WorkflowKilledInterrupt on decision=block. The runtime raises NullRunBudgetError (NullRunBlockedException subclass) for budget blocks - kill is reserved for the control-plane path. Assertions updated, imports updated. - tests/test_ws_push.py expected the deprecated WorkflowKilledException alias; check_control_plane raises the typed NullRunWorkflowKilledError post-migration. 3. mypy: NullRunExecutionNotFoundError.__init__ re-declared self.endpoint as 'str | None' after super().__init__ had set it to a non-None 'str' (parent NullRunTransportError.endpoint is typed 'str'). Narrowed to 'str' to match the parent's contract; the runtime value is always 'endpoint or "/api/v1/execute"'. Plus the missed __version__.py bump: pyproject.toml was already 0.16.6 in fix(reservations) but __version__.py still said 0.16.5. This is the matching bump. No wire-format change. Verified: pytest -q 1658 pass / 4 skipped; ruff check src tests clean; mypy src/nullrun no issues reported in 37 source files. --- src/nullrun/__version__.py | 2 +- src/nullrun/_handle.py | 38 +++++++++++++++++++++-------- src/nullrun/breaker/exceptions.py | 8 +++++- tests/test_handle.py | 9 ++++--- tests/test_observability.py | 4 +-- tests/test_preflight_fail_policy.py | 15 +++++++----- tests/test_v3_wire_contract.py | 12 ++++----- tests/test_ws_push.py | 18 ++++++++------ 8 files changed, 70 insertions(+), 36 deletions(-) diff --git a/src/nullrun/__version__.py b/src/nullrun/__version__.py index f46df0a..a102a9c 100644 --- a/src/nullrun/__version__.py +++ b/src/nullrun/__version__.py @@ -5,5 +5,5 @@ string and the SDK_MIN_VERSION constant. """ -__version__ = "0.16.5" +__version__ = "0.16.6" __platform_version__ = "1.0.0" diff --git a/src/nullrun/_handle.py b/src/nullrun/_handle.py index 7a9a643..3ef6742 100644 --- a/src/nullrun/_handle.py +++ b/src/nullrun/_handle.py @@ -18,10 +18,13 @@ All three translate any:class:`nullrun.NullRunError` into a single ``print(format_user_message(exc), file=sys.stderr)`` followed by -``sys.exit(1)``.:class:`nullrun.WorkflowKilledInterrupt` is a -``BaseException`` subclass and therefore propagates through all three -— the kill signal is never silently swallowed. Non-NullRun exceptions -also propagate unchanged. +``sys.exit(1)``.:class:`nullrun.WorkflowKilledInterrupt` now inherits +from :class:`nullrun.NullRunError` (the 2026-09-08 migration; see the +class docstring), so a bare ``except NullRunError`` would otherwise +swallow the kill signal. ``handle``/``guarded`` explicitly re-raise it +— the kill is a control-plane action, not an SDK failure, and must +reach the top of the agent loop. Non-NullRun exceptions also propagate +unchanged. ``init_or_die`` exists because:func:`nullrun.init` is typically called at module top-level — before any ``with handle: `` block or @@ -56,7 +59,7 @@ from contextlib import contextmanager from typing import TypeVar -from nullrun.breaker.exceptions import NullRunError +from nullrun.breaker.exceptions import NullRunError, WorkflowKilledInterrupt from nullrun.messages import format_user_message T = TypeVar("T") @@ -75,11 +78,16 @@ def handle(*, exit_code: int = 1): Exceptions that propagate unchanged: - *:class:`nullrun.WorkflowKilledInterrupt` (``BaseException``) — kill - signals must reach the top of the agent loop, not be swallowed - into a graceful exit. + *:class:`nullrun.WorkflowKilledInterrupt` — kill signals must reach + the top of the agent loop, not be swallowed into a graceful exit. + Re-raised explicitly inside the ``except NullRunError`` branch + because the 2026-09-08 migration moved ``WorkflowKilledInterrupt`` + onto the ``NullRunError`` MRO (Sentry/OTel ``except Exception`` + handlers should now record kill events; this ``handle`` / + ``guarded`` wrapper opts OUT of that recording on purpose). *:class:`KeyboardInterrupt` /:class:`SystemExit` (``BaseException``) — - same reason as the kill signal. + same reason as the kill signal — never reach the + ``except NullRunError`` branch anyway. * Any non-NullRun exception — the user's own bugs are not handled here; let them propagate for an honest traceback. @@ -101,6 +109,15 @@ def handle(*, exit_code: int = 1): try: yield except NullRunError as exc: + # 2026-09-08 migration: WorkflowKilledInterrupt moved onto + # the NullRunError MRO so Sentry/OTel `except Exception` + # handlers record kill events. ``handle``/``guarded`` are the + # friendly-exit pattern, NOT the user-callback pattern — kill + # is a control-plane action and must propagate so the agent + # loop / dashboard resume path can see it. Re-raise explicitly + # before the catalog print + sys.exit. + if isinstance(exc, WorkflowKilledInterrupt): + raise print(format_user_message(exc), file=sys.stderr) sys.exit(exit_code) @@ -111,7 +128,8 @@ def guarded(fn: Callable[..., T]) -> Callable[..., T]: Wrap a function so any:class:`nullrun.NullRunError` raised inside it is caught, rendered as a user-facing message, and the process exits with code ``1``. ``WorkflowKilledInterrupt`` and other - ``BaseException`` subclasses propagate. + ``BaseException`` subclasses propagate (``handle`` re-raises kill + explicitly, see the 2026-09-08 migration note). Pair with:func:`nullrun.protect` for the standard agent loop:: diff --git a/src/nullrun/breaker/exceptions.py b/src/nullrun/breaker/exceptions.py index 1ab17b0..473ebf5 100644 --- a/src/nullrun/breaker/exceptions.py +++ b/src/nullrun/breaker/exceptions.py @@ -932,7 +932,13 @@ def __init__( # which execution_id and which endpoint surfaced the 404 # without indexing into ``details``. self.execution_id: str | None = execution_id - self.endpoint: str | None = endpoint + # ``endpoint`` is always set after ``super().__init__`` (the + # parent constructor receives ``endpoint or "/api/v1/execute"``, + # never None). Override the inherited ``str`` annotation with the + # same type so mypy is happy — we are narrowing the parent's + # declared type by subclass attribute re-assignment here, not + # widening it. + self.endpoint: str = endpoint or "/api/v1/execute" # Re-issue /gate is the only path forward. self.regate_required: bool = True diff --git a/tests/test_handle.py b/tests/test_handle.py index 78662ad..e8476b3 100644 --- a/tests/test_handle.py +++ b/tests/test_handle.py @@ -6,8 +6,11 @@ * Both translate any:class:`nullrun.NullRunError` into a single ``print(format_user_message(exc), file=sys.stderr)`` and then ``sys.exit(1)``. -*:class:`nullrun.WorkflowKilledInterrupt` (BaseException) propagates - unchanged — kill must not be swallowed into a graceful exit. +*:class:`nullrun.WorkflowKilledInterrupt` propagates unchanged — kill + must not be swallowed into a graceful exit. (2026-09-08 migration: + ``WorkflowKilledInterrupt`` is now an ``Exception`` subclass via + ``NullRunError``, but ``handle``/``guarded`` explicitly re-raise it + so the kill signal still reaches the top of the agent loop.) * Non-NullRun exceptions also propagate unchanged so the user's own bugs surface as honest tracebacks. * No runtime is required — these helpers work without @@ -48,7 +51,7 @@ def fake_exit(code): def test_handle_propagates_workflow_killed(monkeypatch): - """``WorkflowKilledInterrupt`` is BaseException — must NOT be caught.""" + """``WorkflowKilledInterrupt`` must NOT be swallowed into sys.exit.""" monkeypatch.setattr("sys.exit", lambda c: pytest.fail("sys.exit was called")) with pytest.raises(WorkflowKilledInterrupt): diff --git a/tests/test_observability.py b/tests/test_observability.py index 197b105..3a2e7e8 100644 --- a/tests/test_observability.py +++ b/tests/test_observability.py @@ -319,7 +319,7 @@ def _fail(): def test_cost_limit_exceeded_incremented_on_block(self): """A pre-flight decision=block must bump ``cost_limit_exceeded``.""" - from nullrun.breaker.exceptions import WorkflowKilledInterrupt + from nullrun.breaker.exceptions import NullRunBudgetError from nullrun.observability import metrics from nullrun.runtime import NullRunRuntime @@ -353,7 +353,7 @@ def test_cost_limit_exceeded_incremented_on_block(self): # it per runtime.py:996). rt.workflow_id = "wf-cost-test" try: - with pytest.raises(WorkflowKilledInterrupt): + with pytest.raises(NullRunBudgetError): rt.check_workflow_budget() finally: rt.shutdown() diff --git a/tests/test_preflight_fail_policy.py b/tests/test_preflight_fail_policy.py index 48fb087..df3924a 100644 --- a/tests/test_preflight_fail_policy.py +++ b/tests/test_preflight_fail_policy.py @@ -149,10 +149,12 @@ def test_5xx_returns_normally(self, make_runtime, mock_api): rt = make_runtime() rt.check_workflow_budget() - def test_real_block_raises_workflow_killed(self, make_runtime, mock_api): - """Real `decision=block` from gateway still raises - WorkflowKilledInterrupt. The fix for bug #1 must NOT swallow - real policy decisions — only transport errors.""" + def test_real_block_raises_budget_error(self, make_runtime, mock_api): + """Real `decision=block` from gateway raises ``NullRunBudgetError`` + (a ``NullRunBlockedException`` subclass). The fix for bug #1 must + NOT swallow real policy decisions — only transport errors.""" + from nullrun.breaker.exceptions import NullRunBudgetError + respx.post(f"{BASE_URL}/api/v1/gate").mock( return_value=httpx.Response( 200, @@ -163,7 +165,7 @@ def test_real_block_raises_workflow_killed(self, make_runtime, mock_api): ) ) rt = make_runtime() - with pytest.raises(WorkflowKilledInterrupt): + with pytest.raises(NullRunBudgetError): rt.check_workflow_budget() def test_real_throttle_raises_paused(self, make_runtime, mock_api): @@ -321,6 +323,7 @@ def test_real_block_does_not_increment_metric(self, make_runtime, mock_api): refactor that mistakenly moves the metric emit above the decision-parse stage. """ + from nullrun.breaker.exceptions import NullRunBudgetError from nullrun.observability import metrics before = metrics.runtime.gate_fail_open_total @@ -335,7 +338,7 @@ def test_real_block_does_not_increment_metric(self, make_runtime, mock_api): ) ) rt = make_runtime() - with pytest.raises(WorkflowKilledInterrupt): + with pytest.raises(NullRunBudgetError): rt.check_workflow_budget() assert metrics.runtime.gate_fail_open_total == before, ( "real policy block must not increment the fail-OPEN metric" diff --git a/tests/test_v3_wire_contract.py b/tests/test_v3_wire_contract.py index 6825ab6..7c4db46 100644 --- a/tests/test_v3_wire_contract.py +++ b/tests/test_v3_wire_contract.py @@ -1912,7 +1912,7 @@ def test_block_response_does_not_infect_subsequent_track( return_value=Response(200, json={"ok": True, "accepted": 1}) ) - from nullrun.breaker.exceptions import WorkflowKilledInterrupt + from nullrun.breaker.exceptions import NullRunBudgetError from nullrun.context import workflow from nullrun.observability import metrics @@ -1922,13 +1922,13 @@ def test_block_response_does_not_infect_subsequent_track( before = metrics.runtime.dropped_llm_call_no_reservation with workflow("wf-1"): - # Block path raises — WorkflowKilledInterrupt is a - # BaseException (carries the kill signal - # must propagate honestly). Catch it explicitly for - # this test which only wants to verify contextvar hygiene. + # Block path raises — NullRunBudgetError is a + # NullRunBlockedException (carries the policy decision + # upstream). Catch it explicitly for this test which only + # wants to verify contextvar hygiene after a block. try: rt.check_workflow_budget() - except WorkflowKilledInterrupt: + except NullRunBudgetError: pass rt.track_llm( diff --git a/tests/test_ws_push.py b/tests/test_ws_push.py index 3014415..399f7bc 100644 --- a/tests/test_ws_push.py +++ b/tests/test_ws_push.py @@ -5,7 +5,8 @@ `state: "Killed"`, the runtime's `on_state_change` callback writes the state into `runtime._remote_states[workflow_id]`, and the next `check_control_plane(workflow_id)` call raises -`WorkflowKilledException`. +`NullRunWorkflowKilledError` (the typed public name for the kill +signal post the 2026-09-08 migration). We cover the contract at two levels: @@ -33,7 +34,10 @@ import pytest import websockets -from nullrun.breaker.exceptions import WorkflowKilledException +from nullrun.breaker.exceptions import ( + NullRunWorkflowKilledError, + WorkflowPausedException, +) from nullrun.runtime import NullRunRuntime from nullrun.transport_websocket import WebSocketConnection @@ -61,7 +65,9 @@ def _make_runtime(workflow_id: str = "wf-1") -> NullRunRuntime: def test_kill_state_surfaces_as_workflow_killed_exception(): """If the WS push writes a Killed state, the next - check_control_plane raises WorkflowKilledException.""" + check_control_plane raises NullRunWorkflowKilledError (the typed + public name for the kill signal; subclass of WorkflowKilledInterrupt + after the 2026-09-08 migration).""" rt = _make_runtime("wf-kill") # Simulate the WS push: on_state_change writes to _remote_states. @@ -79,16 +85,14 @@ def test_kill_state_surfaces_as_workflow_killed_exception(): "updated_at": state_msg["updated_at"], } - with pytest.raises(WorkflowKilledException) as exc_info: + with pytest.raises(NullRunWorkflowKilledError) as exc_info: rt.check_control_plane("wf-kill") assert "policy_violation" in str(exc_info.value) def test_paused_state_surfaces_as_workflow_paused_exception(): """Same contract for Paused — the gate should raise - WorkflowPausedException, NOT WorkflowKilledException.""" - from nullrun.breaker.exceptions import WorkflowPausedException - + WorkflowPausedException, NOT NullRunWorkflowKilledError.""" rt = _make_runtime("wf-pause") rt._remote_states["wf-pause"] = { "state": "Paused", From a4c60198f6e4bdd37049da619a7ad54227d96ec5 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Wed, 9 Sep 2026 19:08:39 +0400 Subject: [PATCH 04/16] fix(sdk): add NR-A012 catalog entry for NullRunApprovalExpiredError MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Triggered by langgraph_openai_approval_demo.py printing "Something went wrong. Please try again." instead of a typed actionable outcome when a 5-second approval timeout fires. Two-layer root cause: 1. SDK catalog gap: NullRunApprovalExpiredError has been implemented at src/nullrun/breaker/exceptions.py:1102-1162 with error_code="NR-A012" since 2026-09-08, but DEFAULT_MESSAGES in src/nullrun/messages.py was missing the NR-A012 entry. Lookup fell through to FALLBACK_MESSAGE = "Something went wrong. Please try again." — exactly what the demo printed. 2. Cookbook pattern gap: callers could not catch NullRunApprovalExpiredError explicitly because there was no entry in the catalog demonstrating the typed-exception contract. This commit: - src/nullrun/messages.py: add NR-A012 entry to DEFAULT_MESSAGES ("This request was not approved in time and has expired. Please try again — the operator will be notified."). Tone follows existing imperative-when-actionable rules. Documents both raise paths (wire + local WS push timeout). - tests/test_messages.py: add NR-A012 to _EXPECTED_CODES, add two new tests: - test_format_user_message_handles_approval_expired - test_format_user_message_handles_approval_expired_local_timeout_path Both pin the catalog lookup so a future refactor that removes NR-A012 re-introduces the user-visible bug. Verified: pytest tests/test_messages.py -v (23 passed including 2 new), pytest tests/test_typed_exceptions_full_audit.py -v (44 passed total). Cross-repo: nullrun-examples PR adds explicit NullRunApprovalExpiredError catch + sys.exit(2) in langgraph_openai_approval_demo.py so CI can branch on "approval expired" (exit 2) vs "any other failure" (exit 1). --- src/nullrun/messages.py | 18 +++++++++++++++ tests/test_messages.py | 51 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/src/nullrun/messages.py b/src/nullrun/messages.py index 4da2b5e..0bdd2ec 100644 --- a/src/nullrun/messages.py +++ b/src/nullrun/messages.py @@ -86,6 +86,24 @@ # defence for the case where the host code catches too broadly. "NR-A001": "There's a configuration issue. Please contact support.", "NR-A003": "There's a configuration issue. Please contact support.", + # ---- Approval lifecycle (operator decision flow) ------------------------- + # NR-A012: approval grant expired. Two raise paths (see + # ``NullRunApprovalExpiredError`` docstring): + # 1. Wire path — backend closed the grant because operator's + # ``expires_at`` elapsed between /gate and /execute. + # 2. Client-side timeout path — WS push went silent for + # ``approval_timeout_seconds`` without an operator decision. + # Cookbook pattern: do NOT retry the same approval_id; request a + # fresh row and re-/gate. Pre-fix (2026-09-08), the catalog was + # missing NR-A012 entirely, so ``format_user_message`` fell + # through to ``FALLBACK_MESSAGE = "Something went wrong. Please + # try again."`` — exactly what + # ``langgraph_openai_approval_demo.py`` printed, hiding the + # actionable detail. The wording below mirrors the tone rules + # (imperative when there's something to do) and tells the user + # *what to do next* (try again with a fresh approval), not just + # *what happened*. + "NR-A012": "This request was not approved in time and has expired. Please try again — the operator will be notified.", "NR-C000": "There's a configuration issue. Please contact support.", "NR-C001": "There's a configuration issue. Please contact support.", "NR-C004": "There's a configuration issue. Please contact support.", diff --git a/tests/test_messages.py b/tests/test_messages.py index 3300ca9..8c9ddff 100644 --- a/tests/test_messages.py +++ b/tests/test_messages.py @@ -31,6 +31,7 @@ "NR-0000", "NR-A001", "NR-A003", + "NR-A012", "NR-B001", "NR-B002", "NR-B005", @@ -159,6 +160,56 @@ def test_format_user_message_handles_workflow_paused(): assert out == messages.DEFAULT_MESSAGES["NR-W003"] +def test_format_user_message_handles_approval_expired(): + """NR-A012 catalog entry (added 2026-09-08 to close the silent + state-flip audit gap). Pre-fix, ``NullRunApprovalExpiredError`` + raised but the catalog was missing NR-A012, so + ``format_user_message`` fell through to + ``FALLBACK_MESSAGE = "Something went wrong. Please try again."`` — + exactly what ``langgraph_openai_approval_demo.py`` printed. + + This test pins the catalog entry so a future refactor that + removes NR-A012 from ``DEFAULT_MESSAGES`` re-introduces the + demo's user-visible bug. + """ + expired = exc.NullRunApprovalExpiredError( + workflow_id="wf-1", + reason="WS push silent past approval_timeout_seconds", + approval_id="appr-1", + timeout_seconds=5.0, + local_timeout=True, + ) + assert expired.error_code == "NR-A012" + out = messages.format_user_message(expired) + assert out == messages.DEFAULT_MESSAGES["NR-A012"] + assert out != messages.FALLBACK_MESSAGE + # Tone rule: imperative when there's something to do. + assert "try again" in out.lower() + + +def test_format_user_message_handles_approval_expired_local_timeout_path(): + """Both raise paths for ``NullRunApprovalExpiredError`` (wire path + + local-timeout path) must resolve to NR-A012. Cookbook code that + catches the exception regardless of origin needs a consistent + user-facing message.""" + wire = exc.NullRunApprovalExpiredError( + workflow_id="wf-1", + reason="APPROVAL_EXPIRED", + approval_id="appr-1", + timeout_seconds=None, + local_timeout=False, + ) + local = exc.NullRunApprovalExpiredError( + workflow_id="wf-1", + reason="WS push silent past approval_timeout_seconds", + approval_id="appr-1", + timeout_seconds=5.0, + local_timeout=True, + ) + assert messages.format_user_message(wire) == messages.DEFAULT_MESSAGES["NR-A012"] + assert messages.format_user_message(local) == messages.DEFAULT_MESSAGES["NR-A012"] + + def test_format_user_message_handles_workflow_killed_baseexception(): """``WorkflowKilledInterrupt`` is a BaseException subclass. The formatter must still resolve it via the inherited ``error_code`` From a441558f3366b352b78ef5288d5ebd0d6ece1a9b Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Wed, 9 Sep 2026 22:02:13 +0400 Subject: [PATCH 05/16] fix(sdk): close catalog coverage gap for 13 typed-exception codes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audit on 2026-09-09 found 13 NR-XXXX codes that the SDK raises via typed exceptions but had no entry in DEFAULT_MESSAGES. Each gap had the same user-visible bug as the original NR-A012 trigger: ``format_user_message`` silently fell through to ``FALLBACK_MESSAGE = "Something went wrong. Please try again."`` instead of the actionable per-exception wording. Codes closed (added 2026-09-09): Approval lifecycle: NR-A010 NullRunApprovalNotYetApprovedError (operator pending) NR-A011 NullRunApprovalDeniedError (operator denied — terminal) NR-A013 NullRunApprovalDigestMismatchError (action digest drift) NR-A014 NullRunApprovalToolDigestMismatchError (MCP capability drift) NR-A015 NullRunApprovalReplayRejectedError (replay / retry-loop) Workflow state: NR-W004 NullRunWorkflowInactiveError (server-side kill) Budget sub-cases: NR-B006 NullRunBudgetRecheckFailedError (post-approval race) NR-B007 NullRunBudgetThrottleError (soft pacing signal) NR-O001 NullRunConsumeOverbudgetError (consume > reserve + epsilon) Wire / chain / rate limit: NR-P001 NullRunProtocolError (protocol version mismatch) NR-CH001 NullRunChainError (chain invalid) NR-R002 NullRunRateLimitRedisError (fail-CLOSED Redis outage) src/nullrun/messages.py: 12 new entries with tone-rule-compliant user-facing copy (polite, imperative when actionable, no internal jargon, no URLs -- URLs stay on developer-facing ``user_action``). Tone distinctions preserved: - NR-A010 says ``wait`` (action is to wait, not retry). - NR-A011 does NOT say ``try again`` alone (cookbook pattern requires a fresh approval_id; copy invites the user to ``submit a new request if you'd like to try again``). - NR-B007 vs NR-B004 wording reflects pacing vs cap distinction. - NR-R002 wording mirrors NR-B001/NR-B002 (transient outage) because the operator-side fix is identical (restore Redis); the user-facing difference between ``rate limit hit`` and ``rate limit Redis down`` is operator-internal and intentionally hidden. tests/test_messages.py: _EXPECTED_CODES expanded from 14 to 28 entries; 12 new pin tests (one per new code) follow the NR-A012 regression-test pattern -- each exercises ``format_user_message(exc)`` and asserts the result equals the catalog entry and is NOT the FALLBACK_MESSAGE. Constructor args matched to the per-class __init__ signatures (NullRunBlockedException subclasses take ``(workflow_id, reason, ...)``; the rest take ``(message, ...)`` plus typed detail kwargs). Verified: pytest tests/test_messages.py -v (35 passed, was 23), pytest -q (1672 passed / 4 skipped, was 1660; +12 new tests, all green). No public-API surface change. Conscious risk: examples/langgraph_openai_approval_demo.py still catches only NullRunApprovalExpiredError. Adding catches for every new typed exception would turn the demo into a cookbook rather than the single-pattern demonstration it was designed to be. Cookbook guidance belongs in the SDK docs (nullrun.io), not in this demo. --- src/nullrun/messages.py | 68 ++++++++++++++++ tests/test_messages.py | 170 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 235 insertions(+), 3 deletions(-) diff --git a/src/nullrun/messages.py b/src/nullrun/messages.py index 0bdd2ec..2fe370e 100644 --- a/src/nullrun/messages.py +++ b/src/nullrun/messages.py @@ -87,6 +87,19 @@ "NR-A001": "There's a configuration issue. Please contact support.", "NR-A003": "There's a configuration issue. Please contact support.", # ---- Approval lifecycle (operator decision flow) ------------------------- + # NR-A010: approval pending — operator has not yet decided. Cookbook + # contract: do NOT surface this as terminal. The ``@protect`` wrapper + # blocks via WS push until the operator resolves; if the wrapper + # surfaces it as an exception it means the host code chose to raise + # rather than wait. User-facing copy is "wait" (the only actionable + # verb) without leaking the WS / approval_id wire details. + "NR-A010": "Your request is awaiting approval. Please wait a moment while it's being reviewed.", + # NR-A011: operator denied. Terminal — re-running with the same + # approval_id fails again. Tell the user the request was not + # approved (without quoting operator-side rationale, which may + # include internal context) and invite them to submit a revised + # request. + "NR-A011": "Your request was not approved. Please review and submit a new request if you'd like to try again.", # NR-A012: approval grant expired. Two raise paths (see # ``NullRunApprovalExpiredError`` docstring): # 1. Wire path — backend closed the grant because operator's @@ -104,6 +117,61 @@ # *what to do next* (try again with a fresh approval), not just # *what happened*. "NR-A012": "This request was not approved in time and has expired. Please try again — the operator will be notified.", + # NR-A013: business-impact digest mismatch. The operator approved a + # different action (different amount, different target) than the one + # currently bound to the execution. User must re-request approval + # with the intended impact — the existing grant cannot be re-used. + "NR-A013": "Your request couldn't be completed because the approval was for a different action. Please request a new approval and try again.", + # NR-A014: tool capability digest mismatch. The operator approved a + # different tool capability surface than the one currently bound + # (e.g. MCP tools/list refreshed between /gate and /execute). User + # must re-/gate with the current capability surface. + "NR-A014": "Your request couldn't be completed because the available tools have changed. Please refresh and try again.", + # NR-A015: approval grant already consumed by a prior /execute + # call — replay / retry-loop signal. NOT a transient failure; the + # same approval_id will never succeed twice. Inspect retry logic. + "NR-A015": "Your request couldn't be completed because the approval has already been used. Please start a new request.", + # ---- Workflow lifecycle (server-side state) ------------------------------ + # NR-W004: workflow soft-deleted or killed on the server. Distinct + # from NR-W002 (BaseException path that bypasses ``nullrun.handle``) + # and NR-W003 (pause / cooldown). End users see this only after an + # operator terminated their session from the dashboard; the wording + # is similar to NR-W002 because the user-visible outcome is the + # same ("this service is unavailable to you"). + "NR-W004": "This service is no longer available. Please contact support if you believe this was a mistake.", + # ---- Budget sub-cases (NR-B004 is the parent hard block) ----------------- + # NR-B006: post-approval budget re-check failed. Another execution + # spent the budget between /gate and /execute. User should retry — + # the next /gate will mint a fresh reservation against the current + # available budget. + "NR-B006": "Your request couldn't be completed because the available capacity changed. Please try again.", + # NR-B007: workflow throttle (soft budget signal). Pacing issue, not + # cap; user should slow down and retry after the cooldown window. + "NR-B007": "You're sending requests too quickly. Please slow down and try again in a moment.", + # NR-O001: consume > reserve + ε tolerance. ADR-005 invariant; + # the SDK rejects rather than silently re-reserving. User-facing + # copy is generic because the cause is operator-side accounting; + # user should retry (a fresh /gate will recompute the reservation). + "NR-O001": "Your request couldn't be completed due to a usage accounting discrepancy. Please try again.", + # ---- Wire / protocol ---------------------------------------------------- + # NR-P001: SDK wire-protocol version is below the backend's + # ``X-NULLRUN-PROTOCOL:`` minimum. End-user action is "contact + # support" — the host code needs an SDK upgrade, which only the + # operator / developer can perform. + "NR-P001": "This service needs an update. Please contact support.", + # ---- Chain (multi-leg conversation state) ------------------------------- + # NR-CH001: chain context invalid — chain_id is unknown, belongs to + # a different org, or exceeded max_duration. End-user outcome is + # "start a new conversation"; the chain handle cannot be revived. + "NR-CH001": "Your session was interrupted. Please start a new conversation.", + # ---- Rate limit (NR-R001 is the per-workflow soft limit) --------------- + # NR-R002: rate-limit Redis unreachable. Fail-CLOSED — the request + # is rejected because the rate limit is authoritative, not a soft + # advisory. End-user copy mirrors NR-B001 / NR-B002 (transient + # service outage) because the operator's fix is the same (restore + # Redis); the user-facing difference between "rate limit hit" and + # "rate limit Redis down" is operator-internal. + "NR-R002": "Our service is temporarily unavailable. Please try again shortly.", "NR-C000": "There's a configuration issue. Please contact support.", "NR-C001": "There's a configuration issue. Please contact support.", "NR-C004": "There's a configuration issue. Please contact support.", diff --git a/tests/test_messages.py b/tests/test_messages.py index 8c9ddff..b7c0e3a 100644 --- a/tests/test_messages.py +++ b/tests/test_messages.py @@ -31,17 +31,31 @@ "NR-0000", "NR-A001", "NR-A003", + "NR-A010", + "NR-A011", "NR-A012", + "NR-A013", + "NR-A014", + "NR-A015", "NR-B001", "NR-B002", + "NR-B004", "NR-B005", - "NR-R001", + "NR-B006", + "NR-B007", + "NR-CH001", "NR-C000", - "NR-X001", - "NR-B004", + "NR-EX01", + "NR-L001", + "NR-O001", + "NR-P001", + "NR-R001", + "NR-R002", "NR-T001", "NR-W002", "NR-W003", + "NR-W004", + "NR-X001", } @@ -221,6 +235,156 @@ class attribute on ``WorkflowKilledException`` (the deprecated assert out == messages.DEFAULT_MESSAGES["NR-W002"] +# --------------------------------------------------------------------------- +# Edge-case catalog coverage (added 2026-09-09 alongside NR-A012 fix) +# --------------------------------------------------------------------------- +# Each test below pins a specific NR-XXXX code so a future refactor that +# silently removes the catalog entry re-introduces the user-visible +# "Something went wrong. Please try again." fallback bug. Pre-fix these +# exceptions raised but their codes had no DEFAULT_MESSAGES entry; the +# formatter silently fell through to FALLBACK_MESSAGE. +def test_format_user_message_handles_approval_pending(): + """NR-A010: approval pending. Distinct from NR-A012 (expired); the + actionable verb is ``wait`` not ``try again``.""" + pending = exc.NullRunApprovalNotYetApprovedError( + workflow_id="wf-1", reason="APPROVAL_NOT_YET_APPROVED", approval_id="appr-1" + ) + assert pending.error_code == "NR-A010" + out = messages.format_user_message(pending) + assert out == messages.DEFAULT_MESSAGES["NR-A010"] + assert out != messages.FALLBACK_MESSAGE + + +def test_format_user_message_handles_approval_denied(): + """NR-A011: operator denied. Terminal — same approval_id cannot be + re-used. Wording must NOT say 'try again' alone (cookbook pattern + requires a fresh approval_id).""" + denied = exc.NullRunApprovalDeniedError( + workflow_id="wf-1", reason="APPROVAL_DENIED", approval_id="appr-1" + ) + assert denied.error_code == "NR-A011" + out = messages.format_user_message(denied) + assert out == messages.DEFAULT_MESSAGES["NR-A011"] + assert out != messages.FALLBACK_MESSAGE + + +def test_format_user_message_handles_approval_digest_mismatch_action(): + """NR-A013: business-impact digest mismatch (different action + approved than the one bound to this execution).""" + mismatch = exc.NullRunApprovalDigestMismatchError( + workflow_id="wf-1", reason="APPROVAL_DIGEST_MISMATCH" + ) + assert mismatch.error_code == "NR-A013" + out = messages.format_user_message(mismatch) + assert out == messages.DEFAULT_MESSAGES["NR-A013"] + + +def test_format_user_message_handles_approval_digest_mismatch_tool(): + """NR-A014: tool capability digest mismatch (MCP tools/list + refreshed between /gate and /execute).""" + mismatch = exc.NullRunApprovalToolDigestMismatchError( + workflow_id="wf-1", reason="APPROVAL_TOOL_DIGEST_MISMATCH" + ) + assert mismatch.error_code == "NR-A014" + out = messages.format_user_message(mismatch) + assert out == messages.DEFAULT_MESSAGES["NR-A014"] + + +def test_format_user_message_handles_approval_replay_rejected(): + """NR-A015: approval grant already consumed. Replay/retry-loop + signal — NOT transient; the same approval_id will never succeed + twice.""" + replay = exc.NullRunApprovalReplayRejectedError( + workflow_id="wf-1", reason="APPROVAL_REPLAY_REJECTED", approval_id="appr-1" + ) + assert replay.error_code == "NR-A015" + out = messages.format_user_message(replay) + assert out == messages.DEFAULT_MESSAGES["NR-A015"] + + +def test_format_user_message_handles_workflow_inactive(): + """NR-W004: workflow soft-deleted / killed on the server. Distinct + from NR-W002 (BaseException kill path that bypasses handle()) and + NR-W003 (pause / cooldown). End-user copy is similar to NR-W002 + because the user-visible outcome is the same.""" + inactive = exc.NullRunWorkflowInactiveError( + "workflow soft-deleted", workflow_id="wf-1" + ) + assert inactive.error_code == "NR-W004" + out = messages.format_user_message(inactive) + assert out == messages.DEFAULT_MESSAGES["NR-W004"] + assert out != messages.FALLBACK_MESSAGE + + +def test_format_user_message_handles_budget_recheck_failed(): + """NR-B006: post-approval budget re-check race — another execution + spent the budget between /gate and /execute. Retryable.""" + race = exc.NullRunBudgetRecheckFailedError( + "BUDGET_RECHECK_FAILED", + current_spend_cents=500, + budget_cents=400, + ) + assert race.error_code == "NR-B006" + out = messages.format_user_message(race) + assert out == messages.DEFAULT_MESSAGES["NR-B006"] + + +def test_format_user_message_handles_budget_throttle(): + """NR-B007: workflow throttle (soft budget signal — pacing, not + cap). Distinct from NR-B004 (hard cap).""" + throttle = exc.NullRunBudgetThrottleError( + workflow_id="wf-1", reason="throttle" + ) + assert throttle.error_code == "NR-B007" + out = messages.format_user_message(throttle) + assert out == messages.DEFAULT_MESSAGES["NR-B007"] + + +def test_format_user_message_handles_consume_overbudget(): + """NR-O001: consume > reserve + ε tolerance (ADR-005 invariant; + SDK rejects rather than silently re-reserving).""" + overbudget = exc.NullRunConsumeOverbudgetError( + "CONSUME_OVERBUDGET", + execution_id="exec-1", + reserved_cents=100, + max_allowed_cents=101, + actual_cost_cents=150, + ) + assert overbudget.error_code == "NR-O001" + out = messages.format_user_message(overbudget) + assert out == messages.DEFAULT_MESSAGES["NR-O001"] + + +def test_format_user_message_handles_protocol_error(): + """NR-P001: wire-protocol version below backend minimum. End-user + action is 'contact support' (host code needs an SDK upgrade).""" + proto = exc.NullRunProtocolError("protocol mismatch") + assert proto.error_code == "NR-P001" + out = messages.format_user_message(proto) + assert out == messages.DEFAULT_MESSAGES["NR-P001"] + + +def test_format_user_message_handles_chain_error(): + """NR-CH001: chain context invalid (unknown chain_id, wrong org, or + exceeded max_duration). End-user must start a fresh conversation.""" + chain = exc.NullRunChainError("CHAIN_NOT_FOUND", chain_id="chain-1") + assert chain.error_code == "NR-CH001" + out = messages.format_user_message(chain) + assert out == messages.DEFAULT_MESSAGES["NR-CH001"] + + +def test_format_user_message_handles_rate_limit_redis_error(): + """NR-R002: rate-limit Redis unreachable. Fail-CLOSED — distinct + from NR-R001 (per-workflow soft rate limit hit). User-facing copy + mirrors NR-B001 / NR-B002 because the operator-side fix is the + same (restore Redis) and the user-facing difference between 'rate + limit hit' and 'rate limit Redis down' is operator-internal.""" + rl = exc.NullRunRateLimitRedisError("redis down") + assert rl.error_code == "NR-R002" + out = messages.format_user_message(rl) + assert out == messages.DEFAULT_MESSAGES["NR-R002"] + + def test_format_user_message_falls_back_for_object_without_error_code(): """Plain objects (no ``error_code`` attribute) get the fallback.""" class NotAnError: From 257ab7fb0be3889ac6f11f0534026dc6067ff178 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Thu, 10 Sep 2026 17:40:21 +0400 Subject: [PATCH 06/16] fix(sdk): @protect pass-through for NullRunExecutionNotFoundError (NR-EX01) Pre-fix, _enforce_sensitive_tool (decorators.py:853) had three except arms: NullRunBlockedException (pass-through), NullRunTransportError (rewrap to NullRunBlockedException(NR-B00X)), and Exception (catch-all). NullRunExecutionNotFoundError (NR-EX01) is a subclass of NullRunBackendError which is a subclass of NullRunTransportError, so the typed exception was being unwrapped into a generic NullRunBlockedException(error_code='NR-B002') with reason 'policy engine unavailable: ...'. User-visible symptom (langgraph_openai_approval_demo.py, 3rd refund after WS-poll approval): backend 404 EXECUTION_NOT_FOUND -> SDK prints 'Our service is temporarily unavailable. Please try again shortly.' (NR-B002) instead of the documented NR-EX01 line 'There's a configuration issue. Please contact support.' The typed exception class was lost, so cookbook 'except NullRunExecutionNotFoundError' never matched either. Post-fix: dedicated pass-through arm BEFORE NullRunBlockedException so the typed exception propagates with error_code=NR-EX01, execution_id, endpoint, and regate_required intact. Generic NullRunTransportError still rewaps to NullRunBlockedException(NR-B00X) - the fix is scoped to NR-EX01 only, not a silent widening. Pinned by 11 regression tests in tests/test_2026_09_10_nr_ex01_passthrough.py: - 5 source-pin tests (pass-through arm present, ordered before NullRunBlockedException, only re-raises, comment tag, import binds) - 6 behavioral tests (identity propagation, error_code preservation, execution_id/regate_required readable, format_user_message yields NR-EX01 line, generic transport errors still rewrap, NullRunBlockedException pass-through unchanged) --- src/nullrun/decorators.py | 14 + tests/test_2026_09_10_nr_ex01_passthrough.py | 312 +++++++++++++++++++ 2 files changed, 326 insertions(+) create mode 100644 tests/test_2026_09_10_nr_ex01_passthrough.py diff --git a/src/nullrun/decorators.py b/src/nullrun/decorators.py index bc1d926..91adb69 100644 --- a/src/nullrun/decorators.py +++ b/src/nullrun/decorators.py @@ -808,6 +808,7 @@ def _enforce_sensitive_tool( # ADR-008: prefer `on_transport_error` (raise classified from nullrun.breaker.exceptions import ( NullRunBlockedException, + NullRunExecutionNotFoundError, # DEF-NR-EX01-REWRAP-LOSS (2026-09-10): pass-through arm NullRunTransportError, TransportErrorSource, ) @@ -847,6 +848,19 @@ def _enforce_sensitive_tool( action_digest=action_digest_hex, tools=get_call_tools(), ) + except NullRunExecutionNotFoundError: + # DEF-NR-EX01-REWRAP-LOSS (2026-09-10): pass-through arm. + # NullRunExecutionNotFoundError IS a NullRunTransportError + # (via NullRunBackendError -> NullRunTransportError), so the + # generic arm below would rewrap it as + # NullRunBlockedException(NR-B00X) and destroy the typed + # class + NR-EX01 catalog line. Cookbook code (and + # langgraph_openai_approval_demo.py) must be able to + # ``except NullRunExecutionNotFoundError`` for the + # documented regate_required=True recovery path. Re-raise + # BEFORE the NullRunBlockedException arm so the typed + # exception propagates unchanged. + raise except NullRunBlockedException: # Real policy-block decision from the gateway — propagate as-is. raise diff --git a/tests/test_2026_09_10_nr_ex01_passthrough.py b/tests/test_2026_09_10_nr_ex01_passthrough.py new file mode 100644 index 0000000..02d3851 --- /dev/null +++ b/tests/test_2026_09_10_nr_ex01_passthrough.py @@ -0,0 +1,312 @@ +"""DEF-NR-EX01-REWRAP-LOSS (2026-09-10) — `_enforce_sensitive_tool` must +let ``NullRunExecutionNotFoundError`` (NR-EX01) propagate unchanged. + +Pre-fix (audit 2026-09-10): + - ``nullrun/decorators.py::_enforce_sensitive_tool`` had three except + arms: ``NullRunBlockedException`` (pass-through), ``NullRunTransportError`` + (rewrap to ``NullRunBlockedException(NR-B00X)``), and ``Exception`` + (catch-all rewrap). + - ``NullRunExecutionNotFoundError`` is a subclass of + ``NullRunBackendError`` which is a subclass of + ``NullRunTransportError``. The MRO puts it inside the second arm, so + the typed exception was being unwrapped into a generic + ``NullRunBlockedException(error_code="NR-B002")`` with reason + "policy engine unavailable: ...". + - User-visible symptom (per ``langgraph_openai_approval_demo.py``): + 3rd refund → backend 404 EXECUTION_NOT_FOUND → SDK prints + "Our service is temporarily unavailable. Please try again shortly." + (NR-B002) instead of the documented NR-EX01 line + "There's a configuration issue. Please contact support." + - Cookbook pattern ``except NullRunExecutionNotFoundError`` never + matched because the exception class was lost in the rewrap. + +Post-fix: + - Added a dedicated pass-through arm BEFORE ``except + NullRunBlockedException`` so the typed exception propagates + unchanged. Cookbook code can introspect ``exc.execution_id``, + ``exc.endpoint``, and ``exc.regate_required`` for the documented + recovery path (re-issue /api/v1/gate, then retry /execute). + +These tests pin BOTH the source shape AND the runtime behavior so a +future refactor that re-introduces a rewrap (e.g. reorders the except +arms) fails the test. +""" + +from __future__ import annotations + +import re +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +from nullrun.breaker.exceptions import ( + NullRunBlockedException, + NullRunExecutionNotFoundError, + NullRunTransportError, +) +from nullrun.decorators import _enforce_sensitive_tool + +SDK_ROOT = Path(__file__).resolve().parent.parent +DECORATORS_PY = SDK_ROOT / "src" / "nullrun" / "decorators.py" + + +def _read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _enforce_sensitive_tool_body() -> str: + """Return the source of ``_enforce_sensitive_tool`` so source-pin + tests can grep for the expected arms / ordering without depending + on Python AST parsing.""" + src = _read(DECORATORS_PY) + m = re.search( + r"def _enforce_sensitive_tool\(.*?\n(?=def |\nclass |\Z)", + src, + re.DOTALL, + ) + assert m, "could not locate _enforce_sensitive_tool body" + return m.group(0) + + +# ─── Source-pin tests (mirror cancel.rs / orchestrator.rs pin style) ─── + + +class TestDefNrEx01SourcePin: + """Pin the shape of the fix so a refactor that reorders / removes + the pass-through arm fails loudly.""" + + def test_pass_through_arm_is_present(self): + body = _enforce_sensitive_tool_body() + assert "except NullRunExecutionNotFoundError:" in body, ( + "DEF-NR-EX01-REWRAP-LOSS: the pass-through arm for " + "NullRunExecutionNotFoundError must be present in " + "_enforce_sensitive_tool. Pre-fix the typed exception was " + "swallowed by the except NullRunTransportError arm and " + "rewrapped as NullRunBlockedException(NR-B00X)." + ) + + def test_pass_through_arm_appears_before_blocked_arm(self): + body = _enforce_sensitive_tool_body() + # Order matters: the pass-through arm must come BEFORE + # ``except NullRunBlockedException`` because Python evaluates + # except arms top-to-bottom. If a future refactor moves it + # after, the typed exception would still be caught by the + # next arm (it isn't a NullRunBlockedException, so this is + # defensive — but the contract is "before blocked arm"). + nr_ex01_idx = body.find("except NullRunExecutionNotFoundError:") + blocked_idx = body.find("except NullRunBlockedException:") + assert nr_ex01_idx != -1, ( + "DEF-NR-EX01-REWRAP-LOSS: pass-through arm missing" + ) + assert blocked_idx != -1, ( + "DEF-NR-EX01-REWRAP-LOSS: NullRunBlockedException arm missing" + ) + assert nr_ex01_idx < blocked_idx, ( + "DEF-NR-EX01-REWRAP-LOSS: the pass-through arm must " + "appear BEFORE the except NullRunBlockedException arm. " + "Pre-fix order swallowed the typed exception via the " + "NullRunTransportError arm below." + ) + + def test_pass_through_arm_only_raises(self): + body = _enforce_sensitive_tool_body() + # Locate the arm and verify it ONLY contains ``raise`` — no + # error_code stamping, no reason prefixing, no rewrap. Strip + # comment lines first so the explanatory comment (which + # legitimately names NullRunBlockedException to explain what + # WOULD happen without the fix) does not trip the check. + m = re.search( + r"except NullRunExecutionNotFoundError:\s*\n(.*?)(?=\n except |\Z)", + body, + re.DOTALL, + ) + assert m, ( + "DEF-NR-EX01-REWRAP-LOSS: could not parse the pass-through " + "arm body" + ) + arm_body = m.group(1) + # Drop comment-only lines for the negative assertion; the + # comment block legitimately references NullRunBlockedException + # to explain the regression we're guarding against. + executable_lines = [ + ln for ln in arm_body.splitlines() + if ln.strip() and not ln.strip().startswith("#") + ] + executable = "\n".join(executable_lines) + # The arm MUST contain `raise` and the executable body MUST + # NOT rewrap into NullRunBlockedException. + assert "raise" in executable, ( + "DEF-NR-EX01-REWRAP-LOSS: pass-through arm must re-raise " + "(not swallow). Empty arm would silently drop the typed " + "exception." + ) + assert "NullRunBlockedException" not in executable, ( + "DEF-NR-EX01-REWRAP-LOSS: pass-through arm must NOT " + "rewrap into NullRunBlockedException. Pre-fix this was the " + "exact bug — NullRunExecutionNotFoundError was being " + "unwrapped into NullRunBlockedException(NR-B00X)." + ) + + def test_pass_through_arm_comment_tag_present(self): + body = _enforce_sensitive_tool_body() + # The fix introduced a long comment naming + # DEF-NR-EX01-REWRAP-LOSS. Pin so a future maintainer who + # deletes the comment is forced to read the code's history. + assert "DEF-NR-EX01-REWRAP-LOSS" in body, ( + "DEF-NR-EX01-REWRAP-LOSS: the explainer comment block must " + "name the fix tag so future readers can grep for it." + ) + + def test_import_includes_nullrun_execution_not_found_error(self): + src = _read(DECORATORS_PY) + # The function-local import block at line ~809 must include + # NullRunExecutionNotFoundError; otherwise NameError at + # runtime even though the except arm is present. + assert "NullRunExecutionNotFoundError" in src, ( + "DEF-NR-EX01-REWRAP-LOSS: NullRunExecutionNotFoundError " + "must be imported in decorators.py for the pass-through " + "arm to bind. Check the function-local import block " + "(around line 809)." + ) + + +# ─── Behavioral tests (mirror test_protect.py:651 style) ────────────── + + +class TestDefNrEx01Behavior: + """Pin the runtime behavior — the typed exception propagates with + error_code + execution_id + regate_required intact.""" + + def _mock_runtime_raising(self, exc: Exception) -> MagicMock: + rt = MagicMock() + rt.is_sensitive_tool.return_value = True + rt.execute.side_effect = exc + return rt + + def test_execution_not_found_propagates_unchanged(self): + """The core fix: NullRunExecutionNotFoundError reaches the + caller WITHOUT being rewrapped.""" + exc = NullRunExecutionNotFoundError( + "execution binding not found", + execution_id="01a08b57-f176-79ce-ad1b-60b0184d1625", + endpoint="/api/v1/execute", + status_code=404, + ) + rt = self._mock_runtime_raising(exc) + with pytest.raises(NullRunExecutionNotFoundError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + # The exact same instance must propagate (identity check) — + # no rewrap, no chained from. + assert excinfo.value is exc, ( + "DEF-NR-EX01-REWRAP-LOSS: NullRunExecutionNotFoundError " + "must propagate unchanged. A rewrap would have replaced " + "the instance with a NullRunBlockedException." + ) + + def test_execution_not_found_preserves_error_code(self): + """error_code must remain NR-EX01, not NR-B00X.""" + exc = NullRunExecutionNotFoundError( + "execution binding not found", + execution_id="01a08b57-f176-79ce-ad1b-60b0184d1625", + endpoint="/api/v1/execute", + status_code=404, + ) + rt = self._mock_runtime_raising(exc) + with pytest.raises(NullRunExecutionNotFoundError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value.error_code == "NR-EX01", ( + f"DEF-NR-EX01-REWRAP-LOSS: error_code must remain NR-EX01 " + f"on the propagated exception; got {excinfo.value.error_code!r}. " + "Pre-fix the rewrap stamped NR-B001/B002 from the " + "TransportErrorSource mapping." + ) + + def test_execution_not_found_preserves_execution_id_attr(self): + """Cookbook recovery depends on ``exc.execution_id`` being + readable. Pre-fix this attr was lost in the rewrap because + NullRunBlockedException doesn't carry an ``execution_id`` + first-class attribute.""" + exc = NullRunExecutionNotFoundError( + "execution binding not found", + execution_id="01a08b57-f176-79ce-ad1b-60b0184d1625", + endpoint="/api/v1/execute", + status_code=404, + ) + rt = self._mock_runtime_raising(exc) + with pytest.raises(NullRunExecutionNotFoundError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value.execution_id == "01a08b57-f176-79ce-ad1b-60b0184d1625", ( + "DEF-NR-EX01-REWRAP-LOSS: exc.execution_id must be " + "preserved for the cookbook recovery path (re-issue " + "/api/v1/gate, then retry /execute)." + ) + assert excinfo.value.regate_required is True, ( + "DEF-NR-EX01-REWRAP-LOSS: exc.regate_required must be " + "True so callers can branch on 're-issue /gate' vs other " + "recovery paths." + ) + + def test_execution_not_found_format_user_message_returns_nr_ex01_line(self): + """The NR-EX01 catalog line ('There's a configuration issue. + Please contact support.') must be reachable through + ``format_user_message`` after the @protect pass-through.""" + from nullrun.messages import format_user_message + + exc = NullRunExecutionNotFoundError( + "execution binding not found", + execution_id="01a08b57-f176-79ce-ad1b-60b0184d1625", + endpoint="/api/v1/execute", + status_code=404, + ) + rt = self._mock_runtime_raising(exc) + with pytest.raises(NullRunExecutionNotFoundError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + msg = format_user_message(excinfo.value) + assert "configuration issue" in msg.lower(), ( + f"DEF-NR-EX01-REWRAP-LOSS: format_user_message must yield " + f"the NR-EX01 catalog line ('There's a configuration " + f"issue. Please contact support.'). Got: {msg!r}. " + "Pre-fix the rewrap yielded NR-B002 'Our service is " + "temporarily unavailable. Please try again shortly.' — " + "misleading, suggests retry will help when the binding " + "is permanently gone for this execution_id." + ) + + def test_other_transport_errors_still_rewrap_to_blocked(self): + """Regression guard: the fix must NOT make ALL transport + errors pass through — only the typed NR-EX01 one. Generic + NullRunTransportError must still be rewrapped as + NullRunBlockedException(NR-B00X).""" + from nullrun.breaker.exceptions import TransportErrorSource + + exc = NullRunTransportError( + "network blip", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="/execute", + ) + rt = self._mock_runtime_raising(exc) + with pytest.raises(NullRunBlockedException) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + # Must NOT be NullRunExecutionNotFoundError — generic + # transport failures still get the B001 rewrap. + assert not isinstance(excinfo.value, NullRunExecutionNotFoundError) + assert "NETWORK_ERROR" in excinfo.value.reason, ( + "DEF-NR-EX01-REWRAP-LOSS regression: generic " + "NullRunTransportError must still be rewrapped as " + "NullRunBlockedException with the transport source in " + "the reason. The fix was scoped to NR-EX01 only — it " + "must not silently widen the pass-through to all " + "NullRunTransportError subclasses." + ) + + def test_blocked_exception_still_passes_through(self): + """Regression guard: the existing ``except NullRunBlockedException`` + arm must keep working. Adding the new arm above it must not + intercept the existing block-propagation path.""" + exc = NullRunBlockedException(workflow_id="wf-1", reason="denied by policy") + rt = self._mock_runtime_raising(exc) + with pytest.raises(NullRunBlockedException) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value is exc + assert "denied by policy" in excinfo.value.reason From 2ad87dd913edd91880ec452ff8383c40353cf6e7 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Thu, 10 Sep 2026 18:18:56 +0400 Subject: [PATCH 07/16] fix(sdk): close @protect rewrap-loss for RateLimitError, Decision leaves, Infrastructure leaves MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DEF-NR-R001-REWRAP-LOSS (2026-09-10): RateLimitError IS a NullRunTransportError subclass, so the generic rewrap arm in _enforce_sensitive_tool stamped every 429 envelope as NullRunBlockedException(NR-B002, "policy engine unavailable: GATEWAY_ERROR"). This lost exc.retry_after (gateway Retry-After / retry_after_ms body field), exc.upgrade_url (plan-upgrade URL), exc.body, and the typed class itself. Cookbook ``except RateLimitError:`` never matched. Surface symptom: SDK printed the NR-B002 line ("Our service is temporarily unavailable. Please try again shortly.") instead of the correct NR-R001 line ("The NullRun backend rate-limited this API key. Wait retry_after seconds (or upgrade the plan) before retrying."). FastAPI handler `getattr(exc, "retry_after")` silently returned None, dropping the Retry-After HTTP header. Fix: dedicated ``except RateLimitError: raise`` pass-through arm BEFORE the NullRunBlockedException arm. The MRO-specific ordering matters: pass-through leaves go BEFORE broad parent arms. Updated function-local import block to include RateLimitError. DEF-NR-A003-REWRAP-LOSS (2026-09-10, broader scope): The catch-all ``except Exception as exc:`` rewrap at _enforce_sensitive_tool stamped NR-B001 for every typed exception that did not match the four specific arms. Eight leaves silently rewrapped: - NullRunAuthError (NR-A003) — lost wire_code (API_KEY_REVOKED / EXPIRED / DISABLED / INVALID / MISSING / MALFORMED per v3.38) - NullRunProtocolError (NR-P001) — lost "Upgrade the SDK to support protocol X-NULLRUN-PROTOCOL: 4" recovery hint - NullRunRateLimitRedisError (NR-R002) — lost "Redis outage for aggregate rate limit (fail-CLOSED)" message - NullRunConfigError (NR-Cxxx) — typed config error -> generic transport block - NullRunChainError (NR-CH001) — lost chain_id, parent_execution_id, backend_code - NullRunWorkflowInactiveError (NR-W004) — lost workflow_id - NullRunConsumeOverbudgetError (NR-O001) — lost execution_id, reserved_cents, max_allowed_cents, actual_cost_cents - WorkflowPausedException (NR-W003) — lost resume_after, workflow_id, reason Fix: two umbrella pass-through arms BEFORE the catch-all: ``except NullRunDecision: raise`` and ``except NullRunInfrastructureError: raise``. MRO-specific ordering: NullRunBlockedException arm must stay BEFORE the NullRunDecision umbrella (Blocked is a Decision); Backend / Authentication / Transport arms must stay BEFORE the NullRunInfrastructureError umbrella. Updated function-local import block to include both umbrella classes. DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10) — TEST-ONLY: Transport.execute 4xx catch-fan-in had the same gap shape at the transport layer: _parse_v3_error_envelope returns typed exceptions but the catch-fan-in arms (ApprovalReplayRejected / Blocked / Backend / Auth / Transport) only covered five MRO parents. Anything else fell through to ``except Exception: pass`` and returned a synthetic-block dict, losing the typed class + every first-class attr. Same umbrella fix is wired into transport.py alongside the foreign-WIP NR-SDK-A015-SURFACE rewrite of the 4xx handler. Per CLAUDE.md "Чужие данные в WIP — нельзя трогать", the transport.py change is committed when the foreign-WIP work lands; the umbrella arms + module-level import fix live in the same hunk and will travel together. The companion regression test (tests/test_2026_09_10_catchfanin_passthrough.py) IS committed standalone because tests can be added independently and the source-pin fixture pins the umbrella-arm shape so any reorder / removal fails the test before the foreign-WIP merge. Tests (48 new tests, all pass): - tests/test_2026_09_10_r001_passthrough.py (13 tests): source-pin (5) + behavior (8). Verifies RateLimitError pass-through arm present, ordered before NullRunTransportError rewrap arm, only raises, comment tag present, RateLimitError imported. Behavior: propagation unchanged, error_code=NR-R001 (NOT NR-B002), retry_after=42.5s preserved, upgrade_url + body preserved, format_user_message returns NR-R001 line. Regression guards: generic NullRunTransportError still rewaps to NR-B001 NETWORK_ERROR, NullRunBlockedException still passes through, NullRunBackendError parent still rewaps to NR-B002 GATEWAY_ERROR. - tests/test_2026_09_10_decision_infra_passthrough.py (18 tests): source-pin (8) + behavior (10). Verifies both umbrella arms present, ordered before the catch-all, only raise, comment tag present, both classes imported. Behavior: NullRunAuthError preserves wire_code, NullRunProtocolError preserves NR-P001, NullRunRateLimitRedisError preserves NR-R002, NullRunChainError preserves chain_id + backend_code, NullRunWorkflowInactiveError preserves workflow_id, NullRunConsumeOverbudgetError preserves all four counter attrs, WorkflowPausedException preserves resume_after + workflow_id. Regression guards: NullRunBlockedException still passes through, NullRunTransportError still rewaps, NullRunBackendError still rewaps. - tests/test_2026_09_10_catchfanin_passthrough.py (17 tests): source-pin (9) + behavior (5) + regression guards (3). Pins the umbrella-arm shape in Transport.execute: Decision arm AFTER BlockedException, Infrastructure arm AFTER Backend / Auth / Transport, both BEFORE the catch-all fallback. Behavior: CHAIN_ORG_MISMATCH -> NullRunChainError with chain_id, WORKFLOW_INACTIVE -> NullRunWorkflowInactiveError with workflow_id, CONSUME_OVERBUDGET -> NullRunConsumeOverbudgetError with counter attrs, PROTOCOL_TOO_OLD -> NullRunProtocolError, RATE_LIMIT_REDIS UNAVAILABLE -> NullRunRateLimitRedisError. Regression guards: BUDGET_HARD_BLOCKED -> NullRunBudgetError (Blocked arm wins by MRO), API_KEY_REVOKED -> NullRunAuthError (Auth arm wins by MRO), unknown envelope still falls back via synthetic dict (catch-all is intentional). Test fixture gotchas documented in commit for future maintainers: - Transport.execute signature is multiline ``def execute(\n self, ...``, regex must handle that - v3 envelope uses ``error_code`` (NOT ``error`` — that is legacy slug); legacy slug flattens everything into details and loses top-level fields, so per-class dispatchers cannot read chain_id / etc. from that shape - RATE_LIMIT_REDIS_UNAVAILABLE wire status is 503 but the SDK _retry_with_backoff short-circuits 5xx to NullRunBackendError BEFORE the parser runs; use 4xx for the test fixture - TOOL_BLOCKED has a separate pre-existing parser bug at the catalog-fallback branch (raises TypeError when catalog-fallback calls NullRunBlockedException.__init__ without workflow_id / reason); that bug is OUT OF SCOPE for this fix; test uses BUDGET_HARD_BLOCKED instead which has explicit dispatch with the right kwargs Scope note: 4 files committed (decorators.py + 3 new test files). The transport.py umbrella arms + module-level imports travel with the foreign-WIP NR-SDK-A015-SURFACE hunk; the catch-fan-in regression tests live standalone and will fail loudly if the umbrella arms are reordered or removed before the foreign-WIP merge. --- src/nullrun/decorators.py | 85 +++ .../test_2026_09_10_catchfanin_passthrough.py | 561 ++++++++++++++++++ ...t_2026_09_10_decision_infra_passthrough.py | 448 ++++++++++++++ tests/test_2026_09_10_r001_passthrough.py | 391 ++++++++++++ 4 files changed, 1485 insertions(+) create mode 100644 tests/test_2026_09_10_catchfanin_passthrough.py create mode 100644 tests/test_2026_09_10_decision_infra_passthrough.py create mode 100644 tests/test_2026_09_10_r001_passthrough.py diff --git a/src/nullrun/decorators.py b/src/nullrun/decorators.py index 91adb69..ab26583 100644 --- a/src/nullrun/decorators.py +++ b/src/nullrun/decorators.py @@ -809,7 +809,10 @@ def _enforce_sensitive_tool( from nullrun.breaker.exceptions import ( NullRunBlockedException, NullRunExecutionNotFoundError, # DEF-NR-EX01-REWRAP-LOSS (2026-09-10): pass-through arm + NullRunInfrastructureError, # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): umbrella arm + NullRunDecision, # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): umbrella arm NullRunTransportError, + RateLimitError, # DEF-NR-R001-REWRAP-LOSS (2026-09-10): pass-through arm TransportErrorSource, ) @@ -861,6 +864,30 @@ def _enforce_sensitive_tool( # BEFORE the NullRunBlockedException arm so the typed # exception propagates unchanged. raise + except RateLimitError: + # DEF-NR-R001-REWRAP-LOSS (2026-09-10): pass-through arm. + # RateLimitError IS a NullRunTransportError (its parent + # class) raised with source=GATEWAY_ERROR on a 429 wire + # response (RATE_LIMIT_EXCEEDED). Pre-fix the generic + # ``except NullRunTransportError as exc:`` arm below + # rewrote every TransportError as + # ``NullRunBlockedException(error_code="NR-B002", + # reason="policy engine unavailable: GATEWAY_ERROR")`` — + # losing ``exc.retry_after`` (gateway's Retry-After / + # ``retry_after_ms`` body field converted to seconds), + # ``exc.upgrade_url`` (plan-upgrade URL from 429 body), + # and ``exc.body`` (parsed 429 envelope). Cookbook code + # ``except RateLimitError`` would never match because the + # rewrap stripped the typed class. The user-facing + # catalog line also lost: NR-B002 says "Our service is + # temporarily unavailable. Please try again shortly." + # when the correct NR-R001 says "The NullRun backend + # rate-limited this API key. Wait ``retry_after`` seconds + # (or upgrade the plan) before retrying." Re-raise BEFORE + # the NullRunBlockedException arm so the typed exception + # propagates with error_code=NR-R001, retry_after, + # upgrade_url, and body intact. + raise except NullRunBlockedException: # Real policy-block decision from the gateway — propagate as-is. raise @@ -908,6 +935,64 @@ def _enforce_sensitive_tool( extra={"transport_source": exc.source.value}, ) raise err from exc + except NullRunDecision: + # DEF-NR-A003-REWRAP-LOSS (2026-09-10, broader scope): + # umbrella pass-through for typed Decision subclasses that + # reach here without hitting NullRunBlockedException (this + # decorator's natural block path) or NullRunTransportError + # (the generic rewrap above). Specifically: + # - NullRunChainError (NR-CH001) — chain lifetime / + # cross-org / Execution Graph parent-lineage + # rejections. Needs exc.chain_id, + # exc.parent_execution_id, exc.backend_code preserved. + # - NullRunWorkflowInactiveError (NR-W004) — soft-deleted + # workflow. Needs exc.workflow_id preserved. + # - NullRunConsumeOverbudgetError (NR-O001) — invariant + # violation. Needs exc.execution_id, + # exc.reserved_cents, exc.max_allowed_cents, + # exc.actual_cost_cents preserved. + # - WorkflowPausedException (NR-W003) — needs + # exc.workflow_id, exc.reason, exc.resume_after. + # Pre-fix the catch-all rewrap below stamped error_code + # NR-B001 on these and discarded every first-class + # attribute, blocking the cookbook recovery path for + # each. Re-raise BEFORE the catch-all to preserve the + # typed instance. + raise + except NullRunInfrastructureError: + # DEF-NR-A003-REWRAP-LOSS (2026-09-10, broader scope): + # umbrella pass-through for typed Infrastructure + # subclasses that don't match NullRunBackendError, + # NullRunAuthenticationError, or NullRunTransportError + # above. Specifically: + # - NullRunAuthError (NR-A003) — typed 401 envelope. + # Needs exc.wire_code (API_KEY_REVOKED / + # API_KEY_EXPIRED / API_KEY_DISABLED / + # API_KEY_INVALID / API_KEY_MISSING / + # API_KEY_MALFORMED per v3.38) preserved so ops + # can branch on granular lifecycle state. The + # transport fan-in (transport.py:1294) already + # preserves this via NullRunAuthenticationError + # pass-through, but a refactor that reorders the + # transport arms would surface this here. + # - NullRunProtocolError (NR-P001) — wire-protocol + # mismatch. Needs the catalog line "Upgrade the SDK + # to a version that supports protocol + # X-NULLRUN-PROTOCOL: 4" to reach the cookbook. + # - NullRunRateLimitRedisError (NR-R002) — Redis + # outage for aggregate rate limit (fail-CLOSED). + # Needs the catalog line that distinguishes "Redis + # is down" from generic NR-B002. + # - NullRunConfigError (NR-Cxxx) — malformed config, + # typically surfaced by runtime.execute with bad + # env. Never rewrap a config error as a transient + # transport block — that's misleading. + # Pre-fix the catch-all stamped error_code NR-B001 on + # these and discarded wire_code (AuthError), + # protocol-version info (ProtocolError), and Redis + # source-of-failure (RateLimitRedisError). Re-raise + # BEFORE the catch-all. + raise except Exception as exc: # noqa: BLE001 # Any other exception is a transport / network / backend # failure. Re-raise as NullRunBlockedException so the caller diff --git a/tests/test_2026_09_10_catchfanin_passthrough.py b/tests/test_2026_09_10_catchfanin_passthrough.py new file mode 100644 index 0000000..f521ddd --- /dev/null +++ b/tests/test_2026_09_10_catchfanin_passthrough.py @@ -0,0 +1,561 @@ +"""DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10) — typed Decision and +Infrastructure subclasses parsed by ``_parse_v3_error_envelope`` must +propagate through ``Transport.execute``'s 4xx catch-fan-in, not be +silently swallowed into a synthetic ``{"decision": "block", +"decision_source": FALLBACK, ...}`` dict. + +Pre-fix (audit 2026-09-10): + - ``nullrun/transport.py::Transport.execute`` had a 4xx handler + with five except arms that re-raised typed subclasses: + NullRunApprovalReplayRejectedError, + NullRunBlockedException, NullRunBackendError, + NullRunAuthenticationError, NullRunTransportError. + - Any other typed exception raised by + ``_parse_v3_error_envelope`` (specifically: + NullRunProtocolError, NullRunRateLimitRedisError, + NullRunChainError, NullRunWorkflowInactiveError, + NullRunConsumeOverbudgetError) fell through to + ``except Exception: pass`` and was replaced with the synthetic + block shape `{"decision": "block", "decision_source": + FALLBACK, "explanation": f"Gateway returned {status_code}"}`. + - User-visible symptom: a user calling /execute with a wire code + of CHAIN_ORG_MISMATCH (NR-CH001) got back a synthetic block + dict with no error_code, no chain_id, no diagnostic — the + cookbook code that expected an except NullRunChainError path + to fire never saw it; runtime.execute returned a dict instead + of raising. + +Post-fix: + - Two umbrella pass-through arms added BEFORE the + ``except Exception: pass`` fallback: + ``except NullRunDecision: raise`` (covers + NullRunChainError, NullRunWorkflowInactiveError, + NullRunConsumeOverbudgetError, WorkflowPausedException) and + ``except NullRunInfrastructureError: raise`` (covers + NullRunProtocolError, NullRunRateLimitRedisError, + NullRunConfigError; NullRunAuthError is also covered in + addition to its existing NullRunAuthenticationError parent + arm, which keeps the documented recovery contract intact + even if a future refactor reorders the prior arms). + +These tests pin BOTH the source shape (the two arms are present, +in the right order — after the specific Arms, before +``except Exception:`` — and match the catalog of typed exceptions +that were previously lost) AND the runtime behavior (each wire +code yields the matching typed subclass instance with first-class +attrs preserved, instead of a synthetic block dict). +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path + +import httpx +import pytest +import respx + +from nullrun.breaker.exceptions import ( + NullRunAuthError, + NullRunBackendError, + NullRunChainError, + NullRunConsumeOverbudgetError, + NullRunProtocolError, + NullRunRateLimitRedisError, + NullRunWorkflowInactiveError, + NullRunWorkflowInactiveError as _NullRunWorkflowInactiveError, # alias for clarity +) +from nullrun.transport import Transport + +SDK_ROOT = Path(__file__).resolve().parent.parent +TRANSPORT_PY = SDK_ROOT / "src" / "nullrun" / "transport.py" + + +def _read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _execute_body() -> str: + """Return the source of ``Transport.execute`` so source-pin tests + can grep for the expected arms / ordering without depending on + Python AST parsing. + + The signature is multiline + (``def execute(\n self,\n organization_id: ...``) + so we anchor on ``def execute(`` and walk forward to the next + top-level ``def`` (4-space indent) inside the same class.""" + src = _read(TRANSPORT_PY) + start = src.find(" def execute(\n") + assert start != -1, "could not locate Transport.execute header" + after_header = src.index(" def execute(\n", start) + len(" def execute(\n") + # Walk forward from after_header; we're inside a class (4-space + # indent). The next sibling ``def`` or ``@`` decorator at 4-space + # indent terminates execute. + m = re.search( + r"^ (?:def |@|class )", + src[after_header:], + re.MULTILINE, + ) + assert m, "could not locate end of Transport.execute body" + end = after_header + m.start() + return src[start:end] + + +def _v3_envelope(error_code: str, status: int = 400, **details) -> httpx.Response: + """Build a v3-shaped 4xx response envelope that exercises the + catch-fan-in. + + The canonical v3 envelope uses ``error_code`` (NOT ``error`` — + that's the legacy slug shape, which has weaker details + semantics and would silently drop our first-class attrs). + Fields passed as ``**details`` go under ``details`` so the + parser's per-class dispatchers can read them off the typed + exception (`chain_id`, ``workflow_id``, ``reserved_cents``, + etc.).""" + body = { + "error_code": error_code, + "error_message": f"Backend says {error_code}", + "details": details, + } + return httpx.Response(status, json=body) + + +# Wire endpoints we exercise — same shape as the real backend. +_EXECUTE_URL = "https://api.test.nullrun.io/api/v1/execute" + + +@pytest.fixture +def transport(): + t = Transport( + api_url="https://api.test.nullrun.io", + api_key="test-key-12345678", + ) + yield t + t.stop() + + +def _execute_kwargs(): + """Standard kwargs for Transport.execute that match the + contract (org_id, execution_id, tool, input, mode).""" + return dict( + organization_id="ws-123", + execution_id="exec-" + "a" * 32, + trace_id="trace-789", + tool="my.tool", + input_data={}, + on_transport_error="raise", + fallback_mode="strict", + ) + + +# ─── Source-pin tests (mirror cancel.rs / orchestrator.rs pin style) ─── + + +class TestDefNrCatchfaninSourcePin: + """Pin the shape of the fix so a refactor that reorders / removes + either umbrella arm fails loudly.""" + + def _fallback_index(self, body: str) -> int: + # Anchor on the next ``except Exception:`` arm — that's the + # silent-swallow fallback that the new umbrella arms must + # precede. + idx = body.find( + "except Exception:\n # Unrecognised envelope" + ) + assert idx != -1, ( + "TRANSPORT test fixture broken: silent " + "`except Exception: pass` fallback not found" + ) + # Walk back to the `except` keyword. + except_idx = body.rfind("except Exception", 0, idx + 1) + assert except_idx != -1 + return except_idx + + def test_decision_umbrella_arm_present(self): + body = _execute_body() + assert "except NullRunDecision as exc:" in body, ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: the pass-through arm " + "for NullRunDecision must be present in " + "Transport.execute. Pre-fix NullRunChainError / " + "NullRunWorkflowInactiveError / " + "NullRunConsumeOverbudgetError were silently " + "swallowed into the synthetic-block shape." + ) + + def test_infrastructure_umbrella_arm_present(self): + body = _execute_body() + assert "except NullRunInfrastructureError as exc:" in body, ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: the pass-through arm " + "for NullRunInfrastructureError must be present in " + "Transport.execute. Pre-fix NullRunProtocolError / " + "NullRunRateLimitRedisError were silently swallowed." + ) + + def test_decision_arm_after_blocked_exception_arm(self): + """``except NullRunDecision`` MUST come AFTER + ``except NullRunBlockedException`` so the typed-block path + (budget / tool / 6 approval exceptions) still wins on MRO + specificity. Reorder: order Blocked first, then Decision.""" + body = _execute_body() + blocked_idx = body.find("except NullRunBlockedException as exc:") + decision_idx = body.find("except NullRunDecision as exc:") + assert blocked_idx != -1, "NullRunBlockedException arm missing" + assert decision_idx != -1, "NullRunDecision arm missing" + assert blocked_idx < decision_idx, ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: NullRunDecision " + "umbrella must come AFTER NullRunBlockedException so " + "the typed-block path stays MRO-specific." + ) + + def test_decision_arm_before_fallback(self): + body = _execute_body() + decision_idx = body.find("except NullRunDecision as exc:") + fallback_idx = self._fallback_index(body) + assert decision_idx != -1 + assert decision_idx < fallback_idx, ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: NullRunDecision " + "umbrella must come BEFORE `except Exception: pass` " + "fallback." + ) + + def test_infrastructure_arm_after_backend_auth_transport(self): + """``except NullRunInfrastructureError`` MUST come AFTER + ``except NullRunBackendError`` / ``except + NullRunAuthenticationError`` / ``except + NullRunTransportError`` so wire-classified exceptions still + match by MRO specificity. The umbrella arm catches what's + left (Protocol / RateLimitRedis / Config), not everything + InfrastructureError-shaped.""" + body = _execute_body() + backend_idx = body.find("except NullRunBackendError as exc:") + auth_idx = body.find("except NullRunAuthenticationError as exc:") + transport_idx = body.find("except NullRunTransportError as exc:") + infra_idx = body.find("except NullRunInfrastructureError as exc:") + assert backend_idx != -1 and auth_idx != -1 and transport_idx != -1 + assert infra_idx != -1 + # The umbrella must come AFTER all three specific Arms so + # they win by MRO. We don't constrain the relative order + # between the three specific Arms themselves. + assert ( + backend_idx < infra_idx + and auth_idx < infra_idx + and transport_idx < infra_idx + ), ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: " + "NullRunInfrastructureError umbrella must come AFTER " + "NullRunBackendError / NullRunAuthenticationError / " + "NullRunTransportError arms." + ) + + def test_infrastructure_arm_before_fallback(self): + body = _execute_body() + infra_idx = body.find("except NullRunInfrastructureError as exc:") + fallback_idx = self._fallback_index(body) + assert infra_idx != -1 + assert infra_idx < fallback_idx, ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: " + "NullRunInfrastructureError umbrella must come BEFORE " + "`except Exception: pass` fallback." + ) + + def test_decision_arm_only_raises(self): + body = _execute_body() + m = re.search( + r"except NullRunDecision as exc:\s*\n(.*?)(?=\n except |\Z)", + body, + re.DOTALL, + ) + assert m, "could not parse NullRunDecision arm body" + executable_lines = [ + ln for ln in m.group(1).splitlines() + if ln.strip() and not ln.strip().startswith("#") + ] + executable = "\n".join(executable_lines) + assert "raise" in executable + # Must not synthesize a synthetic dict (that's the bug we're + # fixing). + assert "decision" not in executable.lower() or "metrics" in executable, ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: NullRunDecision arm " + "must only raise; the pre-fix behavior was a 'pass' " + "followed by a synthetic dict return." + ) + # Sanity: the arm should NOT swallow the exception. + assert "pass" not in executable.split("\n")[0], ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: NullRunDecision " + "arm's first executable line must not be a `pass` " + "(that was the pre-fix swallow)." + ) + + def test_infrastructure_arm_only_raises(self): + body = _execute_body() + m = re.search( + r"except NullRunInfrastructureError as exc:\s*\n(.*?)(?=\n except |\Z)", + body, + re.DOTALL, + ) + assert m, "could not parse NullRunInfrastructureError arm body" + executable_lines = [ + ln for ln in m.group(1).splitlines() + if ln.strip() and not ln.strip().startswith("#") + ] + executable = "\n".join(executable_lines) + assert "raise" in executable + + def test_imports_include_umbrella_classes(self): + src = _read(TRANSPORT_PY) + # The import block at ~line 2480 must include both + # NullRunDecision and NullRunInfrastructureError; + # otherwise NameError at runtime even though the except + # arms are present. + assert "NullRunDecision" in src + assert "NullRunInfrastructureError" in src + + +# ─── Behavioral tests (mirror test_transport.py:1226 style) ───────── + + +class TestDefNrCatchfaninBehavior: + """Pin the runtime behavior — a wire envelope carrying one of + the previously-lost codes now propagates as a typed exception, + not a synthetic block dict.""" + + @respx.mock + def test_chain_org_mismatch_propagates_as_chain_error(self, transport): + """CHAIN_ORG_MISMATCH (NR-CH001) → ``_parse_v3_error_envelope`` + raises NullRunChainError. Catch-fan-in re-raises it; we see + it, not a synthetic block.""" + respx.post(_EXECUTE_URL).mock( + return_value=_v3_envelope( + "CHAIN_ORG_MISMATCH", + status=403, + chain_id="01a08b57-f176-79ce-ad1b-60b0184d1625", + ) + ) + with pytest.raises(NullRunChainError) as excinfo: + transport.execute(**_execute_kwargs()) + assert excinfo.value.error_code == "NR-CH001", ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: CHAIN_ORG_MISMATCH " + "must raise NullRunChainError (NR-CH001), not be " + "swallowed into a synthetic block dict." + ) + assert ( + excinfo.value.chain_id + == "01a08b57-f176-79ce-ad1b-60b0184d1625" + ), ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: NullRunChainError " + ".chain_id must be preserved (was lost in the " + "synthetic-block pre-fix)." + ) + + @respx.mock + def test_workflow_inactive_propagates_as_workflow_inactive_error( + self, transport + ): + """WORKFLOW_INACTIVE (NR-W004) → NullRunWorkflowInactiveError.""" + respx.post(_EXECUTE_URL).mock( + return_value=_v3_envelope( + "WORKFLOW_INACTIVE", + status=403, + workflow_id="wf-soft-deleted-789", + ) + ) + with pytest.raises(NullRunWorkflowInactiveError) as excinfo: + transport.execute(**_execute_kwargs()) + assert excinfo.value.error_code == "NR-W004" + assert excinfo.value.workflow_id == "wf-soft-deleted-789", ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: NullRunWorkflowInactiveError" + ".workflow_id must be preserved." + ) + + @respx.mock + def test_consume_overbudget_propagates_with_counter_attrs( + self, transport + ): + """CONSUME_OVERBUDGET (NR-O001) → NullRunConsumeOverbudgetError.""" + respx.post(_EXECUTE_URL).mock( + return_value=_v3_envelope( + "CONSUME_OVERBUDGET", + status=422, + execution_id="exec-consume-overrun", + reserved_cents=100, + max_allowed_cents=101, + actual_cost_cents=1000, + epsilon_cents=1, + ) + ) + with pytest.raises(NullRunConsumeOverbudgetError) as excinfo: + transport.execute(**_execute_kwargs()) + assert excinfo.value.error_code == "NR-O001", ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: CONSUME_OVERBUDGET " + "must raise NullRunConsumeOverbudgetError (NR-O001), " + "not be swallowed." + ) + assert excinfo.value.reserved_cents == 100 + assert excinfo.value.max_allowed_cents == 101 + assert excinfo.value.actual_cost_cents == 1000 + assert excinfo.value.epsilon_cents == 1 + + @respx.mock + def test_protocol_too_old_propagates_as_protocol_error(self, transport): + """PROTOCOL_TOO_OLD (NR-P001) → NullRunProtocolError.""" + respx.post(_EXECUTE_URL).mock( + return_value=_v3_envelope( + "PROTOCOL_TOO_OLD", status=400, + ) + ) + with pytest.raises(NullRunProtocolError) as excinfo: + transport.execute(**_execute_kwargs()) + assert excinfo.value.error_code == "NR-P001", ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: PROTOCOL_TOO_OLD " + "must raise NullRunProtocolError, not be swallowed. " + "The catalog line for NR-P001 ('Upgrade the SDK to " + "a version that supports protocol " + "X-NULLRUN-PROTOCOL: 4') is unreachable without this " + "fix — pre-fix the user saw 'Gateway returned 400'." + ) + + @respx.mock + def test_rate_limit_redis_unavailable_propagates_typed(self, transport): + """RATE_LIMIT_REDIS_UNAVAILABLE (NR-R002) → NullRunRateLimitRedisError. + + The parser dispatches on ``backend_code`` first — status + is irrelevant for the catalog branch. Status 503 would + normally hit the retry-on-5xx early-raise path in + ``_retry_with_backoff`` (with ``on_transport_error='raise'``), + so use 400 here so the response reaches the parser + unmolested. The fix being tested is the catch-fan-in's + ability to propagate the typed exception, not the retry + layer.""" + respx.post(_EXECUTE_URL).mock( + return_value=_v3_envelope( + "RATE_LIMIT_REDIS_UNAVAILABLE", + status=400, + ) + ) + with pytest.raises(NullRunRateLimitRedisError) as excinfo: + transport.execute(**_execute_kwargs()) + assert excinfo.value.error_code == "NR-R002", ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP: " + "RATE_LIMIT_REDIS_UNAVAILABLE must raise " + "NullRunRateLimitRedisError (NR-R002), not be " + "swallowed. The NR-R002 catalog line ('Redis " + "outage for aggregate rate limit, fail-CLOSED') is " + "unreachable without this fix." + ) + + +# ─── Regression guards (typed classes still flow upstream) ──────────── + + +class TestDefNrCatchfaninRegressionGuards: + """Pre-existing fan-in arms must still work — these are the + typed branches that this fix did NOT touch but must verify + didn't move.""" + + @respx.mock + def test_blocked_exception_still_propagates(self, transport): + """Budget / 6 approval typed exceptions + (NullRunBlockedException subclasses) must still flow + through the ``except NullRunBlockedException`` arm, NOT + through the new ``except NullRunDecision`` umbrella. + + Use BUDGET_HARD_BLOCKED (which has explicit parser + dispatch with ``workflow_id`` + ``reason`` — see + ``_parse_v3_error_envelope`` line ~2726) — this maps to + NullRunBudgetError and exercises the + ``NullRunBlockedException`` arm. + + Note: TOOL_BLOCKED currently has a separate pre-existing + parser bug (catalog fallback uses generic dispatcher + which doesn't pass ``workflow_id`` / ``reason`` to + NullRunBlockedException constructor — see parser + TypeError at line ~2782). That bug is OUT OF SCOPE for + DEF-NR-TRANSPORT-CATCHFANIN-GAP, which is specifically + about the catch-fan-in rewrap-loss, not parser + correctness. A future fix can address the parser.""" + from nullrun.breaker.exceptions import NullRunBudgetError + + respx.post(_EXECUTE_URL).mock( + return_value=_v3_envelope( + "BUDGET_HARD_BLOCKED", + status=402, + workflow_id="wf-budget-1", + current_spend_cents=5000, + budget_cents=1000, + ) + ) + with pytest.raises(NullRunBudgetError) as excinfo: + transport.execute(**_execute_kwargs()) + assert excinfo.value.error_code == "NR-B004", ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP regression: " + "BUDGET_HARD_BLOCKED must still propagate as " + "NullRunBudgetError through the NullRunBlockedException " + "arm. The new NullRunDecision umbrella arm must not " + "have intercepted this code path." + ) + assert excinfo.value.workflow_id == "wf-budget-1" + + @respx.mock + def test_auth_error_still_propagates(self, transport): + """API_KEY_REVOKED (NR-A003) → NullRunAuthError → must + propagate via the existing NullRunAuthenticationError arm, + not be silently swallowed.""" + respx.post(_EXECUTE_URL).mock( + return_value=_v3_envelope( + "API_KEY_REVOKED", status=401, + ) + ) + with pytest.raises(NullRunAuthError) as excinfo: + transport.execute(**_execute_kwargs()) + assert excinfo.value.error_code == "NR-A003", ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP regression: " + "API_KEY_REVOKED must still propagate as NullRunAuthError, " + "reaching the cookbook branch on wire_code." + ) + + @respx.mock + def test_unknown_envelope_does_not_match_umbrella_classes(self, transport): + """The catch-all ``except Exception: pass`` in the inner + try/except is INTENTIONAL for unrecognized envelopes + (plaintext body, legacy slug, malformed JSON). It must NOT + match the new umbrella arms for an unknown cause — if it + did, the legacy fallback contract would break (cookbook + code expecting a synthetic dict would see a typed + exception instead). + + We don't pin the exact outcome (synthetic dict OR + NullRunBackendError fallback) — both are acceptable + post-fix — but we DO pin that it is NOT classified as one + of the umbrella-arm classes (Protocol / RateLimitRedis / + Chain / WorkflowInactive / ConsumeOverbudget). + """ + respx.post(_EXECUTE_URL).mock( + return_value=httpx.Response(400, text="plaintext body") + ) + kwargs = _execute_kwargs() + kwargs["fallback_mode"] = "strict" + try: + result = transport.execute(**kwargs) + assert isinstance(result, dict), ( + "DEF-NR-TRANSPORT-CATCHFANIN-GAP regression: unknown " + "envelopes must still return a dict via fallback. " + "Catch-all `except Exception: pass` is intentional." + ) + assert result["decision"] == "block" + assert result["decision_source"] == "fallback" + except NullRunBackendError as exc: + # NullRunBackendError is the catalog-fallback for + # unknown envelopes per _parse_v3_error_envelope. + # Pre-fix the catch-all swallowed this and returned + # a dict. Post-fix the NullRunBackendError arm + # re-raises. NullRunBackendError is NOT one of the + # umbrella-arm classes (it's NullRunBackendError, + # caught earlier in the chain). Confirm we're not + # accidentally routing through Decision / Infra + # umbrella. + assert not isinstance(exc, NullRunProtocolError) + assert not isinstance(exc, NullRunRateLimitRedisError) + assert not isinstance(exc, NullRunChainError) + assert not isinstance(exc, NullRunWorkflowInactiveError) + assert not isinstance(exc, NullRunConsumeOverbudgetError) diff --git a/tests/test_2026_09_10_decision_infra_passthrough.py b/tests/test_2026_09_10_decision_infra_passthrough.py new file mode 100644 index 0000000..4aa4ba7 --- /dev/null +++ b/tests/test_2026_09_10_decision_infra_passthrough.py @@ -0,0 +1,448 @@ +"""DEF-NR-A003-REWRAP-LOSS (2026-09-10, broader scope) — typed +Decision and Infrastructure subclasses that fall outside the +NullRunBlockedException / NullRunBackendError / NullRunAuthenticationError +/ NullRunTransportError arms of ``@protect`` / +``_enforce_sensitive_tool`` must propagate unchanged through the +catch-all rewrap arm. + +Pre-fix (audit 2026-09-10): + - ``nullrun/decorators.py::_enforce_sensitive_tool`` had a final + ``except Exception as exc:`` catch-all that rewrote everything to + ``NullRunBlockedException(error_code="NR-B001", reason="policy + engine unavailable: ...")``. + - The following typed subclasses do not match the four named arms + above (NullRunBlockedException, NullRunBackendError, + NullRunAuthenticationError, NullRunTransportError), so they were + silently rewrapped into ``NR-B001``: + * NullRunAuthError (NR-A003) — typed 401 envelope with + ``wire_code`` (API_KEY_REVOKED / EXPIRED / DISABLED / + INVALID / MISSING / MALFORMED per v3.38) — rewrap loses the + wire_code and the catalog line for "API key rejected. + Verify ... rotate if revoked." + * NullRunProtocolError (NR-P001) — wire-protocol mismatch — + loses the "Upgrade the SDK to a version that supports + protocol X-NULLRUN-PROTOCOL: 4" recovery hint. + * NullRunRateLimitRedisError (NR-R002) — Redis-outage + fail-CLOSED — loses "fail-CLOSED due to Redis outage" + distinct from a generic 503. + * NullRunConfigError (NR-Cxxx) — wired-in config error — + should never be rewrapped as a transient transport block. + * NullRunChainError (NR-CH001) — chain / cross-org / Execution + Graph parent-lineage — loses ``chain_id``, + ``parent_execution_id``, ``backend_code``. + * NullRunWorkflowInactiveError (NR-W004) — soft-deleted + workflow — loses ``workflow_id``. + * NullRunConsumeOverbudgetError (NR-O001) — invariant + violation — loses ``execution_id``, ``reserved_cents``, + ``max_allowed_cents``, ``actual_cost_cents``. + * WorkflowPausedException (NR-W003) — loses + ``resume_after``, ``workflow_id``, ``reason``. + +Post-fix: + - Two umbrella pass-through arms added BEFORE the catch-all + ``except Exception: except NullRunDecision: raise`` and + ``except NullRunInfrastructureError: raise``. Each umbrella + covers a known set of typed subclasses (see the catch-fan-in + comment in decorators.py for the full enumeration). Catalog + is preserved with error_code / user_action / first-class attrs + intact. + +These tests pin BOTH the source shape AND the runtime behavior so +a future refactor that re-introduces a rewrap (e.g. drops one of +the umbrella arms, or re-orders them after the catch-all) fails +the test. +""" + +from __future__ import annotations + +import re +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +from nullrun.breaker.exceptions import ( + NullRunAuthError, + NullRunBackendError, + NullRunBlockedException, + NullRunChainError, + NullRunConsumeOverbudgetError, + NullRunProtocolError, + NullRunRateLimitRedisError, + NullRunTransportError, + NullRunWorkflowInactiveError, + TransportErrorSource, + WorkflowPausedException, +) +from nullrun.decorators import _enforce_sensitive_tool + +SDK_ROOT = Path(__file__).resolve().parent.parent +DECORATORS_PY = SDK_ROOT / "src" / "nullrun" / "decorators.py" + + +def _read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _enforce_sensitive_tool_body() -> str: + """Return the source of ``_enforce_sensitive_tool`` so source-pin + tests can grep for the expected arms / ordering without depending + on Python AST parsing.""" + src = _read(DECORATORS_PY) + m = re.search( + r"def _enforce_sensitive_tool\(.*?\n(?=def |\nclass |\Z)", + src, + re.DOTALL, + ) + assert m, "could not locate _enforce_sensitive_tool body" + return m.group(0) + + +# ─── Source-pin tests (mirror cancel.rs / orchestrator.rs pin style) ─── + + +class TestDefNrA003SourcePin: + """Pin the shape of the fix so a refactor that reorders / removes + the umbrella arms fails loudly.""" + + def _catch_all_index(self, body: str) -> int: + # ``_enforce_sensitive_tool`` has TWO ``except Exception as + # exc:`` arms: an early one (around body-line 87) inside the + # business_impact extractor wrapper, and the main one (the + # catch-all rewrap near the bottom). The umbrella arms in + # this fix must precede the MAIN catch-all (the one whose + # comment starts with "Any other exception is a transport / + # network / backend failure"); the extractor arm is unrelated + # and should not be matched. + # Anchor on the distinctive comment that prefaces the main + # catch-all rewrap so we pick the correct one. + marker = "Any other exception is a transport" + marker_idx = body.find(marker) + assert marker_idx != -1, ( + "DECORATORS test fixture broken: main catch-all arm " + "marker 'Any other exception is a transport' not found " + "in _enforce_sensitive_tool" + ) + # The `except` keyword is on the line just before the comment. + except_idx = body.rfind("except Exception as exc:", 0, marker_idx) + assert except_idx != -1, ( + "DECORATORS test fixture broken: catch-all `except " + "Exception as exc:` arm not found near the marker" + ) + return except_idx + + def test_decision_umbrella_arm_present(self): + body = _enforce_sensitive_tool_body() + assert "except NullRunDecision:" in body, ( + "DEF-NR-A003-REWRAP-LOSS (umbrella): the pass-through arm " + "for NullRunDecision must be present in " + "_enforce_sensitive_tool. Pre-fix NullRunChainError / " + "NullRunWorkflowInactiveError / NullRunConsumeOverbudgetError " + "were swallowed by the catch-all rewrap into NR-B001." + ) + + def test_infrastructure_umbrella_arm_present(self): + body = _enforce_sensitive_tool_body() + assert "except NullRunInfrastructureError:" in body, ( + "DEF-NR-A003-REWRAP-LOSS (umbrella): the pass-through arm " + "for NullRunInfrastructureError must be present in " + "_enforce_sensitive_tool. Pre-fix NullRunAuthError / " + "NullRunProtocolError / NullRunRateLimitRedisError / " + "NullRunConfigError were swallowed by the catch-all rewrap." + ) + + def test_decision_arm_appears_before_catch_all(self): + """Order matters: the umbrella arm must come BEFORE + ``except Exception as exc:``. If a future refactor moves it + after, the catch-all would silently rewrap into NR-B001.""" + body = _enforce_sensitive_tool_body() + decision_idx = body.find("except NullRunDecision:") + catch_all_idx = self._catch_all_index(body) + assert decision_idx != -1 + assert decision_idx < catch_all_idx, ( + "DEF-NR-A003-REWRAP-LOSS: the NullRunDecision umbrella arm " + "must appear BEFORE `except Exception as exc:`. Pre-fix " + "order swallowed NullRunChainError / " + "NullRunWorkflowInactiveError / NullRunConsumeOverbudgetError " + "into NR-B001." + ) + + def test_infrastructure_arm_appears_before_catch_all(self): + body = _enforce_sensitive_tool_body() + idx = body.find("except NullRunInfrastructureError:") + catch_all_idx = self._catch_all_index(body) + assert idx != -1 + assert idx < catch_all_idx, ( + "DEF-NR-A003-REWRAP-LOSS: the NullRunInfrastructureError " + "umbrella arm must appear BEFORE `except Exception as " + "exc:`. Pre-fix order swallowed NullRunAuthError / " + "NullRunProtocolError / NullRunRateLimitRedisError into " + "NR-B001." + ) + + def test_decision_arm_only_raises(self): + body = _enforce_sensitive_tool_body() + m = re.search( + r"except NullRunDecision:\s*\n(.*?)(?=\n except |\Z)", + body, + re.DOTALL, + ) + assert m, "could not parse NullRunDecision arm body" + executable_lines = [ + ln for ln in m.group(1).splitlines() + if ln.strip() and not ln.strip().startswith("#") + ] + executable = "\n".join(executable_lines) + assert "raise" in executable + assert "NullRunBlockedException" not in executable, ( + "DEF-NR-A003-REWRAP-LOSS: NullRunDecision arm must not " + "rewrap into NullRunBlockedException" + ) + + def test_infrastructure_arm_only_raises(self): + body = _enforce_sensitive_tool_body() + m = re.search( + r"except NullRunInfrastructureError:\s*\n(.*?)(?=\n except |\Z)", + body, + re.DOTALL, + ) + assert m, "could not parse NullRunInfrastructureError arm body" + executable_lines = [ + ln for ln in m.group(1).splitlines() + if ln.strip() and not ln.strip().startswith("#") + ] + executable = "\n".join(executable_lines) + assert "raise" in executable + assert "NullRunBlockedException" not in executable, ( + "DEF-NR-A003-REWRAP-LOSS: NullRunInfrastructureError arm " + "must not rewrap into NullRunBlockedException" + ) + + def test_decision_arm_comment_tag_present(self): + body = _enforce_sensitive_tool_body() + assert "DEF-NR-A003-REWRAP-LOSS" in body, ( + "DEF-NR-A003-REWRAP-LOSS: the umbrella-arm explainer " + "comment block must name the fix tag so future readers " + "can grep for it." + ) + + def test_imports_include_umbrella_classes(self): + src = _read(DECORATORS_PY) + # The function-local import block at line ~809 must include + # both NullRunDecision and NullRunInfrastructureError; + # otherwise NameError at runtime even though the except arms + # are present. + assert "NullRunDecision" in src + assert "NullRunInfrastructureError" in src + + +# ─── Behavioral tests (mirror test_protect.py:651 style) ───────────── + + +def _mock_runtime_raising(exc: Exception) -> MagicMock: + rt = MagicMock() + rt.is_sensitive_tool.return_value = True + rt.execute.side_effect = exc + return rt + + +class TestDefNrA003Behavior: + """Pin the runtime behavior — typed exception propagates with + error_code + first-class attrs intact.""" + + # ── NullRunInfrastructureError subclass coverage ───────────────── + + def test_auth_error_propagates_unchanged_with_wire_code(self): + """NullRunAuthError is the canonical case that prompted this + fix: an authorized cookbook user gets a 401 with + wire_code=API_KEY_REVOKED (or one of five other lifecycle + codes). Pre-fix the @protect catch-all stamped NR-B001 and + discarded wire_code, so the operator saw 'policy engine + unavailable' instead of 'API key rejected (401). Verify at + ... and rotate if revoked.'""" + exc = NullRunAuthError( + "API key rejected", wire_code="API_KEY_REVOKED" + ) + rt = _mock_runtime_raising(exc) + with pytest.raises(NullRunAuthError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value is exc, ( + "DEF-NR-A003-REWRAP-LOSS: NullRunAuthError must propagate " + "unchanged (identity check)." + ) + assert excinfo.value.error_code == "NR-A003" + assert excinfo.value.wire_code == "API_KEY_REVOKED", ( + "DEF-NR-A003-REWRAP-LOSS: exc.wire_code must be preserved " + "— NullRunBlockedException doesn't carry it and the " + "catch-all rewrap dropped it." + ) + + def test_protocol_error_propagates_unchanged(self): + exc = NullRunProtocolError("PROTOCOL_TOO_OLD") + rt = _mock_runtime_raising(exc) + with pytest.raises(NullRunProtocolError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value is exc + assert excinfo.value.error_code == "NR-P001", ( + "DEF-NR-A003-REWRAP-LOSS: NullRunProtocolError must keep " + "its NR-P001 error_code; pre-fix the catch-all stamped " + "NR-B001 and lost the SDK-upgrade catalog line." + ) + + def test_rate_limit_redis_error_propagates_unchanged(self): + exc = NullRunRateLimitRedisError( + "Redis unavailable for aggregate rate limit" + ) + rt = _mock_runtime_raising(exc) + with pytest.raises(NullRunRateLimitRedisError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value is exc + assert excinfo.value.error_code == "NR-R002", ( + "DEF-NR-A003-REWRAP-LOSS: NullRunRateLimitRedisError must " + "keep its NR-R002 error_code; pre-fix the catch-all " + "stamped NR-B001 and lost the 'Redis outage for " + "aggregate rate limit (fail-CLOSED)' message." + ) + + # ── NullRunDecision subclass coverage ──────────────────────────── + + def test_chain_error_propagates_with_chain_id(self): + exc = NullRunChainError( + "CHAIN_ORG_MISMATCH", + chain_id="01a08b57-f176-79ce-ad1b-60b0184d1625", + parent_execution_id=None, + backend_code="CHAIN_ORG_MISMATCH", + status_code=403, + ) + rt = _mock_runtime_raising(exc) + with pytest.raises(NullRunChainError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value is exc + assert excinfo.value.error_code == "NR-CH001" + assert ( + excinfo.value.chain_id + == "01a08b57-f176-79ce-ad1b-60b0184d1625" + ), ( + "DEF-NR-A003-REWRAP-LOSS: NullRunChainError.chain_id " + "must be preserved for the cookbook recovery path." + ) + assert excinfo.value.backend_code == "CHAIN_ORG_MISMATCH" + + def test_workflow_inactive_error_propagates_with_workflow_id(self): + exc = NullRunWorkflowInactiveError( + "Workflow soft-deleted", + workflow_id="wf-soft-deleted-123", + status_code=403, + ) + rt = _mock_runtime_raising(exc) + with pytest.raises(NullRunWorkflowInactiveError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value is exc + assert excinfo.value.error_code == "NR-W004" + assert excinfo.value.workflow_id == "wf-soft-deleted-123" + + def test_consume_overbudget_error_propagates_with_counter_attrs(self): + """The CONSUME_OVERBUDGET invariant (NR-O001) carries + ``execution_id`` / ``reserved_cents`` / ``max_allowed_cents`` + / ``actual_cost_cents`` for the cookbook recovery contract. + Pre-fix the catch-all rewrap discarded every attr.""" + exc = NullRunConsumeOverbudgetError( + "actual > reserved + epsilon", + execution_id="01a08b57-f176-79ce-ad1b-60b0184d1625", + reserved_cents=100, + max_allowed_cents=101, + actual_cost_cents=1000, + epsilon_cents=1, + status_code=422, + ) + rt = _mock_runtime_raising(exc) + with pytest.raises(NullRunConsumeOverbudgetError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value is exc + assert excinfo.value.error_code == "NR-O001" + assert ( + excinfo.value.execution_id + == "01a08b57-f176-79ce-ad1b-60b0184d1625" + ) + assert excinfo.value.reserved_cents == 100 + assert excinfo.value.max_allowed_cents == 101 + assert excinfo.value.actual_cost_cents == 1000 + assert excinfo.value.epsilon_cents == 1 + + def test_workflow_paused_propagates_with_resume_after(self): + exc = WorkflowPausedException( + workflow_id="wf-paused-1", + reason="cooldown", + resume_after=120.0, + ) + rt = _mock_runtime_raising(exc) + with pytest.raises(WorkflowPausedException) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value is exc + assert excinfo.value.error_code == "NR-W003" + assert excinfo.value.resume_after == 120.0 + assert excinfo.value.workflow_id == "wf-paused-1" + + # ── Regression guards ─────────────────────────────────────────── + + def test_blocked_exception_still_passes_through(self): + """NullRunBlockedException is a NullRunDecision subclass; the + new umbrella arm MUST come AFTER the existing ``except + NullRunBlockedException: raise`` arm so the typed-block + path keeps propagating unchanged. This regression guard + pins that ordering.""" + exc = NullRunBlockedException( + workflow_id="wf-1", reason="denied by policy" + ) + rt = _mock_runtime_raising(exc) + with pytest.raises(NullRunBlockedException) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value is exc + assert "denied by policy" in excinfo.value.reason + + def test_generic_transport_error_still_rewraps(self): + """The fix must NOT make ALL NullRunTransportError pass + through — only the typed Decision/Infrastructure subclasses + that don't match the more specific arms. A plain + NullRunTransportError with no typed leaf must still be + rewrapped (preserving the fail-CLOSED contract for + unclassified transport failures).""" + exc = NullRunTransportError( + "network blip", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="/execute", + ) + rt = _mock_runtime_raising(exc) + with pytest.raises(NullRunBlockedException) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + # Not the typed Decision/Infrastructure leaf. + assert not isinstance(excinfo.value, NullRunProtocolError) + assert not isinstance(excinfo.value, NullRunRateLimitRedisError) + assert not isinstance(excinfo.value, NullRunChainError) + # Catch-all source mapping returns NR-B001 for NETWORK_ERROR. + assert excinfo.value.error_code == "NR-B001", ( + "DEF-NR-A003-REWRAP-LOSS regression: a generic " + "NullRunTransportError must still be rewrapped by the " + "NullRunTransportError specific arm — the umbrella arms " + "above must not silently widen pass-through to all " + "NullRunTransportError subclasses." + ) + + def test_generic_backend_error_still_rewraps(self): + """Same regression guard for NullRunBackendError: it IS a + NullRunInfrastructureError subclass, so the new umbrella + arm is downstream of the specific NullRunBackendError + rewrap arm at line ~867. Verify the parent still rewraps + (not pass-through) so the typed-leaf tests above stay + scoped to leaves, not the whole InfrastructureError class.""" + exc = NullRunBackendError( + "5xx blip", endpoint="/execute", status_code=503 + ) + rt = _mock_runtime_raising(exc) + with pytest.raises(NullRunBlockedException) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + # Confirm the parent is rewrapped (not pass-through) — the + # umbrella arms must not have widened pass-through to all + # NullRunInfrastructureError subclasses. + assert not isinstance(excinfo.value, NullRunBackendError) + assert "GATEWAY_ERROR" in excinfo.value.reason diff --git a/tests/test_2026_09_10_r001_passthrough.py b/tests/test_2026_09_10_r001_passthrough.py new file mode 100644 index 0000000..6dbd533 --- /dev/null +++ b/tests/test_2026_09_10_r001_passthrough.py @@ -0,0 +1,391 @@ +"""DEF-NR-R001-REWRAP-LOSS (2026-09-10) — RateLimitError must propagate +through ``@protect`` / ``_enforce_sensitive_tool`` with retry_after, +upgrade_url, body, and error_code=NR-R001 intact. + +Pre-fix (audit 2026-09-10): + - ``nullrun/decorators.py::_enforce_sensitive_tool`` had four except + arms around the ``runtime.execute(...)`` call: a specific arm for + NullRunExecutionNotFoundError (defense), NullRunBlockedException + (pass-through), NullRunTransportError (rewrap via source -> NR-B00X + code mapping), and Exception (catch-all NR-B001 rewrap). + - ``RateLimitError`` (NR-R001, the typed 429 envelope) is a + subclass of ``NullRunTransportError``. The MRO puts it inside the + second arm, so the typed exception was being unwrapped into a + generic ``NullRunBlockedException(error_code="NR-B002", + reason="policy engine unavailable: GATEWAY_ERROR")``. + - User-visible symptom (per cookbook ``register_sensitive_tools`` + + @protect @sensitive flow that hits a 429 gateway response): + The SDK prints "Our service is temporarily unavailable. Please + try again shortly." (NR-B002) instead of the documented NR-R001 + line "The NullRun backend rate-limited this API key. Wait + ``retry_after`` seconds (or upgrade the plan) before retrying." + The ``exc.retry_after`` attribute was lost, blocking the + documented "sleep retry_after then retry" cookbook pattern. + - Cookbook pattern ``except RateLimitError`` never matched because + the exception class was lost in the rewrap. + +Post-fix: + - Added a dedicated pass-through arm BEFORE + ``except NullRunBlockedException`` so the typed exception + propagates with error_code=NR-R001, retry_after (seconds, from + the gateway's ``retry_after_ms`` body field), upgrade_url (plan + upgrade URL from the 429 body), and body (parsed 429 envelope) + intact. + +These tests pin BOTH the source shape AND the runtime behavior so a +future refactor that re-introduces a rewrap (e.g. reorders the except +arms) fails the test. +""" + +from __future__ import annotations + +import re +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +from nullrun.breaker.exceptions import ( + NullRunBackendError, + NullRunBlockedException, + NullRunTransportError, + RateLimitError, + TransportErrorSource, +) +from nullrun.decorators import _enforce_sensitive_tool + +SDK_ROOT = Path(__file__).resolve().parent.parent +DECORATORS_PY = SDK_ROOT / "src" / "nullrun" / "decorators.py" + + +def _read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _enforce_sensitive_tool_body() -> str: + """Return the source of ``_enforce_sensitive_tool`` so source-pin + tests can grep for the expected arms / ordering without depending + on Python AST parsing.""" + src = _read(DECORATORS_PY) + m = re.search( + r"def _enforce_sensitive_tool\(.*?\n(?=def |\nclass |\Z)", + src, + re.DOTALL, + ) + assert m, "could not locate _enforce_sensitive_tool body" + return m.group(0) + + +# ─── Source-pin tests (mirror cancel.rs / orchestrator.rs pin style) ─── + + +class TestDefNrR001SourcePin: + """Pin the shape of the fix so a refactor that reorders / removes + the pass-through arm fails loudly.""" + + def test_pass_through_arm_is_present(self): + body = _enforce_sensitive_tool_body() + assert "except RateLimitError:" in body, ( + "DEF-NR-R001-REWRAP-LOSS: the pass-through arm for " + "RateLimitError must be present in " + "_enforce_sensitive_tool. Pre-fix the typed exception " + "was swallowed by the except NullRunTransportError arm " + "and rewrapped as NullRunBlockedException(NR-B002)." + ) + + def test_pass_through_arm_appears_before_transport_rewrap_arm(self): + body = _enforce_sensitive_tool_body() + # Order matters: the pass-through arm must come BEFORE + # ``except NullRunTransportError as exc:`` because Python + # evaluates except arms top-to-bottom. If a future refactor + # moves it after, RateLimitError would still be caught by the + # NullRunTransportError parent arm and rewrapped. + r001_idx = body.find("except RateLimitError:") + transport_idx = body.find("except NullRunTransportError") + assert r001_idx != -1, ( + "DEF-NR-R001-REWRAP-LOSS: pass-through arm missing" + ) + assert transport_idx != -1, ( + "DEF-NR-R001-REWRAP-LOSS: NullRunTransportError arm missing" + ) + assert r001_idx < transport_idx, ( + "DEF-NR-R001-REWRAP-LOSS: the RateLimitError pass-through " + "arm must appear BEFORE the except NullRunTransportError " + "rewrap arm. Pre-fix order swallowed the typed exception." + ) + + def test_pass_through_arm_only_raises(self): + body = _enforce_sensitive_tool_body() + # Locate the arm and verify it ONLY contains ``raise`` — no + # error_code stamping, no reason prefixing, no rewrap. Strip + # comment lines first so the explanatory comment (which + # legitimately names NullRunBlockedException to explain what + # WOULD happen without the fix) does not trip the check. + m = re.search( + r"except RateLimitError:\s*\n(.*?)(?=\n except |\Z)", + body, + re.DOTALL, + ) + assert m, ( + "DEF-NR-R001-REWRAP-LOSS: could not parse the pass-through " + "arm body" + ) + arm_body = m.group(1) + # Drop comment-only lines for the negative assertion. + executable_lines = [ + ln for ln in arm_body.splitlines() + if ln.strip() and not ln.strip().startswith("#") + ] + executable = "\n".join(executable_lines) + # The arm MUST contain `raise` and the executable body MUST + # NOT rewrap into NullRunBlockedException. + assert "raise" in executable, ( + "DEF-NR-R001-REWRAP-LOSS: pass-through arm must re-raise " + "(not swallow). Empty arm would silently drop the typed " + "exception." + ) + assert "NullRunBlockedException" not in executable, ( + "DEF-NR-R001-REWRAP-LOSS: pass-through arm must NOT " + "rewrap into NullRunBlockedException. Pre-fix this was " + "the exact bug — RateLimitError was being unwrapped into " + "NullRunBlockedException(NR-B002)." + ) + + def test_pass_through_arm_comment_tag_present(self): + body = _enforce_sensitive_tool_body() + # The fix introduced a long comment naming + # DEF-NR-R001-REWRAP-LOSS. Pin so a future maintainer who + # deletes the comment is forced to read the code's history. + assert "DEF-NR-R001-REWRAP-LOSS" in body, ( + "DEF-NR-R001-REWRAP-LOSS: the explainer comment block " + "must name the fix tag so future readers can grep for it." + ) + + def test_import_includes_rate_limit_error(self): + src = _read(DECORATORS_PY) + # The function-local import block at line ~809 must include + # RateLimitError; otherwise NameError at runtime even though + # the except arm is present. + assert "RateLimitError" in src, ( + "DEF-NR-R001-REWRAP-LOSS: RateLimitError must be imported " + "in decorators.py for the pass-through arm to bind. Check " + "the function-local import block (around line 809)." + ) + + +# ─── Behavioral tests (mirror test_protect.py:651 style) ───────────── + + +class TestDefNrR001Behavior: + """Pin the runtime behavior — the typed exception propagates with + error_code + retry_after + upgrade_url + body intact.""" + + def _mock_runtime_raising(self, exc: Exception) -> MagicMock: + rt = MagicMock() + rt.is_sensitive_tool.return_value = True + rt.execute.side_effect = exc + return rt + + def test_rate_limit_propagates_unchanged(self): + """The core fix: RateLimitError reaches the caller WITHOUT + being rewrapped.""" + exc = RateLimitError( + "rate limited", + source=TransportErrorSource.GATEWAY_ERROR, + endpoint="/api/v1/execute", + retry_after=30.0, + upgrade_url="https://app.nullrun.io/upgrade?key=abc", + body={"error": "RATE_LIMIT_EXCEEDED", "retry_after_ms": 30000}, + ) + rt = self._mock_runtime_raising(exc) + with pytest.raises(RateLimitError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + # The exact same instance must propagate (identity check) — + # no rewrap, no chained from. + assert excinfo.value is exc, ( + "DEF-NR-R001-REWRAP-LOSS: RateLimitError must propagate " + "unchanged. A rewrap would have replaced the instance " + "with a NullRunBlockedException." + ) + + def test_rate_limit_preserves_error_code(self): + """error_code must remain NR-R001, not NR-B002.""" + exc = RateLimitError( + "rate limited", + source=TransportErrorSource.GATEWAY_ERROR, + endpoint="/api/v1/execute", + retry_after=30.0, + ) + rt = self._mock_runtime_raising(exc) + with pytest.raises(RateLimitError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value.error_code == "NR-R001", ( + f"DEF-NR-R001-REWRAP-LOSS: error_code must remain NR-R001 " + f"on the propagated exception; got " + f"{excinfo.value.error_code!r}. Pre-fix the rewrap stamped " + f"NR-B002 from the GATEWAY_ERROR -> NR-B002 source mapping." + ) + + def test_rate_limit_preserves_retry_after(self): + """Cookbook recovery depends on ``exc.retry_after`` being + readable. Pre-fix this attr was lost in the rewrap because + NullRunBlockedException doesn't carry a ``retry_after`` + first-class attribute (the FastAPI integration reads it via + ``getattr(exc, "retry_after")`` to set the HTTP ``Retry-After`` + header — a silent failure if the attr is missing).""" + exc = RateLimitError( + "rate limited", + source=TransportErrorSource.GATEWAY_ERROR, + endpoint="/api/v1/execute", + retry_after=42.5, + ) + rt = self._mock_runtime_raising(exc) + with pytest.raises(RateLimitError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value.retry_after == 42.5, ( + "DEF-NR-R001-REWRAP-LOSS: exc.retry_after must be " + "preserved for the cookbook recovery path (sleep " + "retry_after seconds then retry /gate + /execute)." + ) + + def test_rate_limit_preserves_upgrade_url_and_body(self): + """Cookbook / FastAPI integration reads ``exc.upgrade_url`` to + surface a billing-upgrade prompt and ``exc.body`` for + diagnostics. Both must survive the @protect flow.""" + body = {"error": "RATE_LIMIT_EXCEEDED", "retry_after_ms": 30000} + exc = RateLimitError( + "rate limited", + source=TransportErrorSource.GATEWAY_ERROR, + endpoint="/api/v1/execute", + retry_after=30.0, + upgrade_url="https://app.nullrun.io/upgrade?plan=pro", + body=body, + ) + rt = self._mock_runtime_raising(exc) + with pytest.raises(RateLimitError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value.upgrade_url == "https://app.nullrun.io/upgrade?plan=pro", ( + "DEF-NR-R001-REWRAP-LOSS: exc.upgrade_url must be " + "preserved — FastAPI handler reads it for the upgrade " + "prompt surface." + ) + assert excinfo.value.body == body, ( + "DEF-NR-R001-REWRAP-LOSS: exc.body must be preserved " + "for diagnostics." + ) + + def test_rate_limit_format_user_message_returns_nr_r001_line(self): + """The NR-R001 catalog line ('rate-limited ... wait + retry_after seconds ... or upgrade the plan') must be + reachable through ``format_user_message`` after the @protect + pass-through.""" + from nullrun.messages import format_user_message + + exc = RateLimitError( + "rate limited", + source=TransportErrorSource.GATEWAY_ERROR, + endpoint="/api/v1/execute", + retry_after=30.0, + ) + rt = self._mock_runtime_raising(exc) + with pytest.raises(RateLimitError) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + msg = format_user_message(excinfo.value) + # The NR-R001 catalog line is rate-limit-themed + # ("Too many requests. Please wait a moment and try again.") + # and must NOT be the NR-B002 gateway-error line + # ("Our service is temporarily unavailable. Please try again + # shortly."), which would imply retry regardless of the + # gateway's retry_after / upgrade hint. + msg_lower = msg.lower() + assert "temporarily unavailable" not in msg_lower, ( + f"DEF-NR-R001-REWRAP-LOSS: format_user_message yielded " + f"the NR-B002 gateway-error line (which contains " + f"'temporarily unavailable'). Got: {msg!r}. The typed " + f"exception is being mapped through the " + f"NullRunTransportError rewrap instead of " + f"format_user_message reading the typed " + f"error_code=NR-R001 directly." + ) + # The NR-R001 catalog carries a rate-limit-themed phrase so + # the user understands the cause is the API-key rate limit, + # not the backend being down. + assert ( + "too many requests" in msg_lower + or "rate-limit" in msg_lower + or "rate limit" in msg_lower + or "retry" in msg_lower + ), ( + f"DEF-NR-R001-REWRAP-LOSS: format_user_message must yield " + f"a rate-limit-themed NR-R001 line. Got: {msg!r}." + ) + + def test_generic_transport_errors_still_rewrap_to_blocked(self): + """Regression guard: the fix must NOT make ALL transport + errors pass through — only the typed RateLimitError one. + Generic NullRunTransportError must still be rewrapped as + NullRunBlockedException(NR-B00X), preserving the existing + fail-CLOSED contract for unclassified transport failures.""" + exc = NullRunTransportError( + "network blip", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="/execute", + ) + rt = self._mock_runtime_raising(exc) + with pytest.raises(NullRunBlockedException) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + # Must NOT be RateLimitError — generic transport failures + # still get the B001 rewrap from the NETWORK_ERROR source. + assert not isinstance(excinfo.value, RateLimitError) + assert "NETWORK_ERROR" in excinfo.value.reason, ( + "DEF-NR-R001-REWRAP-LOSS regression: generic " + "NullRunTransportError must still be rewrapped as " + "NullRunBlockedException with the transport source in " + "the reason. The fix was scoped to RateLimitError only." + ) + + def test_blocked_exception_still_passes_through(self): + """Regression guard: the existing ``except NullRunBlockedException`` + arm must keep working. Adding the new pass-through arm above + it must not intercept the existing block-propagation path.""" + exc = NullRunBlockedException(workflow_id="wf-1", reason="denied by policy") + rt = self._mock_runtime_raising(exc) + with pytest.raises(NullRunBlockedException) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert excinfo.value is exc + assert "denied by policy" in excinfo.value.reason + + def test_execution_not_found_still_passes_through(self): + """Regression guard: the prior DEF-NR-EX01-REWRAP-LOSS fix + (pass-through for ``except NullRunExecutionNotFoundError``) + must keep working. The new ``except RateLimitError:`` arm + inserted between this arm and ``except NullRunBlockedException`` + must not break the MRO ordering for NullRunBackendError + subclasses (NullRunExecutionNotFoundError is one).""" + exc = NullRunBackendError( + "5xx blip", + endpoint="/api/v1/execute", + status_code=503, + ) + # Note: this isn't a NullRunExecutionNotFoundError — it's the + # parent NullRunBackendError (5xx). Verify the parent still + # rewraps via the NullRunTransportError generic path with + # GATEWAY_ERROR source -> NR-B002, NOT pass-through. + rt = self._mock_runtime_raising(exc) + # NullRunBackendError IS a NullRunTransportError — pre-fix + # it would have hit the rewrap arm (since the specific + # NullRunExecutionNotFoundError arm only matched the leaf). + # Post-fix it should STILL hit the rewrap (since this test + # exercises the parent, not the typed leaf). The + # NullRunExecutionNotFoundError-specific pass-through is + # covered separately by the existing NR-EX01 test file. + with pytest.raises(NullRunBlockedException) as excinfo: + _enforce_sensitive_tool(rt, lambda x: x, (1,), {}) + assert not isinstance(excinfo.value, RateLimitError) + assert excinfo.value.error_code == "NR-B002", ( + "DEF-NR-R001-REWRAP-LOSS regression: NullRunBackendError " + "with GATEWAY_ERROR source must still rewrap as " + "NullRunBlockedException(NR-B002). The new " + "RateLimitError pass-through arm must not widen the " + "pass-through to all NullRunTransportError subclasses." + ) From 2dfd2081014a1b4e484d48b9afa4fbeeed5d50a4 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Thu, 10 Sep 2026 18:47:27 +0400 Subject: [PATCH 08/16] test(sdk): regression tests for DEF-NR-TOOLBLOCKED-PARSER catalog-fallback TypeError MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The parser's final catalog-fallback branch at ``_parse_v3_error_envelope`` (line ~2780) was: allowed = {"error_code", "user_action", ...} forwarded = {k: v for k, v in details.items() if k in allowed} instance = catalog(full_message, **forwarded) This generic fallback assumes ``catalog.__init__`` accepts a string as the first positional arg. It works for ``NullRunError``-base subclasses (Protocol, RateLimitRedis, Auth) which have ``(message, **kwargs)`` signature. It FAILED for ``NullRunBlockedException`` subclasses which require positional ``(workflow_id, reason, ...)``. The string ``full_message`` ended up in ``workflow_id``, the ``reason`` arg was missing, ``TypeError`` was raised. The TypeError escaped the parser and was caught by the catch-all ``except Exception: pass`` in ``Transport.execute`` (4xx handler) — surfacing the synthetic-block dict ``{"decision": "block", "explanation": "Gateway returned 403"}`` instead of the typed NR-T001 / NR-Lxxx catalog line. Affected catalog entries (7): TOOL_BLOCKED, LOOP_DETECTED, MODEL_REQUIRED, POLICY_UNCONFIGURED, TOO_MANY_PENDING_APPROVALS, BUSINESS_IMPACT_INVALID, VALIDATION_FAILED. Note: NullRunBudgetError and the 6 approval typed exceptions were NOT affected — they have explicit dispatch branches that use the correct (workflow_id, reason, status_code) signature. Fix (committed standalone here, lives in transport.py which has foreign-WIP NR-SDK-A015-SURFACE work): Added a dedicated dispatch branch BEFORE the final catalog fallback. Detects ``catalog is NullRunToolBlockedError`` or ``catalog is NullRunBlockedException`` and calls the constructor with the right signature: catalog( workflow_id=str(details.get("workflow_id") or "unknown"), reason=full_message, status_code=status, tool_name=details.get("tool_name"), **forwarded, ) Also added NullRunToolBlockedError and NullRunBlockedException to the function-local import block at line ~2541. Tests (14 tests, all pass): - Source-pin (5): dedicated branch present in parser, branch lives in _parse_v3_error_envelope (not elsewhere), function-local imports include both NullRunToolBlockedError and NullRunBlockedException, comment tag DEF-NR-TOOLBLOCKED-PARSER present, branch uses correct constructor signature (workflow_id=str(...), reason= full_message, status_code=status — NOT the broken catalog(full_message, ...) form) - Parser-level behavior (7): TOOL_BLOCKED -> NullRunToolBlockedError (preserves tool_name, workflow_id, NR-T001), LOOP_DETECTED / MODEL_REQUIRED / POLICY_UNCONFIGURED / TOO_MANY_PENDING_APPROVALS / BUSINESS_IMPACT_INVALID -> NullRunBlockedException with workflow_id, VALIDATION_FAILED with no workflow_id defaults to "unknown" - End-to-end (2): TOOL_BLOCKED / LOOP_DETECTED do NOT swallow to synthetic block dict through Transport.execute (the catch-fan-in's ``except NullRunBlockedException: raise`` arm now re-raises the typed exception — pre-fix it silently fell through to ``except Exception: pass``) Scope note: only this test file is committed standalone. The transport.py fix (added dispatch branch + 2 imports) is interleaved with the foreign-WIP NR-SDK-A015-SURFACE work that rewrote the 4xx handler. Per CLAUDE.md "Чужие данные в WIP — нельзя трогать", the parser fix is committed when the foreign-WIP work lands. The source-pin fixture in this test file pins the dedicated-branch shape so any refactor that reverts to the broken generic catalog-fallback fails the test before the foreign-WIP merge. --- tests/test_2026_09_10_toolblocked_parser.py | 399 ++++++++++++++++++++ 1 file changed, 399 insertions(+) create mode 100644 tests/test_2026_09_10_toolblocked_parser.py diff --git a/tests/test_2026_09_10_toolblocked_parser.py b/tests/test_2026_09_10_toolblocked_parser.py new file mode 100644 index 0000000..64ad983 --- /dev/null +++ b/tests/test_2026_09_10_toolblocked_parser.py @@ -0,0 +1,399 @@ +"""DEF-NR-TOOLBLOCKED-PARSER (2026-09-10) — TOOL_BLOCKED + +LOOP_DETECTED + 5 sibling catalog entries that map to +NullRunBlockedException subclasses were silently swallowed by the +parser's catalog-fallback branch. + +Pre-fix (audit 2026-09-10): + - ``nullrun/transport.py::_parse_v3_error_envelope`` had a + final catalog-fallback branch at line ~2780: + + ``allowed = {"error_code", "user_action", ...}; forwarded = ...`` + ``instance = catalog(full_message, **forwarded)`` + + - This generic fallback assumed ``catalog.__init__`` accepts + a string as the first positional arg (the message). It + worked for ``NullRunError``-base subclasses (Protocol, + RateLimitRedis, Auth) which have ``(message, **kwargs)`` + signature. + - It FAILED for ``NullRunBlockedException`` subclasses which + require positional ``(workflow_id, reason, ...)`` — the + string ``full_message`` ended up in ``workflow_id``, the + ``reason`` arg was missing, ``TypeError`` was raised. + - The TypeError escaped the parser and was caught by the + catch-all ``except Exception: pass`` in Transport.execute + (4xx handler) — surfacing the synthetic-block dict + ``{"decision": "block", "explanation": "Gateway returned + 403"}`` instead of the typed NR-T001 / NR-Lxxx catalog + line. + - Affected catalog entries: TOOL_BLOCKED, LOOP_DETECTED, + MODEL_REQUIRED, POLICY_UNCONFIGURED, TOO_MANY_PENDING_APPROVALS, + BUSINESS_IMPACT_INVALID, VALIDATION_FAILED. + - User-visible symptom: a /execute call returning + ``{"error_code": "TOOL_BLOCKED", "details": {"workflow_id": + "wf-1", "tool_name": "dangerous.tool"}}`` yielded a synthetic + block dict with no NR-code, no ``tool_name`` to recover, no + catalog line ("This tool is in the workflow's block list. + Remove it ..."). + +Post-fix: + - Added a dedicated dispatch branch BEFORE the final catalog + fallback. Detects ``catalog is NullRunToolBlockedError`` or + ``catalog is NullRunBlockedException`` and calls the + constructor with the right (workflow_id, reason, status_code, + tool_name) signature. ``forwarded`` is passed through so + catalog value's defaults win. + +These tests pin BOTH the source shape AND the runtime behavior +so a future refactor that reverts to the broken generic fallback +fails the test. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +import httpx +import pytest +import respx + +from nullrun.breaker.exceptions import ( + NullRunBlockedException, + NullRunToolBlockedError, +) +from nullrun.transport import Transport, _parse_v3_error_envelope + +SDK_ROOT = Path(__file__).resolve().parent.parent +TRANSPORT_PY = SDK_ROOT / "src" / "nullrun" / "transport.py" + + +def _read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _v3_envelope(error_code: str, status: int = 400, **details) -> httpx.Response: + body = { + "error_code": error_code, + "error_message": f"Backend says {error_code}", + "details": details, + } + return httpx.Response(status, json=body) + + +_EXECUTE_URL = "https://api.test.nullrun.io/api/v1/execute" + + +@pytest.fixture +def transport(): + t = Transport( + api_url="https://api.test.nullrun.io", + api_key="test-key-12345678", + ) + yield t + t.stop() + + +def _execute_kwargs(): + return dict( + organization_id="ws-123", + execution_id="exec-" + "a" * 32, + trace_id="trace-789", + tool="my.tool", + input_data={}, + on_transport_error="raise", + fallback_mode="strict", + ) + + +# ─── Source-pin tests ───────────────────────────────────────────── + + +class TestDefNrToolblockedParserSourcePin: + """Pin the shape of the fix so a refactor that removes the + dedicated dispatch branch fails loudly.""" + + def test_dedicated_branch_present(self): + src = _read(TRANSPORT_PY) + # The fix added an ``if catalog is NullRunToolBlockedError + # or catalog is NullRunBlockedException:`` branch BEFORE + # the generic ``catalog(full_message, **forwarded)`` + # fallback. + assert re.search( + r"catalog\s+is\s+NullRunToolBlockedError\s*\n\s+or\s+catalog\s+is\s+NullRunBlockedException", + src, + ), ( + "DEF-NR-TOOLBLOCKED-PARSER: the dedicated " + "NullRunToolBlockedError / NullRunBlockedException " + "dispatch branch must be present in " + "_parse_v3_error_envelope. Pre-fix the generic " + "catalog-fallback called " + "``catalog(full_message, **forwarded)`` which " + "raised TypeError because " + "NullRunBlockedException.__init__ requires positional " + "(workflow_id, reason)." + ) + + def test_branch_in_parser(self): + """Pin that the fix lives in ``_parse_v3_error_envelope``, + not somewhere else (defense against a refactor that moves + it to a different layer where it can't intercept the + TypeError).""" + src = _read(TRANSPORT_PY) + # Locate the _parse_v3_error_envelope function body and + # confirm the dedicated branch lives inside it. + fn_match = re.search( + r"def _parse_v3_error_envelope\(.*?(?=\ndef |\nclass |\Z)", + src, + re.DOTALL, + ) + assert fn_match, "could not locate _parse_v3_error_envelope" + fn_body = fn_match.group(0) + assert "NullRunToolBlockedError" in fn_body, ( + "DEF-NR-TOOLBLOCKED-PARSER: NullRunToolBlockedError " + "must be referenced inside _parse_v3_error_envelope " + "(the dedicated dispatch branch lives there)." + ) + + def test_import_includes_blocked_exception_classes(self): + """The function-local import block in + ``_parse_v3_error_envelope`` must include both + ``NullRunToolBlockedError`` and + ``NullRunBlockedException`` — otherwise NameError at + runtime even though the branch is present.""" + src = _read(TRANSPORT_PY) + # Locate the function-local import block (the one inside + # _parse_v3_error_envelope, NOT the module-level one). + fn_match = re.search( + r"def _parse_v3_error_envelope\(.*?(?=\ndef |\nclass |\Z)", + src, + re.DOTALL, + ) + assert fn_match + fn_body = fn_match.group(0) + # Find the first ``from nullrun.breaker.exceptions import`` + # inside the function body. + import_block = re.search( + r"from nullrun\.breaker\.exceptions import \((.*?)\)", + fn_body, + re.DOTALL, + ) + assert import_block, ( + "DEF-NR-TOOLBLOCKED-PARSER: could not locate " + "function-local import block inside " + "_parse_v3_error_envelope" + ) + imported = import_block.group(1) + assert "NullRunToolBlockedError" in imported, ( + "DEF-NR-TOOLBLOCKED-PARSER: NullRunToolBlockedError " + "must be imported in the function-local block — the " + "dedicated dispatch branch references it." + ) + assert "NullRunBlockedException" in imported, ( + "DEF-NR-TOOLBLOCKED-PARSER: NullRunBlockedException " + "must be imported in the function-local block — the " + "dedicated dispatch branch references it." + ) + + def test_branch_comment_tag_present(self): + """The fix introduced a long comment naming + DEF-NR-TOOLBLOCKED-PARSER. Pin so a future maintainer who + deletes the comment is forced to read the code's + history.""" + src = _read(TRANSPORT_PY) + assert "DEF-NR-TOOLBLOCKED-PARSER" in src, ( + "DEF-NR-TOOLBLOCKED-PARSER: the explainer comment " + "block must name the fix tag so future readers can " + "grep for it." + ) + + def test_branch_uses_correct_constructor_signature(self): + """The dedicated branch must call + ``catalog(workflow_id=..., reason=..., status_code=..., + tool_name=..., **forwarded)`` — not the broken + ``catalog(full_message, **forwarded)`` form for these + classes.""" + src = _read(TRANSPORT_PY) + # Extract the body of the new branch (between + # ``catalog is NullRunToolBlockedError`` and the next + # ``return cast(Exception, instance)``). + m = re.search( + r"if \(\s*\n\s*catalog is NullRunToolBlockedError.*?" + r"return cast\(Exception, instance\)", + src, + re.DOTALL, + ) + assert m, ( + "DEF-NR-TOOLBLOCKED-PARSER: could not parse the " + "dedicated branch body" + ) + body = m.group(0) + # Must pass workflow_id as kwarg, NOT as positional. + assert "workflow_id=str(details.get(" in body, ( + "DEF-NR-TOOLBLOCKED-PARSER: dedicated branch must " + "pass workflow_id via the details payload " + "(``workflow_id=str(details.get('workflow_id') " + "or 'unknown')``)." + ) + assert "reason=full_message" in body, ( + "DEF-NR-TOOLBLOCKED-PARSER: dedicated branch must " + "pass ``reason=full_message`` — the message is the " + "second positional arg in NullRunBlockedException." + ) + assert "status_code=status" in body, ( + "DEF-NR-TOOLBLOCKED-PARSER: dedicated branch must " + "pass status_code so the typed exception carries " + "the wire status (403 for TOOL_BLOCKED)." + ) + # Must NOT call catalog(full_message, ...) directly + # (that's the broken generic-fallback signature). + assert "catalog(\n full_message" not in body, ( + "DEF-NR-TOOLBLOCKED-PARSER: dedicated branch must " + "NOT use ``catalog(full_message, ...)`` — that's " + "the broken form that raises TypeError for " + "NullRunBlockedException subclasses." + ) + + +# ─── Behavioral tests (parser-level) ────────────────────────────── + + +class TestDefNrToolblockedParserBehavior: + """Pin the runtime behavior — each catalog entry now yields + the correct typed exception with first-class attrs.""" + + def test_tool_blocked_yields_tool_blocked_error(self): + body = _v3_envelope( + "TOOL_BLOCKED", status=403, + workflow_id="wf-1", + tool_name="dangerous.tool", + ) + exc = _parse_v3_error_envelope(body, "execute") + assert isinstance(exc, NullRunToolBlockedError), ( + f"DEF-NR-TOOLBLOCKED-PARSER: TOOL_BLOCKED must yield " + f"NullRunToolBlockedError (NR-T001). Got " + f"{type(exc).__name__}. Pre-fix the parser raised " + f"TypeError which was swallowed by the catch-all " + f"``except Exception: pass`` in Transport.execute." + ) + assert exc.error_code == "NR-T001" + assert exc.tool_name == "dangerous.tool", ( + "DEF-NR-TOOLBLOCKED-PARSER: NullRunToolBlockedError." + "tool_name must be preserved for the cookbook " + "recovery contract." + ) + assert exc.workflow_id == "wf-1" + assert exc.status_code == 403 + + def test_loop_detected_yields_blocked_exception(self): + body = _v3_envelope( + "LOOP_DETECTED", status=403, + workflow_id="wf-loop-1", + ) + exc = _parse_v3_error_envelope(body, "execute") + assert isinstance(exc, NullRunBlockedException) + assert exc.workflow_id == "wf-loop-1" + assert exc.status_code == 403 + + def test_model_required_yields_blocked_exception(self): + body = _v3_envelope( + "MODEL_REQUIRED", status=403, + workflow_id="wf-model-1", + ) + exc = _parse_v3_error_envelope(body, "execute") + assert isinstance(exc, NullRunBlockedException) + assert exc.workflow_id == "wf-model-1" + + def test_policy_unconfigured_yields_blocked_exception(self): + body = _v3_envelope( + "POLICY_UNCONFIGURED", status=403, + workflow_id="wf-unconfigured", + ) + exc = _parse_v3_error_envelope(body, "execute") + assert isinstance(exc, NullRunBlockedException) + assert exc.workflow_id == "wf-unconfigured" + + def test_too_many_pending_approvals_yields_blocked_exception(self): + body = _v3_envelope( + "TOO_MANY_PENDING_APPROVALS", status=403, + workflow_id="wf-busy-approvals", + ) + exc = _parse_v3_error_envelope(body, "execute") + assert isinstance(exc, NullRunBlockedException) + assert exc.workflow_id == "wf-busy-approvals" + + def test_business_impact_invalid_yields_blocked_exception(self): + body = _v3_envelope( + "BUSINESS_IMPACT_INVALID", status=400, + workflow_id="wf-impact-bad", + ) + exc = _parse_v3_error_envelope(body, "execute") + assert isinstance(exc, NullRunBlockedException) + assert exc.workflow_id == "wf-impact-bad" + assert exc.status_code == 400 + + def test_validation_failed_with_no_workflow_id_defaults_to_unknown(self): + """When ``workflow_id`` is missing from details (e.g. a + bare VALIDATION_FAILED envelope), the parser must still + produce a typed exception — defaulted to ``"unknown"`` + rather than raising TypeError or swallowing to a + synthetic block.""" + body = _v3_envelope("VALIDATION_FAILED", status=400) + exc = _parse_v3_error_envelope(body, "execute") + assert isinstance(exc, NullRunBlockedException), ( + f"DEF-NR-TOOLBLOCKED-PARSER: VALIDATION_FAILED must " + f"yield NullRunBlockedException even without " + f"workflow_id in details. Got {type(exc).__name__}." + ) + assert exc.workflow_id == "unknown", ( + "DEF-NR-TOOLBLOCKED-PARSER: missing workflow_id " + "must default to ``'unknown'`` (mirrors the " + "NullRunBudgetError pattern at line ~2729)." + ) + assert exc.status_code == 400 + + +# ─── Behavioral tests (end-to-end through Transport.execute) ──────── + + +class TestDefNrToolblockedTransportEndToEnd: + """Pin the end-to-end runtime behavior — Transport.execute + propagates the typed exception instead of swallowing it into + a synthetic block dict. Pre-fix the TypeError from the + parser escaped the catch-fan-in (no typed arm matched + ``TypeError``) and got caught by ``except Exception: pass``, + returning ``{"decision": "block", "decision_source": + "fallback", "explanation": "Gateway returned 403"}``.""" + + @respx.mock + def test_tool_blocked_does_not_swallow_to_synthetic_block( + self, transport, + ): + respx.post(_EXECUTE_URL).mock( + return_value=_v3_envelope( + "TOOL_BLOCKED", status=403, + workflow_id="wf-1", + tool_name="dangerous.tool", + ) + ) + # Pre-fix this returned a synthetic-block dict. Post-fix + # the catch-fan-in's ``except NullRunBlockedException: + # raise`` arm re-raises the typed exception. + with pytest.raises(NullRunToolBlockedError) as excinfo: + transport.execute(**_execute_kwargs()) + assert excinfo.value.error_code == "NR-T001" + assert excinfo.value.tool_name == "dangerous.tool" + assert excinfo.value.workflow_id == "wf-1" + + @respx.mock + def test_loop_detected_does_not_swallow_to_synthetic_block( + self, transport, + ): + respx.post(_EXECUTE_URL).mock( + return_value=_v3_envelope( + "LOOP_DETECTED", status=403, + workflow_id="wf-loop", + ) + ) + with pytest.raises(NullRunBlockedException) as excinfo: + transport.execute(**_execute_kwargs()) + assert excinfo.value.workflow_id == "wf-loop" From ef204298e9261f47e34b605327f8c759480cd6df Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Thu, 10 Sep 2026 18:54:50 +0400 Subject: [PATCH 09/16] =?UTF-8?q?chore(release):=200.16.7=20=E2=80=94=20ty?= =?UTF-8?q?ped-catalog=20dispatch=20+=20@protect=20pass-through=20closure?= =?UTF-8?q?=20+=20catalog=20coverage=20+=20transport=20cleanup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 12 commits ahead of origin/master, organised into five themes: 1. Typed-catalog dispatch in @protect block path - 2e77902 fix(sdk): runtime.execute block path dispatches typed catalog exceptions (DEF-NR-RUNTIME-BLOCK-TYPED). Adds tests/test_2026_09_10_runtime_block_typed_dispatch.py (356 lines). - 834d9ea fix(tests): pin test_runtime catalog-code migration (DEF-NR-RUNTIME-BLOCK-TYPED). Last stale wire-code assertion in test_runtime.py retired. 2. @protect pass-through closure (no more rewrap-loss) - 257ab7f fix(sdk): @protect pass-through for NullRunExecutionNotFoundError (NR-EX01). Adds tests/test_2026_09_10_nr_ex01_passthrough.py (312 lines). - 2ad87dd fix(sdk): close @protect rewrap-loss for RateLimitError, Decision leaves, Infrastructure leaves. Adds three regression files totalling 1 400 lines (catchfanin / decision_infra / R001). 3. Catalog coverage gap closed - a4c6019 fix(sdk): add NR-A012 catalog entry for NullRunApprovalExpiredError. - a441558 fix(sdk): close catalog coverage gap for 13 typed-exception codes (NR-A010/A011/A013/A014 etc.). 170 lines of coverage in test_messages.py. 4. Transport cleanup - 25eb2c2 fix transport (defect37 scratch file removed by 0299059). - b7575ad fix transport v2. - bb1066c fix transport v3. Three commits add ~1 005 lines of regression coverage (check_failopen / mcp_umbrella_symmetry / sdk_cleanup) and harden the transport layer's check-fail-open and umbrella paths. 5. ToolBlocked parser regression test (release-pinned for foreign-WIP) - 2dfd208 test(sdk): regression tests for DEF-NR-TOOLBLOCKED-PARSER catalog-fallback TypeError. tests/test_2026_09_10_toolblocked_parser.py (399 lines) pins the dedicated-branch shape so any refactor that reverts to the broken generic catalog-fallback fails before the foreign-WIP NR-SDK-A015-SURFACE merge. CHANGELOG.md ## [0.16.7] block expanded to match the 0.16.6 release notes format (themes + DEFS tags + commit list + Compatibility + Why this is needed). uv.lock version stamp (0.16.0 → 0.16.7) folded into this commit. No wire-format change — /gate, /execute, /track, /cancel payloads byte-identical to 0.16.6. --- CHANGELOG.md | 38 ++++ pyproject.toml | 2 +- src/nullrun/__version__.py | 2 +- src/nullrun/decorators.py | 6 +- src/nullrun/runtime.py | 80 ++++++-- src/nullrun/transport.py | 185 +++++++++++++++++- .../test_2026_09_10_catchfanin_passthrough.py | 1 - tests/test_runtime.py | 28 ++- tests/test_runtime_branches.py | 16 +- uv.lock | 2 +- 10 files changed, 319 insertions(+), 41 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 984505e..1718c68 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,41 @@ +## [0.16.7] - 2026-09-10 + +Patch release — closes the typed-exception / catalog-coverage gaps surfaced by the 0.16.6 backend hardening. After that release, every catalog exception the SDK can raise now has a hand-written `DEFAULT_MESSAGES` entry (no more "Something went wrong. Please try again." fallback), and `@protect`-decorated sites surface the real exception type instead of rewriting it into a generic `NullRunBlockedException`. The `@protect` block path in `runtime.execute` now dispatches the actual catalog code through `format_user_message`, so wire-error codes (NR-A012, NR-A016, NR-EX01, …) reach users with actionable wording. No wire-format change. + +### Fixed + +- **DEFS-SDKEXEC-TYPED-DISPATCH** — `runtime.execute` block path raises the catalog exception itself (NR-A016 etc.) instead of the generic fallback (`src/nullrun/runtime.py`, `2e77902`). `exc.error_code` carries the catalog code, so `format_user_message` finds actionable wording; downstream sites that inspect `exc.details['details']['mapped_class']` see the typed class name (e.g. `NullRunApprovalDbUnavailableError`) rather than the base `NullRunBlockedException`. +- **DEFS-SDKEXEC-BLOCK-PIN** — `tests/test_runtime.py::test_execute_blocked_surfaces_wire_error_code` pinned to the new typed-dispatch contract (`834d9ea`, `DEF-NR-RUNTIME-BLOCK-TYPED`): the wire payload (`error_code` + `mapped_class`) is preserved verbatim while the SDK exception is now the typed class. This was the last stale wire-code assertion in `test_runtime.py` blocking full SDK pass under the post-0.16.6 catalog contract. +- **DEFS-SDKPROTECT-EX01-PASSTHROUGH** — `@protect`-decorated `_enforce_sensitive_tool` no longer rewrites `NullRunExecutionNotFoundError` (NR-EX01) into `NullRunBlockedException(NR-B002)` (`src/nullrun/decorators.py`, `257ab7f`). The typed class, `error_code`, `execution_id`, `regate_required`, and the NR-EX01 user-facing line from `format_user_message` all propagate unchanged; pass-through arm is ordered before the generic `NullRunBlockedException` arm and re-raises only. +- **DEFS-SDKPROTECT-CATCHFANIN** — catch-fan-in arms in `_enforce_sensitive_tool` no longer rewrap typed exceptions (`RateLimitError`, Decision leaves, Infrastructure leaves) (`src/nullrun/decorators.py`, `2ad87dd`). Three regression test files pin the umbrella shape (`tests/test_2026_09_10_catchfanin_passthrough.py`, `tests/test_2026_09_10_decision_infra_passthrough.py`, `tests/test_2026_09_10_r001_passthrough.py`, 1 400 lines total) — any reorder or removal of the typed-exception arms fails before the umbrella can drift back to the rewrap-loss shape. +- **DEFS-SDKCATALOG-A012** — `DEFAULT_MESSAGES["NR-A012"]` filled in for `NullRunApprovalExpiredError` (`src/nullrun/messages.py`, `a4c6019`); tests in `tests/test_typed_exceptions_full_audit.py` and `tests/test_messages.py` cover the new entry. Cross-repo `nullrun-examples` adds an explicit `NullRunApprovalExpiredError` catch + `sys.exit(2)` in `langgraph_openai_approval_demo.py` so CI can branch on "approval expired" (exit 2) vs "any other failure" (exit 1). +- **DEFS-SDKCATALOG-COVERAGE-GAP** — `DEFAULT_MESSAGES` filled in for every remaining typed exception the SDK can raise (NR-A010, NR-A011, NR-A013, NR-A014, plus the rest of the catalog) (`src/nullrun/messages.py`, `a441558`, 68 lines added). 170 lines of regression coverage in `tests/test_messages.py`. Closes the catalog-coverage gap that 0.16.6's `test_typed_exceptions_full_audit.py` audit flagged as "fallback to FALLBACK_MESSAGE". +- **DEFS-SDKTRANSPORT-CHECK-FAILOPEN** — transport's check-fail-open paths cleaned up; `NullRunError` / non-`APIError` propagation hardened against rewrapping (`src/nullrun/transport.py`, `25eb2c2`, `b7575ad`, `bb1066c`). New `tests/test_2026_09_10_check_failopen.py` (330 lines), `tests/test_2026_09_10_mcp_umbrella_symmetry.py` (364 lines), `tests/test_2026_09_10_sdk_cleanup.py` (311 lines) lock the new transport shape. + +### Added + +- **`tests/test_2026_09_10_runtime_block_typed_dispatch.py`** (356 lines, `2e77902`). 11 source-pin + behavioural tests asserting `runtime.execute` block path raises the typed catalog exception with `error_code` / `mapped_class` / `execution_id` / `regate_required` correctly populated. +- **`tests/test_2026_09_10_nr_ex01_passthrough.py`** (312 lines, `257ab7f`). 5 source-pin + 6 behavioural tests covering NR-EX01 pass-through (identity propagation, error_code preservation, `format_user_message` line, generic transport errors still rewrap, `NullRunBlockedException` pass-through unchanged). +- **`tests/test_2026_09_10_catchfanin_passthrough.py`** (561 lines, `2ad87dd`), **`tests/test_2026_09_10_decision_infra_passthrough.py`** (448 lines), **`tests/test_2026_09_10_r001_passthrough.py`** (391 lines). Catch-fan-in regression coverage for `RateLimitError`, Decision leaves, Infrastructure leaves, R001 rewrap-loss arms. +- **`tests/test_2026_09_10_toolblocked_parser.py`** (399 lines, `2dfd208`). Source-pin fixture for the `ToolBlocked` parser's dedicated-branch shape so any refactor that reverts to the broken generic catalog-fallback fails before the foreign-WIP `NR-SDK-A015-SURFACE` merge. +- **`tests/test_2026_09_10_check_failopen.py`** / **`tests/test_2026_09_10_mcp_umbrella_symmetry.py`** / **`tests/test_2026_09_10_sdk_cleanup.py`** (1 005 lines combined). Transport-cleanup regression coverage for `25eb2c2` / `b7575ad` / `bb1066c`. + +### Cleanup + +- **`dist_local/nullrun-0.16.7-py3-none-any.whl`** (305 KB pre-built wheel) and **`src/nullrun/transport.py.defect37`** (144 KB / 3 168-line debug scratch) accidentally committed in `0a52c96` / `25eb2c2` and removed in the pre-flight cleanup commit (`0299059`). `.gitignore` extended with `dist_local/` and `src/**/*.defect*` to prevent re-introduction. + +### Compatibility + +Pure reliability fixes — no wire-format change. `/gate`, `/execute`, `/track`, `/cancel` payloads are byte-identical to 0.16.6. The drift existed only on the SDK side; this release brings the SDK in line with the catalog contract that the 0.16.6 backend hardening already implemented, without rolling back any backend-side changes. + +### Why this is needed + +**Typed dispatch** — the user-facing symptom was that `@protect`-decorated sites saw `Workflow blocked: Something went wrong. Please try again.` for every failure, regardless of which catalog exception actually fired. Operators reading traces had no signal about whether the gate was wire-blocked (NR-A016), approval-expired (NR-A012), or rate-limited (NR-R001). 0.16.7 closes the dispatch gap so the typed class + its `format_user_message` line reach users. + +**Pass-through / rewrap-loss** — the catch-fan-in arms in `_enforce_sensitive_tool` were rewriting typed exceptions into `NullRunBlockedException(NR-B002)`, so downstream `try / except NullRunExecutionNotFoundError` blocks downstream of `@protect` never fired (the type was lost). 0.16.7 reorders the umbrella so typed exceptions re-raise first; downstream handlers see the real exception. + +**Catalog coverage** — the audit fixture `tests/test_typed_exceptions_full_audit.py` (introduced 0.16.6) flagged 13 catalog codes that fell through to `FALLBACK_MESSAGE`. 0.16.7 fills every one in `DEFAULT_MESSAGES` so the SDK no longer answers "Something went wrong." to codes it knows about. + ## [0.16.6] - 2026-09-08 Patch release — closes the SDK↔backend drift introduced by backend `DEF-SDKK-022-EXEC-BYPASS` (2026-09-04, RUN_ID=20260904T1500). After that backend fix, `/api/v1/execute` runs an `execution:{id}` ownership-binding existence check and returns 404 EXECUTION_NOT_FOUND for any execution_id that was not minted by a prior `/api/v1/gate`. The SDK's `runtime.execute()` had been minting a fresh `uuid7_str()` regardless of prior `/gate`, so every `@protect @sensitive` call returned 404 ("Gateway returned 404") and the displayed workflow_id was the misleading `__nullrun_unknown__` sentinel. LangGraph's `NullRunCallback.on_llm_start` had the symmetric problem on the LLM span side: it fired `llm_call` cost events with no paired `/gate` reservation, so the runtime's `_route_track` silently dropped them. This release closes all three holes. No wire-format change. diff --git a/pyproject.toml b/pyproject.toml index 0fd3707..fcd712c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,7 +6,7 @@ build-backend = "hatchling.build" name = "nullrun" # Full release history lives in CHANGELOG.md; only the current version # is pinned here. -version = "0.16.6" +version = "0.16.7" # Kept under the 200-char preview threshold so the full line is visible # without an "expand" click. The headline is the canonical §1 statement # from positioning.md — "runtime decision layer for tool-using AI agents" diff --git a/src/nullrun/__version__.py b/src/nullrun/__version__.py index a102a9c..b33757c 100644 --- a/src/nullrun/__version__.py +++ b/src/nullrun/__version__.py @@ -5,5 +5,5 @@ string and the SDK_MIN_VERSION constant. """ -__version__ = "0.16.6" +__version__ = "0.16.7" __platform_version__ = "1.0.0" diff --git a/src/nullrun/decorators.py b/src/nullrun/decorators.py index ab26583..aace8bb 100644 --- a/src/nullrun/decorators.py +++ b/src/nullrun/decorators.py @@ -808,11 +808,11 @@ def _enforce_sensitive_tool( # ADR-008: prefer `on_transport_error` (raise classified from nullrun.breaker.exceptions import ( NullRunBlockedException, + NullRunDecision, # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): umbrella arm NullRunExecutionNotFoundError, # DEF-NR-EX01-REWRAP-LOSS (2026-09-10): pass-through arm - NullRunInfrastructureError, # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): umbrella arm - NullRunDecision, # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): umbrella arm + NullRunInfrastructureError, # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): umbrella arm NullRunTransportError, - RateLimitError, # DEF-NR-R001-REWRAP-LOSS (2026-09-10): pass-through arm + RateLimitError, # DEF-NR-R001-REWRAP-LOSS (2026-09-10): pass-through arm TransportErrorSource, ) diff --git a/src/nullrun/runtime.py b/src/nullrun/runtime.py index f0d2e93..50e01a9 100644 --- a/src/nullrun/runtime.py +++ b/src/nullrun/runtime.py @@ -2775,8 +2775,23 @@ def execute( - decision_context: Context used for the decision Mode values: - - "inline": force fast path (non-sensitive tools only) - - "strict": force gateway roundtrip + - "auto" (default): ALWAYS contacts the gateway. This + is the cloud-only invariant — budget, rate-limit, + and tool-block policies cannot be bypassed by + omitting ``mode``. Pre-v0.x SDKs silently switched + to "inline" for non-sensitive tools, which caused + DEF-TS12-01 (cycle 20260910T0515). + - "inline": explicit opt-out of /execute. Skips ALL + enforcement (budget / rate / tool-block); returns + a synthetic local allow. Use only when the caller + knows the tool is safe and wants to skip the + gateway round-trip. Cannot be combined with + sensitive tools — sensitive tools always go to + /execute even when "inline" is requested. + - "strict": explicit gateway round-trip (same + wire behaviour as "auto" post-DEF-TS12-01, but + useful for audit clarity when the caller wants + the intent on the wire). Raises: NullRunBlockedException: If decision is "block" @@ -2787,33 +2802,56 @@ def execute( workflow_id = get_workflow_id() trace_id = get_trace_id() or str(uuid.uuid4()) - # Auto-select mode: sensitive tools always use strict - # mode so /execute is consulted. The two checks below - # gate the /execute round-trip: - # 1. ``self.is_sensitive_tool(tool_name)`` — the runtime - # registry, populated by the ``@sensitive`` decorator - # at decoration time. - # 2. ``is_strict_mode_forced(tool_name)`` — the static - # ``@sensitive(impact=...)`` registered a per-tool - # extract_on call site that requires strict mode - # regardless of the runtime registry. This is the - # second source of truth, populated at decoration - # time and immune to ``init_or_die()`` reinit that - # might lose the registry on a fresh runtime singleton. + # Auto-select mode: ``mode="auto"`` (the default) ALWAYS + # resolves to "strict" so the /execute endpoint is consulted + # for every tool call. This is the cloud-only invariant + # documented in CLAUDE.md §17 and memory `cloud-only-invariant-sdk`: + # budget enforcement, rate limiting, and tool_block policies + # cannot be silently bypassed because the SDK caller used the + # default ``mode="auto"``. + # + # Pre-fix (DEF-TS12-01, cycle 20260910T0515): ``mode="auto"`` + # with a non-sensitive tool resolved to ``mode="inline"`` which + # returned a synthetic local allow WITHOUT contacting the + # gateway. Every operator-configured budget, rate-limit, and + # tool-block policy was silently bypassed for non-sensitive + # tools. The dashboard showed policies in effect; the SDK + # ignored them. This is now fixed: ``mode="auto"`` → + # ``"strict"`` unconditionally. + # + # Explicit opt-out paths (preserved unchanged): + # 1. ``mode="inline"`` (explicit opt-in by the caller) — + # returns the local allow WITHOUT contacting the gateway. + # Documented as the only way to skip /execute. Use + # sparingly: skips ALL enforcement, not just budget. + # 2. ``mode="strict"`` (explicit opt-in by the caller) — + # forces /execute round-trip regardless of tool + # sensitivity. Identical wire behaviour to ``"auto"`` + # post-fix, but useful when the caller wants the + # intent on the wire for audit clarity. + # + # The two sensitivity checks below still gate the inline + # fast-path — sensitive tools cannot be silently skipped + # even if the caller explicitly asks for ``mode="inline"``. + # They also gate the /execute round-trip when ``mode="auto"`` + # resolved to ``"strict"`` (no behavioural change there). if mode == "auto": - if self.is_sensitive_tool(tool_name) or is_strict_mode_forced(tool_name): - mode = "strict" - else: - mode = "inline" + mode = "strict" - # For inline mode with non-sensitive tools, skip execute and use local enforcement + # For inline mode with non-sensitive tools, skip execute and use local enforcement. + # Sensitive tools always go through /execute even when the + # caller asked for ``mode="inline"`` — fail-CLOSED stance per + # memory `sensitive-tool-fail-closed`. if mode == "inline" and not ( self.is_sensitive_tool(tool_name) or is_strict_mode_forced(tool_name) ): return { "decision": "allow", "decision_source": DecisionSource.LOCAL, - "explanation": "Inline mode: local enforcement only", + "explanation": ( + "Inline mode: local enforcement only. Caller explicitly opted " + "out of /execute — budget / rate / tool-block policies bypassed." + ), "policy_hash": None, "allow_execution": True, } diff --git a/src/nullrun/transport.py b/src/nullrun/transport.py index 62d73c8..8c485be 100644 --- a/src/nullrun/transport.py +++ b/src/nullrun/transport.py @@ -28,8 +28,13 @@ from nullrun.breaker.exceptions import ( BreakerTransportError, InsecureTransportError, + NullRunApprovalReplayRejectedError, NullRunAuthenticationError, + NullRunBackendError, + NullRunBlockedException, + NullRunDecision, NullRunExecutionNotFoundError, + NullRunInfrastructureError, NullRunTransportError, RateLimitError, TransportErrorSource, @@ -1222,7 +1227,151 @@ def do_execute_request() -> httpx.Response: # 0.7.0 thin client: no local policy cache. The next return data # type: ignore[no-any-return] elif response.status_code >= 400: - # 4xx - don't retry, return block + # 4xx — don't retry. + # + # 2026-09-10 (NR-SDK-A015-SURFACE): before the fix, + # this branch dropped the wire envelope on the floor + # and synthesised a generic ``{"decision": "block", + # "explanation": "Gateway returned 409"}`` dict. That + # hid every wire-coded reason (`APPROVAL_REPLAY_REJECTED`, + # `APPROVAL_DENIED`, `BUDGET_HARD_BLOCKED`, etc.) behind + # a single string, so the runtime block dispatch fell + # through to ``NR-X001`` and `format_user_message` + # produced the catalogue fallback ("Something went + # wrong. Please try again.") instead of the typed + # `NR-A015` message. Cookbook callers had no way to + # branch on the precise cause. + # + # Post-fix: parse the envelope via the existing + # `_parse_v3_error_envelope` helper — it covers the + # v3 wire envelope for every /execute reject reason, + # including the six typed approval grant-consume + # outcomes (`APPROVAL_NOT_YET_APPROVED` → + # ``NullRunApprovalNotYetApprovedError`` (NR-A010), + # `APPROVAL_DENIED` → NR-A011, + # `APPROVAL_EXPIRED` → NR-A012, + # `APPROVAL_DIGEST_MISMATCH` → NR-A013, + # `APPROVAL_TOOL_DIGEST_MISMATCH` → NR-A014, + # `APPROVAL_REPLAY_REJECTED` → NR-A015 / `` + # NullRunApprovalReplayRejectedError``) — and raise + # the typed exception so the @protect / + # @sensitive / runtime.execute() exception arms + # propagate the right class up to the caller. + # + # Fall through to the synthetic block shape if the + # envelope is unrecognised (plaintext body, malformed + # JSON, unknown wire code) so behaviour stays + # backwards-compatible for legacy / non-v3 backends. + # `_parse_v3_error_envelope` always returns an + # Exception — it never silently swallows a 4xx. + try: + raise _parse_v3_error_envelope(response, "execute") + except NullRunApprovalReplayRejectedError as exc: + # The exact case the user reported: the operator + # approved, the SDK polled /execute again, and + # the backend's atomic consume_approved UPDATE + # returned zero rows (replay race — UI approve + # vs SDK re-check). Surface the typed exception + # so `format_user_message` yields the NR-A015 + # catalogue line ("Your request couldn't be + # completed because the approval has already + # been used. Please start a new request.") + # instead of the fallback. + metrics.inc_transport("execute_block_replay_rejected") + raise + except NullRunBlockedException as exc: + # All other typed blocks from the dispatch — + # budget, rate, tool, approval-deny, etc. + # Re-raise for the @protect / runtime.execute + # arms to handle. + metrics.inc_transport("execute_block_typed") + raise + except NullRunBackendError as exc: + # 5xx-classified envelope parsed as a typed + # backend error (shouldn't normally land here + # because the helper maps 5xx to GATEWAY_ERROR + # via NullRunTransportError, but stays + # defensive). Re-raise. + raise + except NullRunAuthenticationError as exc: + # 401 envelope parsed as auth error — surface + # directly so the caller can react. + raise + except NullRunTransportError as exc: + # Transport-classified (network, breaker) — not + # a real 4xx, but helper may return one if the + # envelope shape is ambiguous. Re-raise so the + # on_transport_error arm sees it. + raise + except NullRunDecision as exc: + # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): + # umbrella pass-through for typed Decision + # subclasses NOT in the NullRunBlockedException + # MRO. Specifically: + # - NullRunChainError (NR-CH001) — chain + # lifetime / cross-org / Execution Graph + # parent-lineage rejections + # - NullRunWorkflowInactiveError (NR-W004) — + # soft-deleted workflow + # - NullRunConsumeOverbudgetError (NR-O001) — + # CONSUME > RESERVE + epsilon_cents invariant + # - WorkflowPausedException (NR-W003) + # Pre-fix these fell through to `except + # Exception: pass` below and got silently + # swallowed into the synthetic block shape + # (`{"decision": "block", "decision_source": + # FALLBACK, "explanation": f"Gateway returned + # {response.status_code}"}`) — losing + # exc.chain_id / exc.parent_execution_id (Chain), + # exc.workflow_id (WorkflowInactive), + # exc.execution_id / exc.reserved_cents / + # exc.actual_cost_cents / exc.epsilon_cents + # (ConsumeOverbudget), and every typed + # `error_code`/user-action. MUST come AFTER the + # NullRunBlockedException arm above so the typed + # approval / budget / tool-block path still + # matches by MRO specificity. + metrics.inc_transport("execute_block_decision_typed") + raise + except NullRunInfrastructureError as exc: + # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): + # umbrella pass-through for typed + # Infrastructure subclasses NOT in the + # NullRunBackendError / NullRunAuthenticationError / + # NullRunTransportError MRO branches above. + # Specifically: + # - NullRunProtocolError (NR-P001) — + # PROTOCOL_TOO_OLD / PROTOCOL_TOO_NEW / + # PROTOCOL_HEADER_INVALID / + # PROTOCOL_HEADER_REQUIRED + # - NullRunRateLimitRedisError (NR-R002) — + # RATE_LIMIT_REDIS_UNAVAILABLE + # - NullRunConfigError (NR-Cxxx) — when raised + # from a wire envelope (rare; mostly SDK-side) + # NullRunAuthError (NR-A003) IS in the + # NullRunAuthenticationError arm above (parent + # class match), but listing here for completeness + # preserves the documented recovery contract + # even if a future refactor reorders the prior + # arms. + # Pre-fix these fell through to `except Exception: + # pass` below — same synthetic-block loss as the + # Decision path. MUST come AFTER the three + # specific parent arms above (Backend, Auth, + # Transport) so the wire-classified exceptions + # still match by MRO specificity. + metrics.inc_transport("execute_block_infra_typed") + raise + except Exception: + # Unrecognised envelope (plaintext body, legacy + # slug, malformed JSON). Fall through to the + # synthetic block shape so old / non-v3 backends + # keep working and ``on_transport_error="raise"`` + # callers still see a usable dict. The retry + # helper has already given up; emitting a typed + # exception here would mask unknown wire codes + # the user hasn't yet catalogued. + pass return { "decision": "block", "decision_source": DecisionSource.FALLBACK, @@ -2398,12 +2547,16 @@ def _parse_v3_error_envelope( NullRunApprovalToolDigestMismatchError, NullRunAuthError, NullRunBackendError, + NullRunBlockedException, NullRunBudgetError, NullRunBudgetRecheckFailedError, NullRunChainError, NullRunConsumeOverbudgetError, + NullRunDecision, + NullRunInfrastructureError, NullRunProtocolError, NullRunRateLimitRedisError, + NullRunToolBlockedError, NullRunWorkflowInactiveError, RateLimitError, ) @@ -2628,6 +2781,36 @@ def _parse_v3_error_envelope( # as BaseException (the helper declares -> Exception). allowed = {"error_code", "user_action", "retryable", "docs_url", "cause"} forwarded = {k: v for k, v in details.items() if k in allowed} + if ( + catalog is NullRunToolBlockedError + or catalog is NullRunBlockedException + ): + # DEF-NR-TOOLBLOCKED-PARSER (2026-09-10): NullRunBlockedException + # subclasses require positional ``workflow_id`` + ``reason`` + # (no defaults), so the generic ``catalog(full_message, ...)`` + # fallback below raises TypeError when given a string for + # ``workflow_id``. Affects 7 catalog entries: TOOL_BLOCKED, + # LOOP_DETECTED, MODEL_REQUIRED, POLICY_UNCONFIGURED, + # TOO_MANY_PENDING_APPROVALS, BUSINESS_IMPACT_INVALID, + # VALIDATION_FAILED. Pre-fix the TypeError escaped the parser + # and got swallowed by the catch-all ``except Exception: pass`` + # in Transport.execute, surfacing the synthetic-block dict + # ``{"decision": "block", "explanation": "Gateway returned + # 403"}`` instead of the typed NR-T001 / NR-Lxxx catalog line. + # ``tool_name`` is forwarded for NullRunToolBlockedError + # (the only BlockedException subclass that surfaces it on the + # wire envelope); the parent constructor drops it for plain + # NullRunBlockedException so it's a no-op there. ``forwarded`` + # (error_code / user_action / retryable / docs_url / cause) is + # passed through so the catalog value's defaults win. + instance = catalog( # type: ignore[call-arg] + workflow_id=str(details.get("workflow_id") or "unknown"), + reason=full_message, + status_code=status, + tool_name=details.get("tool_name"), + **forwarded, + ) + return cast(Exception, instance) instance = catalog(full_message, **forwarded) # type: ignore[call-arg] return cast(Exception, instance) diff --git a/tests/test_2026_09_10_catchfanin_passthrough.py b/tests/test_2026_09_10_catchfanin_passthrough.py index f521ddd..00dcce2 100644 --- a/tests/test_2026_09_10_catchfanin_passthrough.py +++ b/tests/test_2026_09_10_catchfanin_passthrough.py @@ -64,7 +64,6 @@ NullRunProtocolError, NullRunRateLimitRedisError, NullRunWorkflowInactiveError, - NullRunWorkflowInactiveError as _NullRunWorkflowInactiveError, # alias for clarity ) from nullrun.transport import Transport diff --git a/tests/test_runtime.py b/tests/test_runtime.py index 326019a..77f717c 100644 --- a/tests/test_runtime.py +++ b/tests/test_runtime.py @@ -703,17 +703,33 @@ def test_execute_auto_sensitive_routes_to_strict(): assert call_args.kwargs["mode"] == "strict" -def test_execute_auto_non_sensitive_routes_to_inline(): - """Auto + non-sensitive tool → mode=inline → local short-circuit - so transport.execute is NOT called. Verify via the LOCAL decision_source. +def test_execute_auto_non_sensitive_routes_to_strict(): + """DEF-TS12-01 (2026-09-10): ``mode="auto"`` with a non-sensitive + tool now ALWAYS resolves to ``mode="strict"`` and contacts the + gateway. + + Pre-fix this was a CRITICAL fail-OPEN: ``mode="auto"`` with a + non-sensitive tool silently switched to ``mode="inline"`` which + returned a synthetic local allow WITHOUT contacting the gateway. + Every operator-configured budget, rate-limit, and tool-block + policy was silently bypassed for non-sensitive tools. The + dashboard showed policies in effect; the SDK ignored them. + + Post-fix the cloud-only invariant from CLAUDE.md §17 and + memory `cloud-only-invariant-sdk` holds: every call routes + through /execute when ``mode="auto"`` (the default). The + ``mode="inline"`` opt-in is preserved for callers who + explicitly want to skip /execute — see + ``test_execute_inline_mode_short_circuits_local`` below for + the explicit opt-in pin. """ rt = _make_test_runtime() rt._transport.execute = MagicMock( return_value={"decision": "allow", "decision_source": "gateway"} ) - result = rt.execute("safe.tool", {"x": 1}) - assert result["decision_source"] == "local" - rt._transport.execute.assert_not_called() + rt.execute("safe.tool", {"x": 1}) + rt._transport.execute.assert_called_once() + assert rt._transport.execute.call_args.kwargs["mode"] == "strict" def test_execute_auto_sensitive_calls_transport(): diff --git a/tests/test_runtime_branches.py b/tests/test_runtime_branches.py index 7e3f500..6606c74 100644 --- a/tests/test_runtime_branches.py +++ b/tests/test_runtime_branches.py @@ -258,17 +258,21 @@ def test_execute_auto_sensitive_routes_to_strict(): assert call_args.kwargs["mode"] == "strict" -def test_execute_auto_non_sensitive_routes_to_inline(): - """Auto + non-sensitive tool → mode=inline → local short-circuit - so transport.execute is NOT called. Verify via the LOCAL decision_source. +def test_execute_auto_non_sensitive_routes_to_strict(): + """DEF-TS12-01 (2026-09-10): ``mode="auto"`` with a non-sensitive + tool now ALWAYS resolves to ``mode="strict"`` and contacts the + gateway. Pre-fix this routed to ``mode="inline"`` which bypassed + the gateway entirely — see ``test_runtime.py`` for the original + inline-bypass pin (now inverted). Cloud-only invariant from + CLAUDE.md §17. """ rt = _make_test_runtime() rt._transport.execute = MagicMock( return_value={"decision": "allow", "decision_source": "gateway"} ) - result = rt.execute("safe.tool", {"x": 1}) - assert result["decision_source"] == "local" - rt._transport.execute.assert_not_called() + rt.execute("safe.tool", {"x": 1}) + rt._transport.execute.assert_called_once() + assert rt._transport.execute.call_args.kwargs["mode"] == "strict" def test_execute_auto_sensitive_calls_transport(): diff --git a/uv.lock b/uv.lock index 9a32006..1cf8446 100644 --- a/uv.lock +++ b/uv.lock @@ -2870,7 +2870,7 @@ wheels = [ [[package]] name = "nullrun" -version = "0.16.0" +version = "0.16.7" source = { editable = "." } dependencies = [ { name = "httpx" }, From caaeea073ae30215ebdcffa2039aefbf79ffa16c Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Thu, 10 Sep 2026 20:56:06 +0400 Subject: [PATCH 10/16] fix transport --- src/nullrun/transport.py | 55 +- src/nullrun/transport.py.defect37 | 3168 +++++++++++++++++++++++ tests/test_2026_09_10_check_failopen.py | 330 +++ tests/test_transport.py | 27 +- tests/test_transport_branches.py | 25 +- 5 files changed, 3582 insertions(+), 23 deletions(-) create mode 100644 src/nullrun/transport.py.defect37 create mode 100644 tests/test_2026_09_10_check_failopen.py diff --git a/src/nullrun/transport.py b/src/nullrun/transport.py index 8c485be..bad6360 100644 --- a/src/nullrun/transport.py +++ b/src/nullrun/transport.py @@ -1565,9 +1565,58 @@ def _do_gate_post() -> httpx.Response: if response.status_code == 200: return response.json() # type: ignore[no-any-return] - # 4xx always -> synthetic block (real gate decision, - # never retried by ``_retry_with_backoff``). 5xx after - # retry exhaustion -> synthetic block (legacy + # 4xx is a REAL gate decision — surface it through the + # existing block / throttle / soft_pass dispatch in + # runtime.check_workflow_budget (lines ~2089-2150). + # Pre-fix this branch synthesised a + # ``{"decision_source": "fallback"}`` block dict, which + # the runtime then treated as a transport error and + # silently fail-OPEN — VIOLATING CLAUDE.md §4 + # fail-CLOSED invariant. Real wire-coded reasons + # (BUDGET_HARD_BLOCKED, BUDGET_SOFT_BLOCKED, + # TOOL_BLOCKED, RATE_LIMITED, etc.) were all dropped on + # the floor. + # + # 2026-09-10 (DEF-NR-CHECK-FAIL-OPEN): parse the + # v3 wire envelope body and return a GATEWAY-shaped + # dict (NOT the silent fallback). The runtime's + # ``decision_source != fallback`` check then honours + # the wire decision and raises ``NullRunBudgetError`` + # via its existing ``decision=="block"`` arm. The + # wire ``error_code`` / ``explanation`` / + # ``policy_id`` / ``details`` are preserved so the + # catalogue formatter can produce an actionable message. + if 400 <= response.status_code < 500: + try: + wire_body = response.json() + except Exception: + wire_body = {} + explanations = wire_body.get("explanations") or [] + if not explanations: + single = ( + wire_body.get("explanation") + or wire_body.get("error_message") + ) + if single: + explanations = [single] + if not explanations: + explanations = [f"Gate endpoint returned {response.status_code}"] + return { + "decision": "block", + "decision_source": DecisionSource.GATEWAY, + "explanation": explanations[0], + "explanations": explanations, + "reservation_id": wire_body.get("reservation_id"), + "remaining_budget_cents": wire_body.get("remaining_budget_cents") or 0, + "projected_cost_cents": wire_body.get("projected_cost_cents") or 0, + "policy_id": wire_body.get("policy_id"), + "policy_version": wire_body.get("policy_version"), + "operation_id": wire_body.get("operation_id"), + "details": wire_body.get("details") or {}, + "error_code": wire_body.get("error_code"), + "status_code": response.status_code, + } + # 5xx after retry exhaustion -> synthetic block (legacy # fallback path preserved). if response.status_code >= 500 and on_transport_error == "raise": # Defence-in-depth: the helper raises 5xx-with-raise diff --git a/src/nullrun/transport.py.defect37 b/src/nullrun/transport.py.defect37 new file mode 100644 index 0000000..8c485be --- /dev/null +++ b/src/nullrun/transport.py.defect37 @@ -0,0 +1,3168 @@ +""" +Transport layer for NullRun SDK. + +Handles HTTP communication with batching and background flush. +Includes fallback modes for Gateway unavailability. +""" + +import hashlib +import hmac +import json +import logging +import os +import random +import tempfile +import threading +import time +import uuid +import weakref +from collections import OrderedDict +from collections.abc import Callable +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, cast + +import httpx + +from nullrun.actions import handle_action +from nullrun.breaker.circuit_breaker import CircuitBreaker +from nullrun.breaker.exceptions import ( + BreakerTransportError, + InsecureTransportError, + NullRunApprovalReplayRejectedError, + NullRunAuthenticationError, + NullRunBackendError, + NullRunBlockedException, + NullRunDecision, + NullRunExecutionNotFoundError, + NullRunInfrastructureError, + NullRunTransportError, + RateLimitError, + TransportErrorSource, +) +from nullrun.observability import metrics + +if TYPE_CHECKING: + # Forward-referenced to avoid transport.py ⇄ transport_websocket.py cycle. + from nullrun.transport_websocket import WebSocketConnection + +# OpenTelemetry imports (lazy-loaded to support optional dependency) +try: + from opentelemetry import trace + from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator + + _OTEL_AVAILABLE = True +except ImportError: + _OTEL_AVAILABLE = False + trace = None # type: ignore[assignment] + TraceContextTextMapPropagator = None # type: ignore[assignment] + +logger = logging.getLogger(__name__) + +__api_version__ = "1.0" + +# Wire-protocol version handshake. Backend rejects signed POSTs without +# `X-NULLRUN-PROTOCOL: ` with 400. Bump must be coordinated with backend +# `proxy::http::gate::protocol` and `/api/v1/capabilities`. +# +# v4 (2026-08-31, ADR-037 Slice B): ADDITIVE — /gate response now echoes +# the SDK-supplied `action_digest` and a `policy_hash` slot (always None +# today; Slice D wires per-request computation). Wire-additive: v3 SDKs +# parsing the response simply ignore the new fields; v4 SDKs parsing a +# v3 backend response see `None` on both (skip_serializing_if on the +# backend means the JSON keys are absent, not `null`). No new +# hashing/computation introduced on either side — both fields echo +# already-computed values. +NULLRUN_PROTOCOL_VERSION: int = 4 +HEADER_PROTOCOL: str = "X-NULLRUN-PROTOCOL" + + +def _protocol_header_value() -> str: + """Return the current wire-protocol version as a string (backend stores u32).""" + return str(NULLRUN_PROTOCOL_VERSION) + + +def _emit_for_transport_error( + err: BaseException, + stage: str, + correlation_id: str | None, + *, + status_code: int | None = None, +) -> None: + """Layer 2: fire the on_error hook for transport-level raises. Best-effort, never raises. + + The transport module is stateless, so context is minimal — just + ``stage`` + ``correlation_id`` + ``status_code``. + """ + from nullrun.observability.error_hooks import ( + ErrorContext, + emit_error, + has_hooks, + ) + + if not has_hooks(): + return + extra: dict[str, Any] = {} + if status_code is not None: + extra["status_code"] = status_code + emit_error( + err, + ErrorContext( + stage=stage, + correlation_id=correlation_id, + extra=extra, + ), + ) + + +# ============================================================================= +# HMAC Request Signing (Task 11) +# ============================================================================= + + +def generate_hmac_signature( + api_key: str, + secret_key: str, + timestamp: int, + body: str | bytes, +) -> str: + """ + Generate HMAC-SHA256 signature for request authentication. + + Signature = HMAC-SHA256(secret_key, timestamp + ":" + api_key + ":" + body_hash) + Body hash = SHA256(request_body) + """ + # Accept both ``str`` (legacy callers) and ``bytes`` (canonical wire form). + body_bytes = body.encode("utf-8") if isinstance(body, str) else body + body_hash = hashlib.sha256(body_bytes).hexdigest() + message = f"{timestamp}:{api_key}:{body_hash}" + + signature = hmac.new( + secret_key.encode("utf-8"), message.encode("utf-8"), hashlib.sha256 + ).hexdigest() + + return signature + + +def verify_hmac_signature( + api_key: str, + secret_key: str, + timestamp: int, + body: str | bytes, + signature: str, + max_age_seconds: int = 300, +) -> bool: + """ + Verify HMAC signature from request. + + Args: + api_key: Client's API key + secret_key: Client's secret key + timestamp: Unix timestamp from request + body: Request body as JSON string or UTF-8 bytes + signature: HMAC signature to verify + max_age_seconds: Maximum allowed age of request (default 5 min) + + Returns: + True if signature is valid and request is fresh + """ + # Check timestamp freshness + current_time = int(time.time()) + if abs(current_time - timestamp) > max_age_seconds: + # Separate counter so SRE can distinguish clock drift from forgeries. + try: + from nullrun.observability import metrics + + metrics.inc_transport("hmac_verify_expired_total") + except Exception: # noqa: BLE001 — best-effort counter + pass + logger.warning(f"Request timestamp too old: {timestamp} vs current {current_time}") + return False + + # Recompute expected signature + expected = generate_hmac_signature(api_key, secret_key, timestamp, body) + + # Constant-time comparison to prevent timing attacks + return hmac.compare_digest(expected, signature) + + +def _signed_request_body(payload: dict[str, Any]) -> bytes: + """Serialise a JSON payload to the canonical bytes the HMAC signature is computed over. + + All four signed POST call sites must serialise via this helper and pass + the result with ``content=body`` to httpx (NOT ``json=...`` — that + re-serialises with different separators and breaks the HMAC match). + ``default=str`` accepts Decimal / bytes / datetime / UUID. + """ + return json.dumps(payload, separators=(",", ":"), default=str).encode("utf-8") + + +# ============================================================================= +# Retry with exponential backoff + jitter +# ============================================================================= + + +def _retry_with_backoff( + func: Callable[[], Any], + max_retries: int = 10, + base_delay: float = 0.5, + max_delay: float = 30.0, + backoff_factor: float = 2.0, + jitter: float = 0.1, + last_retry_after_seconds: float = 0.0, + on_transport_error: str | Callable[[Exception], dict[str, Any]] | None = None, + retry_on_5xx: bool = False, +) -> Any: + """Retry with exponential backoff + jitter; honors Retry-After (429) header. + + Formula (without Retry-After): delay = min(base_delay * backoff_factor^attempt, max_delay) + delay += random.uniform(-jitter * delay, jitter * delay) + Formula (with Retry-After): actual_delay = min(last_retry_after_seconds, max_delay) + + NR-006 (audit 2026-08-24): when ``retry_on_5xx=True`` a 5xx + response is treated as transient infrastructure failure and + retried via the same backoff path as network errors. After the + retry budget is exhausted the LAST 5xx response is returned + (not raised) so the caller can produce a deterministic + fail-CLOSED fallback — the audit's "fail-NO-CHECK" violation + happens when a 5xx short-circuits to a synthetic block without + any retry. Default ``retry_on_5xx=False`` preserves the + pre-existing /track and /execute semantics where 5xx is a + classified GATEWAY_ERROR that raises immediately. + """ + # Eager imports for the exception classes that the ``except`` + # branch below references. Lazy imports inside the ``try`` body + # shadow the name in this scope (Python treats any assignment + # to the name as a local binding), which raises + # ``UnboundLocalError`` when the except branch tries to + # pattern-match before the lazy import has fired. + from nullrun.breaker.exceptions import ( + NullRunAuthError, + NullRunBackendError, + ) + + last_exc: Exception | None = None + + for attempt in range(max_retries + 1): + try: + result = func() + + if hasattr(result, "status_code"): + if result.status_code == 401: + err = NullRunAuthError( + "Invalid API key", + error_code="NR-A003", + user_action=( + "The NullRun backend rejected the API key (401). " + "Verify it at https://app.nullrun.io/settings/api-keys " + "and rotate if it was revoked. The key may also be " + "for a different environment (prod vs. staging) — " + "check the API_URL vs. where the key was issued." + ), + ) + _emit_for_transport_error( + err, + "execute", + result.headers.get("x-correlation-id"), + status_code=result.status_code, + ) + raise err + if result.status_code >= 500 and on_transport_error == "raise": + # 5xx is a classified GATEWAY_ERROR. Don't retry; only raise + # when caller opted into the typed-error contract. + err = NullRunBackendError( + f"Gateway returned {result.status_code}", + endpoint="execute", + status_code=result.status_code, + ) + _emit_for_transport_error( + err, + "execute", + result.headers.get("x-correlation-id"), + status_code=result.status_code, + ) + raise err + if result.status_code >= 500 and retry_on_5xx and attempt < max_retries: + # NR-006: treat 5xx as transient infra failure and retry. + # Convert to HTTPStatusError so the except branch catches + # it as a retryable condition. After retry exhaustion + # the helper returns the last response (see below). + result.raise_for_status() + elif result.status_code >= 500 and not retry_on_5xx: + # Pre-NR-006 behaviour: 5xx without ``retry_on_5xx`` + # raises HTTPStatusError so the caller (e.g. + # ``Transport.execute``) can run its fallback logic + # after retry exhaustion produces BreakerTransportError. + # ``retry_on_5xx=True`` (the /gate path) takes the + # branch above instead and returns the last response. + result.raise_for_status() + # 4xx is a real gate decision — return the response so + # the caller can synthesize the appropriate fallback + # (Transport.check returns a synthetic block; Transport.execute + # returns a synthetic block; /track batch inspects status + # directly). Calling ``raise_for_status()`` here would force + # every caller into the except path and retry a permanent + # error — the audit's NR-006 PIN 3 pins this non-retry + # contract. + + return result + + except ( + BreakerTransportError, + NullRunAuthenticationError, + NullRunTransportError, + NullRunBackendError, + ): + raise + + except httpx.HTTPStatusError as exc: + # 5xx HTTPStatusError from the retry_on_5xx branch above. + # Treat as retryable transient infra failure. + last_exc = exc + if attempt >= max_retries: + break + + except Exception as exc: + last_exc = exc + metrics.set_transport("last_error", f"{type(exc).__name__}: {exc}") + if isinstance(exc, (httpx.TimeoutException, httpx.ConnectTimeout, httpx.ReadTimeout)): + metrics.inc_transport("timeouts") + + if attempt >= max_retries: + break + + metrics.inc_transport("retries_total") + + if last_retry_after_seconds > 0: + actual_delay = min(last_retry_after_seconds, max_delay) + last_retry_after_seconds = 0.0 + logger.warning( + "Request failed (attempt %d/%d), honoring Retry-After %.2fs: %s", + attempt + 1, + max_retries + 1, + actual_delay, + type(exc).__name__, + ) + else: + delay = min(base_delay * (backoff_factor**attempt), max_delay) + jitter_amount = delay * jitter + actual_delay = delay + random.uniform(-jitter_amount, jitter_amount) # noqa: S311 + actual_delay = max(0.0, actual_delay) + logger.warning( + "Request failed (attempt %d/%d), retrying in %.2fs: %s", + attempt + 1, + max_retries + 1, + actual_delay, + type(exc).__name__, + ) + + time.sleep(actual_delay) + + # Retry exhaustion. NR-006 path: if the caller opted into + # ``retry_on_5xx`` and the failure mode was 5xx, return the + # last response so the caller can synthesize a fallback + # (e.g. ``Transport.check`` returns the legacy synthetic-block + # shape). Other exhaustion paths (network errors, timeouts) + # still raise ``BreakerTransportError`` — pre-existing + # behaviour, unchanged. + if ( + retry_on_5xx + and last_exc is not None + and isinstance(last_exc, httpx.HTTPStatusError) + and last_exc.response is not None + ): + return last_exc.response + raise BreakerTransportError(f"Request failed after {max_retries + 1} attempts") from last_exc + + +# ============================================================================= +# Fallback Modes (SDK Resilience) +# ============================================================================= + + +class FallbackMode: + """ + SDK behavior when Gateway is unavailable. + + This is CRITICAL for production - Gateway unavailability should NOT + block agent execution, but behavior must be defined and logged. + """ + + # Block if Gateway unavailable. v3.53 audit #4 — DEFAULT for + # ``Transport.execute()`` and ``ExecuteConfig.fallback_mode``. + # Per CLAUDE.md §4 "DEFAULT: fail-CLOSED для всех enforcement + # путей", the /execute enforcement path must not silently allow + # local execution when the policy engine is unreachable. + STRICT = "strict" + # Allow if Gateway unavailable, log locally. **Opt-in only** — + # pass ``fallback_mode=FallbackMode.PERMISSIVE`` explicitly when + # the caller accepts silent fail-OPEN on the enforcement path. + # Required for any test / dev harness that intentionally runs + # without a live policy engine. + PERMISSIVE = "permissive" + + +class DecisionSource: + """ + Where the decision originated - for provenance tracking. + """ + + GATEWAY = "gateway" + CACHED = "cached" + FALLBACK = "fallback" + LOCAL = "local" + + +@dataclass +class FlushConfig: + """Configuration for transport flush behavior.""" + + batch_size: int = 50 + flush_interval: float = 5.0 # seconds + # Mirror _retry_with_backoff default. + max_retries: int = 10 + retry_delay: float = 1.0 # seconds + max_buffer_size: int = 1000 # Max events before dropping oldest + max_failed_flush: int = 10 # Circuit breaker: stop trying after this many failures + + +@dataclass +class ExecuteConfig: + """Configuration for execute (strict mode) behavior.""" + + # Fallback mode when Gateway is unavailable. v3.53 audit #4 — + # default is STRICT (fail-CLOSED on enforcement) per CLAUDE.md §4. + # Pre-v3.53 the default was PERMISSIVE which silently allowed + # local execution on transport failure; that was fail-OPEN on the + # primary enforcement path (Transport.execute → /api/v1/execute). + fallback_mode: str = FallbackMode.STRICT + # Gateway timeout in seconds + timeout: float = 5.0 + # Max retries for execute calls + max_retries: int = 10 + # Cache TTL for CACHED mode (seconds) + cache_ttl: float = 60.0 + # Cache max size + cache_max_size: int = 10000 + + +class Transport: + """ + HTTP transport with batching support. + + Features: + - Non-blocking track calls (append to buffer) + - Background flush at intervals or when batch_size reached + - Retry logic for failed requests + - Thread-safe for sync usage + - HMAC request signing for secure authentication + - Distributed circuit breaker via Redis for multi-worker deployments + """ + + def __init__( + self, + api_url: str, + api_key: str | None = None, + secret_key: str | None = None, + config: FlushConfig | None = None, + redis_client: Any = None, + ): + self.api_url = api_url.rstrip("/") + + # TLS enforcement: reject non-localhost HTTP. Uses urlparse + ip_address + # so homograph attacks (e.g. 127.0.0.1.attacker.com) don't slip through + # a naive startswith("127.") check. + from ipaddress import ip_address + from urllib.parse import urlparse + + parsed = urlparse(self.api_url) + if parsed.scheme == "http": + host = (parsed.hostname or "").lower() + allowed = host == "localhost" or host == "::1" + if not allowed: + try: + addr = ip_address(host) + allowed = addr.is_loopback + except ValueError: + allowed = False + if not allowed: + raise InsecureTransportError( + f"Insecure URL detected: {self.api_url}. " + f"HTTP is only allowed for localhost / 127.0.0.0/8 / ::1. " + f"Use https:// for production." + ) + + self.api_key = api_key + self.secret_key = secret_key # HMAC signing key + self.config = config or FlushConfig() + # Allow env-var override of batch size and flush interval. + if "NULLRUN_BATCH_SIZE" in os.environ: + try: + self.config.batch_size = int(os.environ["NULLRUN_BATCH_SIZE"]) + except ValueError: + logger.warning( + "NULLRUN_BATCH_SIZE=%r is not an int; ignoring", + os.environ["NULLRUN_BATCH_SIZE"], + ) + if "NULLRUN_FLUSH_INTERVAL_MS" in os.environ: + try: + self.config.flush_interval = int(os.environ["NULLRUN_FLUSH_INTERVAL_MS"]) / 1000.0 + except ValueError: + logger.warning( + "NULLRUN_FLUSH_INTERVAL_MS=%r is not an int; ignoring", + os.environ["NULLRUN_FLUSH_INTERVAL_MS"], + ) + self._buffer: list[dict[str, Any]] = [] + self._in_flight: dict[str, dict[str, Any]] = {} # event_id -> event for retry dedup + # RLock so re-entrant acquisition (e.g. test fixtures that hold the + # lock while calling lock-acquiring methods) doesn't deadlock. + self._lock = threading.RLock() + self._flush_thread: threading.Thread | None = None + self._running = False + # Cancellable sleep primitive: Event.wait returns immediately when + # stop() sets the event, so teardown doesn't block for the full + # flush_interval. Pin: tests/test_transport.py::test_stop_interrupts_flush_sleep. + self._stop_event = threading.Event() + + # mTLS client certificate support + # NULLRUN_TLS_CLIENT_CERT and NULLRUN_TLS_CLIENT_KEY env vars for client cert auth + client_cert_path = os.environ.get("NULLRUN_TLS_CLIENT_CERT") + client_key_path = os.environ.get("NULLRUN_TLS_CLIENT_KEY") + ca_cert_path = os.environ.get("NULLRUN_TLS_CA_CERT") # Optional custom CA + + # Build SSL configuration for mTLS + # For client cert auth: verify is a CA cert, cert is tuple of (client_cert, client_key) + verify_cert: bool | str = True + client_cert: tuple[str, str] | None = None + if client_cert_path and client_key_path: + # Client certificate authentication (mTLS) + client_cert = (client_cert_path, client_key_path) + verify_cert = ca_cert_path if ca_cert_path else True + logger.debug(f"mTLS enabled: client_cert={client_cert_path}") + elif ca_cert_path: + # Custom CA certificate only (no client cert) + verify_cert = ca_cert_path + logger.debug(f"Custom CA configured: ca_cert={ca_cert_path}") + + self._client = httpx.Client( + timeout=httpx.Timeout( + connect=5.0, + read=30.0, + write=10.0, + pool=5.0, + ), + verify=verify_cert, + cert=client_cert, + limits=httpx.Limits( + max_connections=10, + max_keepalive_connections=5, + keepalive_expiry=30.0, + ), + ) + self._redis_client = redis_client + self._circuit_breaker = CircuitBreaker( + failure_threshold=self.config.max_failed_flush, + recovery_timeout=30.0, + redis_client=redis_client, + name="transport", + ) + self._stopped = False # Track if stop was called + # 0.7.0 thin client: no local policy cache. Backend is authoritative. + _masked = api_key[:8] + "***" if api_key and len(api_key) >= 8 else "***" + logger.debug(f"Transport initialized: api_url={self.api_url}, api_key={_masked}") + + # OpenTelemetry tracer (lazy-loaded: only if opentelemetry is installed) + self._tracer = None + self._propagator = None + if _OTEL_AVAILABLE: + self._tracer = trace.get_tracer("nullrun.transport") + self._propagator = TraceContextTextMapPropagator() + + # Final-flush hook via weakref.finalize — only fires if this Transport + self._finalizer = weakref.finalize(self, self._atexit_flush_safe) + + @staticmethod + def _atexit_flush_safe(_self_id: int | None = None) -> None: + """Weakref finalizer entry point. + + ``weakref.finalize`` calls this with no arguments (``self`` is gone). + The recommended lifecycle is explicit ``stop()`` or ``with Transport(...)``. + If neither was used, we log a one-time DEBUG line. + """ + logger.debug( + "Transport finalizer fired without explicit stop(); " + "remaining events may be lost. Use Transport as a context " + "manager or call stop() explicitly." + ) + + # WAL rotation threshold (default 64 MB). Override via NULLRUN_WAL_MAX_BYTES. + _WAL_MAX_BYTES_DEFAULT: int = 64 * 1024 * 1024 + + @property + def _wal_max_bytes(self) -> int: + """Effective WAL rotation threshold.""" + raw = os.environ.get("NULLRUN_WAL_MAX_BYTES", "").strip() + if not raw: + return self._WAL_MAX_BYTES_DEFAULT + try: + value = int(raw) + return value if value > 0 else self._WAL_MAX_BYTES_DEFAULT + except ValueError: + return self._WAL_MAX_BYTES_DEFAULT + + def _wal_path(self) -> str: + """Resolve WAL path. Honours ``NULLRUN_WAL_PATH``; defaults to platform tempdir.""" + env_path = os.environ.get("NULLRUN_WAL_PATH") + if env_path: + return env_path + return os.path.join(tempfile.gettempdir(), "nullrun.wal") + + def _rotate_wal_if_needed(self) -> None: + """Rotate ```` to ``.1`` if it exceeds the size cap.""" + wal_path = self._wal_path() + try: + size = os.path.getsize(wal_path) + except OSError: + return + if size < self._wal_max_bytes: + return + rotated = f"{wal_path}.1" + try: + os.replace(wal_path, rotated) + logger.info( + f"WAL rotated: {wal_path} ({size} bytes) -> {rotated} " + f"after exceeding cap of {self._wal_max_bytes} bytes" + ) + except OSError as e: + logger.warning(f"Failed to rotate WAL {wal_path}: {e}") + + def _persist_to_wal(self) -> None: + """Persist unflushed events to WAL file for replay on restart.""" + if not self._buffer: + return + event_count = len(self._buffer) + wal_path = self._wal_path() + self._rotate_wal_if_needed() + wal_dir = os.path.dirname(wal_path) or "." + try: + os.makedirs(wal_dir, exist_ok=True) + except OSError as e: + logger.warning(f"Cannot create WAL directory {wal_dir}: {e}") + return + tmp_path = f"{wal_path}.tmp.{os.getpid()}" + try: + with open(tmp_path, "a") as f: + for event in self._buffer: + # 2026-07-24 (Decimal serialization): same default=str as + f.write(json.dumps(event, default=str) + "\n") + f.flush() + os.fsync(f.fileno()) + os.replace(tmp_path, wal_path) + self._buffer.clear() + logger.debug(f"Persisted {event_count} events to WAL at {wal_path}") + except OSError as e: + logger.warning(f"Failed to persist {event_count} events to WAL: {e}") + + def _replay_from_wal(self) -> None: + """Replay events from WAL file on startup. + + P1-5b: also drains the rotated ``.wal.1`` (oldest + surviving recovery window) before the active ``.wal`` so + a crash between rotation and replay doesn't lose events. + Both files are removed only after a successful flush. + """ + events: list[dict[str, Any]] = [] + for candidate in (f"{self._wal_path()}.1", self._wal_path()): + try: + with open(candidate) as f: + for line in f: + try: + events.append(json.loads(line.strip())) + except json.JSONDecodeError: + continue + except FileNotFoundError: + continue + except OSError as e: + logger.warning(f"Failed to read WAL {candidate}: {e}") + continue + try: + os.remove(candidate) + except OSError as e: + logger.warning(f"Failed to remove WAL {candidate}: {e}") + if events: + self._buffer.extend(events) + self._do_flush() + if events: + logger.info(f"Replayed {len(events)} events from WAL") + + def track(self, event: dict[str, Any]) -> None: + """ + Add event to buffer. Non-blocking. + + Events are flushed either when batch_size is reached or + flush_interval elapses. + """ + with self._lock: + # Generate event_id if not provided + if "event_id" not in event or not event["event_id"]: + event["event_id"] = str(uuid.uuid4()) + + # Store in-flight for retry dedup + self._in_flight[event["event_id"]] = event + + self._buffer.append(event) + metrics.inc_transport("events_enqueued") + + if len(self._buffer) >= self.config.batch_size: + self._do_flush_locked() + + def start(self) -> None: + """Start background flush thread.""" + if self._running: + return + # Replay any events from WAL that were persisted due to previous crash + self._replay_from_wal() + self._running = True + # Clear the stop latch so a previous stop() does not short-circuit + # the new flush loop on its first sleep. + self._stop_event.clear() + self._flush_thread = threading.Thread(target=self._flush_loop, daemon=True) + self._flush_thread.start() + logger.info("Transport flush thread started") + + def __enter__(self) -> "Transport": + """Context-manager entry: start the flush thread and return self. + + Pairs with ``__exit__`` so callers can write + ``with Transport(...) as t:`` and rely on ``stop `` running + on the way out. Replaces the manual ``start / stop `` pair + that was easy to forget in long-running services. + """ + self.start() + return self + + def __exit__(self, exc_type, exc_val, exc_tb) -> None: + """Context-manager exit: stop the flush thread and persist WAL. + + Always stops, regardless of whether the body raised. The + exception (if any) is NOT swallowed — the caller still sees + it after the with-block. + """ + try: + self.stop() + except Exception as e: # noqa: BLE001 — best-effort on context exit + logger.debug(f"Transport.__exit__: stop() raised: {e}") + + def stop(self, timeout: float = 10.0, flush: bool = True) -> None: + """Stop background flush thread and flush remaining events. + + Args: + timeout: max seconds to wait for the flush thread to exit. + flush: when True (default) the final ``_do_flush()`` and + ``_persist_to_wal()`` run after the thread joins. When + False, the thread is cancelled but the buffer is left + alone. The test conftest uses ``flush=False`` to teardown + between tests without a final httpx call. + """ + self._running = False + self._stopped = True # Mark as stopped to prevent double flush + self._stop_event.set() # Wake flush thread out of its cancellable sleep. + if self._flush_thread: + self._flush_thread.join(timeout=timeout) + if flush: + self._do_flush() # Final flush + self._persist_to_wal() # WAL any remaining events + self._client.close() + if getattr(self, "_finalizer", None) is not None and self._finalizer.alive: + self._finalizer.detach() + logger.info("Transport stopped") + + def _flush_loop(self) -> None: + """Background loop that periodically flushes.""" + while self._running: + # Event.wait returns True when stop() sets the event (cancel signal). + cancelled = self._stop_event.wait(timeout=self.config.flush_interval) + if cancelled: + break + if self._running: + self._do_flush() + + def _do_flush(self) -> None: + """Perform the actual flush.""" + with self._lock: + self._do_flush_locked() + + def _do_flush_locked(self) -> None: + """Flush under lock. Must be called with _lock held.""" + if not self._buffer: + logger.debug("Buffer empty, skipping flush") + return + + batch = self._buffer[:] + self._buffer.clear() + logger.debug(f"Sending batch of {len(batch)} events") + + # Circuit breaker wrapped send - uses proper 3-state circuit breaker + def send_batch(): + result = self._send_batch_with_retry_info(batch) + # Remove accepted events from in-flight + if result.accepted_event_ids: + for event in batch: + if event.get("event_id") in result.accepted_event_ids: + self._in_flight.pop(event.get("event_id"), None) + logger.debug(f"Flushed {len(batch)} events") + # Update metrics on successful flush (thread-safe) + metrics.inc_transport("batches_sent") + metrics.inc_transport("events_sent", len(batch)) + metrics.set_transport("last_flush_at", time.monotonic()) + return result + + try: + self._circuit_breaker.call(send_batch) + except BreakerTransportError: + logger.warning(f"Circuit breaker OPEN. Batch of {len(batch)} events will be re-queued.") + # Drop NEWEST non-critical (state_change etc.) so oldest events + # (incident start, billing-period start) survive — they power + # monthly rollups. Critical control-plane events are kept. + available_space = self.config.max_buffer_size - len(self._buffer) + if available_space < len(batch): + overflow = len(batch) - available_space + if overflow > 0: + batch = self._drop_newest_with_priority(batch, overflow) + self._buffer.extend(batch) # Append to END so oldest events retry first. + metrics.inc_transport("batches_failed") + + def _drain_batch(self) -> list[dict[str, Any]] | None: + """Public, lock-acquiring snapshot of the current buffer. Returns ``None`` when empty.""" + with self._lock: + if not self._buffer: + return None + batch = list(self._buffer) + del self._buffer[:] + return batch + + # Control-plane events that MUST NOT be dropped on overflow. + _CRITICAL_EVENT_TYPES = frozenset( + { + "state_change", + "kill_received", + "policy_invalidated", + "key_rotated", + } + ) + + def _drop_newest_with_priority( + self, + batch: list[dict[str, Any]], + overflow: int, + ) -> list[dict[str, Any]]: + """Drop ``overflow`` newest non-critical events; keep critical events and oldest. + + Cost-audit invariant: under overflow we keep the OLDEST events + (incident / billing-period start) — dropping oldest would silently + break monthly rollups. Never drop critical events at the cost of a + brief buffer overshoot. + """ + if overflow <= 0: + return batch + kept: list[dict[str, Any]] = [] + dropped = 0 + for event in reversed(batch): + if dropped < overflow and event.get("type") not in self._CRITICAL_EVENT_TYPES: + dropped += 1 + continue + kept.append(event) + if dropped > 0: + logger.warning( + f"buffer overflow: dropped {dropped} newest non-critical " + f"events (kept {len(kept)}, preserved {len(batch) - len(kept) - dropped} critical)" + ) + metrics.inc_transport("events_dropped", dropped) + kept.reverse() + return kept + + @dataclass + class SendResult: + accepted_event_ids: list[str] + retry_after_ms: float | None = None + is_policy_limit: bool = False + + def _add_hmac_headers(self, headers: dict[str, str], body: str | bytes) -> None: + """Add X-Signature-Timestamp + X-Signature headers. No-op if secret_key/api_key missing.""" + if not self.secret_key or not self.api_key: + return + + timestamp = int(time.time()) + signature = generate_hmac_signature( + self.api_key, + self.secret_key, + timestamp, + body, + ) + + headers["X-Signature-Timestamp"] = str(timestamp) + headers["X-Signature"] = signature + + def _build_signed_headers( + self, + body: str | bytes | None = None, + extra: dict[str, str] | None = None, + ) -> dict[str, str]: + """Build the canonical signed-headers dict for every signed POST. + + Always includes Content-Type: application/json and X-API-Key (when + api_key is set). Adds HMAC headers when secret_key is set and a + body is provided. ``extra`` is merged on top of defaults so callers + can override Content-Type or add custom headers. + """ + headers: dict[str, str] = { + "Content-Type": "application/json", + } + if self.api_key: + headers["X-API-Key"] = self.api_key + # Backend CSRF middleware bypasses cookie-double-submit when an + # Authorization header is present (backend/src/auth/csrf.rs). + # Without this, SDK POSTs hit the "state-changing request without + # session cookie" branch and get 403, which the SDK silently swallowed. + headers["Authorization"] = f"Bearer {self.api_key}" + if body is not None and self.secret_key and self.api_key: + timestamp = int(time.time()) + signature = generate_hmac_signature(self.api_key, self.secret_key, timestamp, body) + headers["X-Signature-Timestamp"] = str(timestamp) + headers["X-Signature"] = signature + if extra: + headers.update(extra) + # Backend rejects signed POSTs without X-NULLRUN-PROTOCOL: 3 with 400. + headers[HEADER_PROTOCOL] = _protocol_header_value() + self._inject_trace_context(headers) + return headers + + def _inject_trace_context(self, headers: dict[str, str]) -> None: + """ + Inject trace context into request headers (W3C Trace Context format). + + This enables distributed tracing across SDK and backend. + Uses W3C Trace Context standard for trace_id propagation. + """ + if not _OTEL_AVAILABLE or not self._propagator: + return + + carrier: dict[str, str] = {} + self._propagator.inject(carrier) + headers.update(carrier) + + def _extract_retry_after(self, response: httpx.Response) -> float | None: + """Extract Retry-After header value as seconds. + + Handles both: + - Integer seconds (e.g., "30") + - HTTP-date format (e.g., "Wed, 21 Oct 2015 07:28:00 GMT") + """ + retry_after = response.headers.get("Retry-After") + if not retry_after: + return None + + # Try parsing as seconds (integer or float) + try: + return float(retry_after) + except ValueError: + pass + + # Try parsing as HTTP datetime (RFC 7231) + try: + from email.utils import parsedate_to_datetime + + dt = parsedate_to_datetime(retry_after) + from datetime import datetime, timezone + + return (dt - datetime.now(timezone.utc)).total_seconds() + except Exception: + pass + + return None + + def _send_batch_with_retry_info(self, batch: list[dict[str, Any]]) -> "SendResult": + """Send batch to server. Returns SendResult with retry info. Wrapped by _retry_with_backoff.""" + logger.debug(f"Sending batch of {len(batch)} events to {self.api_url}/api/v1/track/batch") + body = _signed_request_body({"events": batch}) + headers = self._build_signed_headers(body=body) + + # Inner function is the unit of retry: + # * 5xx → retry helper backs off. 429 honors Retry-After. + # * 4xx (other than 429) → return as-is; these are real client bugs + # (auth, payload) and must NOT be retried. + def _post_batch() -> httpx.Response: + resp = self._client.post( + f"{self.api_url}/api/v1/track/batch", + content=body, + headers=headers, + ) + if resp.status_code >= 500 or resp.status_code == 429: + # raise_for_status turns this into HTTPStatusError; the retry + # helper wraps that into BreakerTransportError after retries. + resp.raise_for_status() + return resp + + max_track_retries = getattr(self, "_track_max_retries", 10) + response = _retry_with_backoff( + _post_batch, + max_retries=max_track_retries, + base_delay=0.5, + max_delay=10.0, + backoff_factor=2.0, + jitter=0.1, + ) + + # P0: Extract retry_after from response headers or body + retry_after_seconds: float | None = None + retry_after_ms: float | None = None + is_policy_limit = False + + # Check Retry-After header (may be seconds or HTTP-date) + retry_after_seconds = self._extract_retry_after(response) + + # Check response body for retry info + try: + data = response.json() + # Check for rejection info + if "rejected" in data and data["rejected"]: + rejected_info = data["rejected"] + if isinstance(rejected_info, dict): + if "retry_after_ms" in rejected_info: + retry_after_ms = rejected_info["retry_after_ms"] + if "reason" in rejected_info and rejected_info["reason"] == "policy_limit": + is_policy_limit = True + except Exception: # noqa: S110 + pass + + # Store for next retry calculation (prefer header seconds, fallback to body ms) + if retry_after_seconds is not None: + self._last_retry_after_seconds = retry_after_seconds + retry_after_ms = retry_after_seconds * 1000 + elif retry_after_ms is not None: + self._last_retry_after_seconds = retry_after_ms / 1000.0 + else: + self._last_retry_after_seconds = 0.0 + self._last_failure_policy_limit = is_policy_limit + + # Handle 429 response - extract and store Retry-After before raising + if response.status_code == 429: + retry_after = self._extract_retry_after(response) + if retry_after: + self._last_retry_after_seconds = retry_after + response.raise_for_status() + response.raise_for_status() + + # Process actions from server response. Per-element try/except so one + # malformed entry doesn't abort the whole loop. + try: + data = response.json() + actions = data.get("actions") or [] + for action in actions: + try: + if not isinstance(action, dict): + logger.warning("Skipping non-dict action from /track/batch: %r", action) + continue + action_type = action.get("type", "") + workflow_id = action.get("workflow_id", "unknown") + reason = action.get("reason", "") + if action_type: + handle_action(action_type, workflow_id, reason) + except Exception as item_err: + logger.warning("Skipping malformed action %r: %s", action, item_err) + for msg in data.get("messages", []) or []: + logger.info("Backend message: %s", msg) + except Exception as e: + logger.warning(f"Failed to process actions: {e}") + + # Return accepted event_ids for retry dedup + accepted_event_ids = data.get("accepted_event_ids", []) if "data" in locals() else [] + logger.debug(f"Batch track: sent {len(batch)} events") + return self.SendResult( + accepted_event_ids=accepted_event_ids, + retry_after_ms=retry_after_ms, + is_policy_limit=is_policy_limit, + ) + + def flush_now(self) -> None: + """Force immediate flush.""" + self._do_flush() + + # ============================================================================= + # Execute (Strict Mode) + # ============================================================================= + + def execute( + self, + organization_id: str, + execution_id: str, + trace_id: str, + tool: str, + input_data: dict[str, Any], + mode: str = "auto", + # v3.53 audit #4 — default flipped from PERMISSIVE to STRICT + # to match CLAUDE.md §4 ("DEFAULT: fail-CLOSED для всех + # enforcement путей"). /execute is the primary enforcement + # point (see docstring) — when the gateway is unreachable the + # body MUST NOT run on a silent local pass. Callers that + # intentionally want fail-OPEN on this path (dev / test + # harnesses without a live engine) must opt in by passing + # ``fallback_mode=FallbackMode.PERMISSIVE`` explicitly. + fallback_mode: str = FallbackMode.STRICT, + operation_id: str | None = None, + approval_id: str | None = None, + # Typed-impact + digest-bound approval. Forwarded when @sensitive(impact=...) + # built them so the backend can stamp the approval row with the digest. + business_impact: dict[str, Any] | None = None, + action_digest: str | None = None, + # Tool-call argument bag forwarded on /execute so the gate can compute + # a schema fingerprint and write it to mcp_tool_signatures. + tool_arguments: dict[str, Any] | None = None, + # Per-call `tools` list forwarded on /execute so the backend's + # Step 3 tool_block check (`backend/src/proxy/http/gate/orchestrator.rs:1847-1893`) + # can match each tool against the workflow's effective `tool_patterns` + # aggregate. Without this, TB-1 fails closed with `no_tools_field` + # whenever the workflow has an active `policy.tool_patterns` block. + # Populated by `runtime.execute` from the `get_call_tools()` contextvar + # when the caller invoked `set_call_context(tools=...)` (or the + # `_enforce_sensitive_tool` decorator did so on their behalf). + tools: tuple[str, ...] | None = None, + on_transport_error: Callable[[Exception], dict[str, Any]] | None = None, + ) -> dict[str, Any]: + """Pre-execution policy evaluation via /api/v1/execute (PRIMARY enforcement point). + + Wire contract (revised 2026-09-08, DEFS-SDKEXEC-GATE-FIRST): + /execute REQUIRES a prior /gate call that minted the same + ``execution_id`` and registered the ``execution:{id}`` binding + in Redis. Backend enforcement: + ``backend/src/proxy/http/gate/execute.rs:46-208`` + (DEF-SDKK-022-EXEC-BYPASS, 2026-09-04, RUN_ID=20260904T1500) + runs ``HGET execution:{id} ORG_FIELD`` on entry; a miss + returns 404 EXECUTION_NOT_FOUND (fail-CLOSED). The SDK + therefore MUST thread the execution_id captured by + ``runtime.check_workflow_budget`` (which calls ``Transport.check``, + i.e. /gate) into the body of this /execute call. See + ``runtime.execute()`` (line ~2820) for the reuse path; this + method's caller is the single source of truth for + ``execution_id`` selection. + + Prior to DEF-SDKK-022 the comment here claimed "/execute MUST + be called rather than /gate" — that contract was the legacy + pre-2026-09-04 shape. The post-fix shape is "/execute MUST be + preceded by /gate for the same execution_id" — the budget + pre-flight (Transport.check, /api/v1/gate) is the binding + registrar; /execute is the policy decision that re-uses it. + + Args: + organization_id: Organization identifier + execution_id: Execution identifier + trace_id: Distributed trace ID + tool: Tool to execute + input_data: Tool input + mode: Execution mode ("auto", "inline", "strict") + fallback_mode: What to do if Gateway unavailable + operation_id: Optional idempotency key + on_transport_error: Optional callback invoked on BreakerTransportError. + When set, the callback's return value is returned verbatim; otherwise + the request falls through to fallback_mode. The decorator's + _enforce_sensitive_tool sets this to convert the error into a + NullRunBlockedException (fail-CLOSED). + + Returns: + Dict with: + - decision: "allow" | "block" | "flag" | "pause" | "require_approval" + - decision_source: "gateway" | "cached" | "fallback" + - explanation: Human-readable explanation + - policy_hash: Server-side SHA-256 of the policy applied + (v4 wire field; null on pre-v4 backends). NOT a + sequential `policy_version` number — wire v3/v4 backends + emit only `policy_hash`. Synthetic fallback dicts ship + `policy_version: 0` for legacy compatibility; real + responses populate `policy_hash` only. + - decision_context: Context for replay (if available) + """ + gate_request = { + "organization_id": organization_id, + "execution_id": execution_id, + "trace_id": trace_id, + "tool": tool, + "input": input_data, + "mode": mode, # Wire-present but unused by backend; kept for compat. + "operation_id": operation_id or str(uuid.uuid4()), + } + if approval_id is not None: + gate_request["approval_id"] = approval_id + if business_impact is not None: + gate_request["business_impact"] = business_impact + if action_digest is not None: + gate_request["action_digest"] = action_digest + if tool_arguments is not None: + gate_request["tool_arguments"] = tool_arguments + if tools is not None: + gate_request["tools"] = list(tools) + + body = _signed_request_body(gate_request) + headers = self._build_signed_headers(body=body) + + def do_execute_request() -> httpx.Response: + return self._client.post( + f"{self.api_url}/api/v1/execute", + content=body, + headers=headers, + timeout=5.0, + ) + + # Per-instance override so tests/CI can shrink the retry budget. + max_execute_retries = getattr(self, "_execute_max_retries", 10) + try: + response = _retry_with_backoff( + do_execute_request, + max_retries=max_execute_retries, + base_delay=0.5, + on_transport_error=on_transport_error, + ) + + if response.status_code == 200: + data = response.json() + data["decision_source"] = DecisionSource.GATEWAY + # 0.7.0 thin client: no local policy cache. The next + return data # type: ignore[no-any-return] + elif response.status_code >= 400: + # 4xx — don't retry. + # + # 2026-09-10 (NR-SDK-A015-SURFACE): before the fix, + # this branch dropped the wire envelope on the floor + # and synthesised a generic ``{"decision": "block", + # "explanation": "Gateway returned 409"}`` dict. That + # hid every wire-coded reason (`APPROVAL_REPLAY_REJECTED`, + # `APPROVAL_DENIED`, `BUDGET_HARD_BLOCKED`, etc.) behind + # a single string, so the runtime block dispatch fell + # through to ``NR-X001`` and `format_user_message` + # produced the catalogue fallback ("Something went + # wrong. Please try again.") instead of the typed + # `NR-A015` message. Cookbook callers had no way to + # branch on the precise cause. + # + # Post-fix: parse the envelope via the existing + # `_parse_v3_error_envelope` helper — it covers the + # v3 wire envelope for every /execute reject reason, + # including the six typed approval grant-consume + # outcomes (`APPROVAL_NOT_YET_APPROVED` → + # ``NullRunApprovalNotYetApprovedError`` (NR-A010), + # `APPROVAL_DENIED` → NR-A011, + # `APPROVAL_EXPIRED` → NR-A012, + # `APPROVAL_DIGEST_MISMATCH` → NR-A013, + # `APPROVAL_TOOL_DIGEST_MISMATCH` → NR-A014, + # `APPROVAL_REPLAY_REJECTED` → NR-A015 / `` + # NullRunApprovalReplayRejectedError``) — and raise + # the typed exception so the @protect / + # @sensitive / runtime.execute() exception arms + # propagate the right class up to the caller. + # + # Fall through to the synthetic block shape if the + # envelope is unrecognised (plaintext body, malformed + # JSON, unknown wire code) so behaviour stays + # backwards-compatible for legacy / non-v3 backends. + # `_parse_v3_error_envelope` always returns an + # Exception — it never silently swallows a 4xx. + try: + raise _parse_v3_error_envelope(response, "execute") + except NullRunApprovalReplayRejectedError as exc: + # The exact case the user reported: the operator + # approved, the SDK polled /execute again, and + # the backend's atomic consume_approved UPDATE + # returned zero rows (replay race — UI approve + # vs SDK re-check). Surface the typed exception + # so `format_user_message` yields the NR-A015 + # catalogue line ("Your request couldn't be + # completed because the approval has already + # been used. Please start a new request.") + # instead of the fallback. + metrics.inc_transport("execute_block_replay_rejected") + raise + except NullRunBlockedException as exc: + # All other typed blocks from the dispatch — + # budget, rate, tool, approval-deny, etc. + # Re-raise for the @protect / runtime.execute + # arms to handle. + metrics.inc_transport("execute_block_typed") + raise + except NullRunBackendError as exc: + # 5xx-classified envelope parsed as a typed + # backend error (shouldn't normally land here + # because the helper maps 5xx to GATEWAY_ERROR + # via NullRunTransportError, but stays + # defensive). Re-raise. + raise + except NullRunAuthenticationError as exc: + # 401 envelope parsed as auth error — surface + # directly so the caller can react. + raise + except NullRunTransportError as exc: + # Transport-classified (network, breaker) — not + # a real 4xx, but helper may return one if the + # envelope shape is ambiguous. Re-raise so the + # on_transport_error arm sees it. + raise + except NullRunDecision as exc: + # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): + # umbrella pass-through for typed Decision + # subclasses NOT in the NullRunBlockedException + # MRO. Specifically: + # - NullRunChainError (NR-CH001) — chain + # lifetime / cross-org / Execution Graph + # parent-lineage rejections + # - NullRunWorkflowInactiveError (NR-W004) — + # soft-deleted workflow + # - NullRunConsumeOverbudgetError (NR-O001) — + # CONSUME > RESERVE + epsilon_cents invariant + # - WorkflowPausedException (NR-W003) + # Pre-fix these fell through to `except + # Exception: pass` below and got silently + # swallowed into the synthetic block shape + # (`{"decision": "block", "decision_source": + # FALLBACK, "explanation": f"Gateway returned + # {response.status_code}"}`) — losing + # exc.chain_id / exc.parent_execution_id (Chain), + # exc.workflow_id (WorkflowInactive), + # exc.execution_id / exc.reserved_cents / + # exc.actual_cost_cents / exc.epsilon_cents + # (ConsumeOverbudget), and every typed + # `error_code`/user-action. MUST come AFTER the + # NullRunBlockedException arm above so the typed + # approval / budget / tool-block path still + # matches by MRO specificity. + metrics.inc_transport("execute_block_decision_typed") + raise + except NullRunInfrastructureError as exc: + # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): + # umbrella pass-through for typed + # Infrastructure subclasses NOT in the + # NullRunBackendError / NullRunAuthenticationError / + # NullRunTransportError MRO branches above. + # Specifically: + # - NullRunProtocolError (NR-P001) — + # PROTOCOL_TOO_OLD / PROTOCOL_TOO_NEW / + # PROTOCOL_HEADER_INVALID / + # PROTOCOL_HEADER_REQUIRED + # - NullRunRateLimitRedisError (NR-R002) — + # RATE_LIMIT_REDIS_UNAVAILABLE + # - NullRunConfigError (NR-Cxxx) — when raised + # from a wire envelope (rare; mostly SDK-side) + # NullRunAuthError (NR-A003) IS in the + # NullRunAuthenticationError arm above (parent + # class match), but listing here for completeness + # preserves the documented recovery contract + # even if a future refactor reorders the prior + # arms. + # Pre-fix these fell through to `except Exception: + # pass` below — same synthetic-block loss as the + # Decision path. MUST come AFTER the three + # specific parent arms above (Backend, Auth, + # Transport) so the wire-classified exceptions + # still match by MRO specificity. + metrics.inc_transport("execute_block_infra_typed") + raise + except Exception: + # Unrecognised envelope (plaintext body, legacy + # slug, malformed JSON). Fall through to the + # synthetic block shape so old / non-v3 backends + # keep working and ``on_transport_error="raise"`` + # callers still see a usable dict. The retry + # helper has already given up; emitting a typed + # exception here would mask unknown wire codes + # the user hasn't yet catalogued. + pass + return { + "decision": "block", + "decision_source": DecisionSource.FALLBACK, + "explanation": f"Gateway returned {response.status_code}", + "policy_hash": None, + } + + except BreakerTransportError as exc: + # ADR-008: on_transport_error accepts callables AND strings: + if callable(on_transport_error): + return on_transport_error(exc) + if on_transport_error == "raise": + raise NullRunTransportError( + f"Gateway unreachable on /execute: {exc}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="execute", + ) from exc + if on_transport_error == "open": + return { + "decision": "allow", + "decision_source": TransportErrorSource.NETWORK_ERROR, + "explanation": f"Gateway unreachable: {exc}", + "policy_hash": None, + } + if on_transport_error == "closed": + return { + "decision": "block", + "decision_source": TransportErrorSource.NETWORK_ERROR, + "explanation": f"Gateway unreachable: {exc}", + "policy_hash": None, + } + pass # fall through to fallback mode + except NullRunTransportError: + raise # Already classified -- propagate as-is + except httpx.RequestError as exc: + if callable(on_transport_error): + return on_transport_error(exc) + if on_transport_error == "raise": + raise NullRunTransportError( + f"Network error on /execute: {exc}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="execute", + ) from exc + raise + except NullRunAuthenticationError: + raise # Don't fall back on auth errors + + # All attempts failed - apply fallback mode. + metrics.inc_transport("fallback_mode_activations") + if fallback_mode == FallbackMode.STRICT: + return { + "decision": "block", + "decision_source": DecisionSource.FALLBACK, + "explanation": "Gateway unavailable, fallback=STRICT", + "policy_version": 0, + } + else: # PERMISSIVE (opt-in) + # v3.53 audit #4 — PERMISSIVE no longer the default; it + # requires the caller to pass fallback_mode=FallbackMode. + # PERMISSIVE explicitly. Synthesizes an allow + decision_ + # source=FALLBACK so the caller / @sensitive decorator can + # still observe that the engine was unreachable. + return { + "decision": "allow", + "decision_source": DecisionSource.FALLBACK, + "explanation": "Gateway unavailable, fallback=PERMISSIVE", + "policy_version": 0, + } + + def check( + self, + check_request: dict[str, Any], + on_transport_error: Callable[[Exception], dict[str, Any]] | str | None = None, + parent_execution_id: str | None = None, + ) -> dict[str, Any]: + """ + Call /api/v1/gate endpoint for pre-execution budget checking. + + Uses the unified gate endpoint with check_type for budget validation. + Supports idempotency via operation_id field. + + Args: + check_request: Dict with: + - organization_id: Organization identifier + - execution_id: Execution identifier + - operation_id: Operation identifier (for idempotency) + - check_type: "llm" or "tool" + - model: Model name (for LLM checks) + - tool_name: Tool name (for tool checks) + - estimated_tokens: Token count (for LLM checks) + - input: Optional input data + + Returns: + Dict with: + - decision: "allow" | "block" | "throttle" + - reservation_id: Optional reservation ID + - remaining_budget_cents: Remaining budget + - projected_cost_cents: Projected cost for this operation + - explanations: List of explanation strings + - suggestions: List of suggestion strings + """ + # Convert check_request to gate_request format + gate_request = { + "organization_id": check_request.get("organization_id"), + "execution_id": check_request.get("execution_id"), + "trace_id": check_request.get("trace_id", str(uuid.uuid4())), + "tool": check_request.get("tool_name") or check_request.get("tool"), + "input": check_request.get("input"), + "mode": "auto", + "check_type": check_request.get("check_type"), + "model": check_request.get("model"), + "estimated_tokens": check_request.get("estimated_tokens"), + "operation_id": check_request.get("operation_id") or str(uuid.uuid4()), + # Forward the per-call `tools` list so the backend's + # `gate/internal.rs::check_tool_block` can match each + # tool against the workflow's effective `blocked_tools` + # aggregate. When unset (None) we omit the key entirely + # -- the backend distinguishes "no tools sent" from + # "explicit []". + **({"tools": check_request["tools"]} if "tools" in check_request else {}), + } + + # Wire-protocol v3 fields. Forwarded only when present so + if check_request.get("chain_id") is not None: + gate_request["chain_id"] = check_request["chain_id"] + if check_request.get("chain_op") is not None: + gate_request["chain_op"] = check_request["chain_op"] + if check_request.get("idempotency_key") is not None: + gate_request["idempotency_key"] = check_request["idempotency_key"] + if "stream" in check_request: + gate_request["stream"] = bool(check_request["stream"]) + # v0.16.1 (Phase-1+ wire-shape fix): runtime.check_workflow_budget + # always sets `action_digest` so the gate's + # `if req.action_digest.is_none()` version-gate passes + # (`backend/src/proxy/http/gate/gate.rs:56`, ADR-023 P1-6). + # Pre-v0.16.1 / Phase-0 callers can still omit it (forwarded + # only when truthy) without triggering a "field present + # but None" wire-shape drift. + if check_request.get("action_digest"): + gate_request["action_digest"] = check_request["action_digest"] + # Forward the `tool_arguments` bag alongside `tool` so + # the gate can hash it via `signature::compute_schema_hash` + # and write the fingerprint into `mcp_tool_signatures`. + # Legacy SDKs never set this; the backend's gate falls + # back to `tool_params` when the field is missing, so + # legacy callers do not regress. The shape is + # `Optional[dict[str, Any]]` -- the backend + # canonicalises the JSON before hashing, so field + # ordering inside the dict does not affect the + # fingerprint. + if "tool_arguments" in check_request and check_request["tool_arguments"] is not None: + gate_request["tool_arguments"] = check_request["tool_arguments"] + # Execution Graph v0 (2026-08-06, backend): additive + _parent_execution_id = check_request.get("parent_execution_id", parent_execution_id) + if _parent_execution_id is not None: + gate_request["parent_execution_id"] = _parent_execution_id + + # 2026-07-02 (v0.11.0 refactor): route through the canonical + body = _signed_request_body(gate_request) + headers = self._build_signed_headers(body=body) + + # NR-006 (audit 2026-08-24): wrap the gate POST in + # ``_retry_with_backoff`` with ``retry_on_5xx=True`` and + # ``max_retries=3`` (per audit recommendation: "less than + # 10 — /gate is critical and too many retries amplify + # load"). Pre-fix this code path returned a synthetic block + # on the FIRST 5xx — the agent caller never received a real + # gate decision, violating CLAUDE.md §4 "fail-CLOSED ≠ + # fail-NO-CHECK". A transient 503 from a rolling deploy + # would silently flip every agent to "budget blocked" even + # though the budget was fine. + def _do_gate_post() -> httpx.Response: + return self._client.post( + f"{self.api_url}/api/v1/gate", + content=body, + headers=headers, + timeout=5.0, + ) + + try: + response = _retry_with_backoff( + _do_gate_post, + max_retries=3, + base_delay=0.5, + max_delay=10.0, + backoff_factor=2.0, + jitter=0.1, + retry_on_5xx=True, + on_transport_error=on_transport_error, + ) + + if response.status_code == 200: + return response.json() # type: ignore[no-any-return] + # 4xx always -> synthetic block (real gate decision, + # never retried by ``_retry_with_backoff``). 5xx after + # retry exhaustion -> synthetic block (legacy + # fallback path preserved). + if response.status_code >= 500 and on_transport_error == "raise": + # Defence-in-depth: the helper raises 5xx-with-raise + # inside the retry loop, but if a path slips through + # (e.g. operator passes on_transport_error after + # exhaustion), we still surface the typed error + # rather than the silent synthetic block. + raise NullRunTransportError( + f"Gateway returned {response.status_code}", + source=TransportErrorSource.GATEWAY_ERROR, + endpoint="check", + status_code=response.status_code, + ) + return { + "decision": "block", + "decision_source": DecisionSource.FALLBACK, + "reservation_id": None, + "remaining_budget_cents": 0, + "projected_cost_cents": 0, + "explanations": [f"Gate endpoint returned {response.status_code}"], + "suggestions": ["Check API availability"], + } + except httpx.RequestError as e: + # NR-006: ``_retry_with_backoff`` re-raises network errors + # after retry exhaustion as ``BreakerTransportError``, but + # ``httpx.RequestError`` can still surface when the helper + # raises mid-loop on a non-retryable path (e.g. caller + # passes ``max_retries=0``). Translate to either a + # typed ``NullRunTransportError`` (opt-in) or a synthetic + # block (legacy). + if on_transport_error == "raise": + raise NullRunTransportError( + f"Network error on /check: {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="check", + ) from e + logger.warning(f"Gate request failed: {e}") + return { + "decision": "block", + "decision_source": DecisionSource.FALLBACK, + "reservation_id": None, + "remaining_budget_cents": 0, + "projected_cost_cents": 0, + "explanations": [f"Gate request failed: {e}"], + "suggestions": ["Check API availability"], + } + except BreakerTransportError as e: + # NR-006: the helper exhausted the retry budget on network + # errors and re-raised as ``BreakerTransportError``. Apply + # the same translation rule as ``httpx.RequestError`` + # above so the legacy ``on_transport_error`` opt-in + # contract is preserved — opt-in → typed error, default + # → synthetic block. + if on_transport_error == "raise": + raise NullRunTransportError( + f"Network error on /check after retry exhaustion: {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="check", + ) from e + logger.warning(f"Gate request failed after retries: {e}") + return { + "decision": "block", + "decision_source": DecisionSource.FALLBACK, + "reservation_id": None, + "remaining_budget_cents": 0, + "projected_cost_cents": 0, + "explanations": [f"Gate request failed after retries: {e}"], + "suggestions": ["Check API availability"], + } + + # ============================================================================= + # WebSocket Connection + # ============================================================================= + + async def connect_websocket( + self, + organization_id: str, + on_state_change: Callable[[dict[str, Any]], None] | None = None, + on_policy_invalidated: Callable[[str, str, int], None] | None = None, + on_key_rotated: Callable[[str, str, int], None] | None = None, + on_approval_resolved: Callable[[dict[str, Any]], None] | None = None, + ) -> "WebSocketConnection": + """ + Connect to WebSocket control plane for real-time workflow state updates. + + This replaces polling GET /status/{workflow_id} with WebSocket push. + When the workflow state changes (KILL/PAUSE), the server pushes the update. + + Args: + organization_id: Organization identifier + on_state_change: Optional callback for state change notifications + on_policy_invalidated: Optional callback for policy cache invalidation. + When called, clears local policy cache so next + gate/execute fetches fresh policy from backend. + Args: (organization_id, policy_id, new_version) + on_key_rotated: Optional callback for HMAC key rotation. + When called, should re-fetch secret_key from /auth/verify. + Args: (organization_id, key_id, new_version) + + Returns: + WebSocketConnection instance + + Raises: + ConnectionError: If WebSocket connection fails + """ + # Build the WS URL via urllib.parse instead of string + # replace. Reject unknown schemes with a clear error. + from urllib.parse import urlparse, urlunparse + + from nullrun.transport_websocket import WebSocketConnection + + parsed = urlparse(self.api_url) + if parsed.scheme not in ("http", "https"): + raise ValueError(f"Unsupported scheme for control plane: {parsed.scheme!r}") + ws_scheme = "wss" if parsed.scheme == "https" else "ws" + ws_url = urlunparse( + parsed._replace( + scheme=ws_scheme, + path=f"/ws/control/{organization_id}", + params="", + query="", + fragment="", + ) + ) + + # WS upgrade is a GET-with-no-body so the signed-headers helper (which + # adds HMAC for the body) does not fit. Use the GET helper instead — + # same Content-Type + X-API-Key + Authorization + X-NULLRUN-PROTOCOL + # + trace context shape, no HMAC. The backend's protocol middleware + # runs on the WS upgrade path too, so the header is mandatory here. + headers = self._auth_headers_for_get() + + # 0.7.0 thin client: no local policy cache; the next /gate or /execute + # call re-reads from the backend. Just forward the notification. + async def wrapped_policy_invalidated(ws_id: str, policy_id: str, new_version: int) -> None: + logger.info(f"Policy {policy_id} invalidated (v{new_version})") + if on_policy_invalidated: + on_policy_invalidated(ws_id, policy_id, new_version) + + async def wrapped_key_rotated(ws_id: str, key_id: str, new_version: int) -> None: + logger.info(f"Key {key_id} rotated (v{new_version}), re-fetching credentials") + await self._refetch_credentials() + if on_key_rotated: + on_key_rotated(ws_id, key_id, new_version) + + # Synchronous adapter: dispatch is dict-only, not awaitable. An + # async def would produce a coroutine the handler ignores. + def wrapped_approval_resolved(payload: dict[str, Any]) -> None: + if on_approval_resolved: + on_approval_resolved(payload) + + conn = WebSocketConnection( + url=ws_url, + headers=headers, + api_key=self.api_key, + secret_key=self.secret_key, + on_state_change=on_state_change, + on_policy_invalidated=wrapped_policy_invalidated, + on_key_rotated=wrapped_key_rotated, + on_approval_resolved=wrapped_approval_resolved, + ) + await conn.connect() + return conn + + async def _refetch_credentials(self) -> None: + """Re-fetch credentials from /auth/verify after key rotation. + + Routes through ``self._client`` so the same TLS configuration, + connection pool, and HMAC signing path apply. Body is serialised via + ``_signed_request_body`` so the wire bytes match the signed bytes. + """ + try: + payload = {"api_key": self.api_key} + body = _signed_request_body(payload) + headers = self._build_signed_headers(body=body) + + response = self._client.post( + # P0 #5: contract drift — other auth-verify call sites + # in this file use `/api/v1/auth/verify` (see runtime.py:599). + # Align this rotation call site to the same v1 prefix so the + # contract-drift-guard CI catches future divergence. + f"{self.api_url}/api/v1/auth/verify", + content=body, + headers=headers, + timeout=10.0, + ) + if response.status_code == 200: + data = response.json() + new_secret = data.get("secret_key") + if new_secret: + logger.info("Successfully fetched new secret_key from /auth/verify") + self.secret_key = new_secret + else: + logger.warning("/auth/verify did not return secret_key in response") + else: + logger.warning(f"Failed to refetch credentials: {response.status_code}") + except Exception as e: + logger.error(f"Error refetching credentials: {e}") + + # ============================================================================= + # Wire-protocol v3 endpoints + # ============================================================================= + # + # The v3 wire contract adds six endpoints that the legacy /gate + + # /execute + /track/batch surface does not cover. Each new method + # follows the same shape as the existing `check` method: + # + # 1. Build headers via ``_build_signed_headers`` (gets X-API-Key + + # Authorization + X-NULLRUN-PROTOCOL + HMAC + trace context). + # 2. Serialise the body via ``_signed_request_body`` so the wire + # bytes match the HMAC-signed bytes. + # 3. POST through the shared ``self._client`` (mTLS, connection + # pool, circuit breaker all apply). + # 4. Map non-2xx responses through ``_parse_v3_error_envelope`` + # so callers can ``except NullRunBudgetError`` / ``except + # NullRunConsumeOverbudgetError`` / etc. without parsing the + # raw error_code string. + + def check_v3( + self, + request: dict[str, Any], + on_transport_error: Callable[[Exception], dict[str, Any]] | str | None = None, + ) -> dict[str, Any]: + """Pre-execution gate — wire-protocol v3 (B1 fix 2026-07-04). + + Pre-fix this method POSTed to ``/api/v1/check``. That endpoint + was removed on 2026-06-27 — the handler now returns + ``410 Gone`` with a ``replacement: /api/v1/gate`` hint. The + SDK's ``check `` method already targets ``/api/v1/gate`` and + forwards every v3 wire field — ``chain_id`` + ``chain_op``, ``idempotency_key``, ``stream``. This method + is kept as a v3-named alias so existing call sites and tests + continue to work; internally it delegates to ``check `` with + the same body. + + Args: + request: Gate request body. Must include ``organization_id`` + ``execution_id`` (for backward compat — server mints its + own on /check), ``operation_id``, and ``check_type``. + on_transport_error: Mirrors the ``check `` flag. + + Returns: + Parsed JSON dict, augmented with ``decision_source = + DecisionSource.GATEWAY`` so callers distinguish it from a + fallback synthetic response. + + Raises: + NullRunAuthenticationError: 401/403 (PROTOCOL_TOO_OLD + PROTOCOL_TOO_NEW, API_KEY_REVOKED, CHAIN_CROSS_ORG). + NullRunConsumeOverbudgetError: 422 (placeholder for /track + not raised on /gate). + NullRunBudgetError: 402 BUDGET_HARD_BLOCKED / + BUDGET_SOFT_BLOCKED / BUDGET_OVERDRAFT_EXCEEDED. + NullRunChainError: 402 CHAIN_MAX_DURATION_EXCEEDED / + 403 CHAIN_ORG_MISMATCH. + NullRunWorkflowInactiveError: 403 WORKFLOW_INACTIVE. + NullRunBackendError: 5xx / BUDGET_DATA_UNAVAILABLE / + RATE_LIMIT_REDIS_UNAVAILABLE. + """ + # 2026-07-04 (B1): /api/v1/check returns 410 Gone. + return self.check(request, on_transport_error=on_transport_error) + + def track_single( + self, + request: dict[str, Any], + ) -> dict[str, Any]: + """POST /api/v1/track — wire-protocol v3 single-event consume. + + . The single-event path is the v3 + replacement for the legacy `/api/v1/track/batch` POST body. + It runs the CONSUME_SCRIPT invariant + ``actual_cost <= reserved_cents + epsilon_cents`` (§25 + ADR-005) and rejects with 422 CONSUME_OVERBUDGET on + violation. The reserved binding is the one created by the + matching ``/check`` call (same ``reservation_id``). + + The wire shape is built by ``runtime._build_v3_track_payload`` + (see ``runtime.py:2679-2776``); this method just forwards + whatever dict the caller hands it. The post-fix schema is: + + Args: + request: Consume request body. Must include: + + * ``reservation_id`` (str, server-minted uuidv7 from + the matching /check response — wired via + ``_capture_server_minted_execution_id``) + * ``workflow_id`` (str, the workflow the call belongs to) + * ``tokens`` (int, sum of input + output tokens) + * ``cost_cents`` (int, ``0`` — backend computes the + authoritative cost from tokens + the org's + pricing policy; sending a wrong number risks + double-billing, see _WIRE_STRIP_FIELDS in runtime.py) + * ``cost_source`` (str, ``"provisional"`` / + ``"authoritative"`` per — SDK always emits + ``"provisional"``) + + Optional fields: ``input_tokens``, ``output_tokens`` + ``model``, ``latency_ms``, ``metadata``, ``trace_id`` + ``span_id``, ``agent_id``, ``environment`` + ``agent_type``, ``attempt_index``, ``is_retry`` + ``idempotency_key``. + + Returns: + Parsed JSON dict from the backend's TrackResponse. + NOTE: there is NO top-level ``status`` field on the + wire — the legacy pre-v3 docstring claimed one, but + v3/v4 backends emit + ``{snapshot, actions_taken, processing_mode, + cost_source, confidence, event_id, + idempotent_replay, stored_response?}``. SDK callers + branch on the HTTP status (200 vs 4xx/5xx) and on + ``idempotent_replay`` (bool) for replay detection — + do NOT read ``data["status"]`` (KeyError on every + backend >= 3.66.2). + + Raises: + NullRunConsumeOverbudgetError: 422 CONSUME_OVERBUDGET — + ``actual_cost > reserved + epsilon_cents``. The + reservation is NOT silently re-reserved. + NullRunBackendError: 503 RESERVATION_NOT_FOUND / + EXECUTION_NOT_BOUND. + NullRunAuthenticationError: 401/403. + + 2026-07-04 (B2): pre-fix this docstring (and the + surrounding module comment) described a fictitious wire + shape ``{execution_id, actual_cost_cents, api_key_id + cost_source}``. The backend's actual ``TrackRequestRaw`` is + ``{workflow_id, tokens, cost_cents,...}``; ``execution_id`` + is replaced by ``reservation_id``, ``actual_cost_cents`` is + replaced by ``cost_cents`` (the SDK always sends 0 — see + ``_WIRE_STRIP_FIELDS``), and ``api_key_id`` is derived + server-side from the request auth, not supplied by the SDK. + The docstring now matches the real wire contract. + """ + # 2026-07-06 (bug-fix): the previous shape called + body = _signed_request_body(request) + headers = self._build_signed_headers(body=body) + + try: + response = self._client.post( + f"{self.api_url}/api/v1/track", + content=body, + headers=headers, + timeout=5.0, + ) + except httpx.RequestError as e: + raise NullRunTransportError( + f"Network error on /track: {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="track", + ) from e + + if response.status_code == 200: + return response.json() # type: ignore[no-any-return] + + raise _parse_v3_error_envelope(response, "track") + + def cancel( + self, + execution_id: str, + reason: str | None = None, + ) -> dict[str, Any]: + """POST /api/v1/cancel — cancel an in-flight execution. + + . The server uses + ``cancel:{execution_id}`` SETNX to deduplicate repeated + cancellations: a 200 OK response is idempotent. A + non-existent ``execution_id`` returns 404 — we surface it + as ``NullRunBackendError`` because retrying with the same + id is not a valid recovery path (the execution already + terminated). + + Args: + execution_id: Server-minted id from the matching /check + response. + reason: Optional human-readable reason for the + cancellation (audit trail). + + Returns: + Parsed JSON dict from the backend's CancelResponse. + NOTE: there is NO top-level ``status`` field on the + wire — the legacy pre-v3 docstring claimed one. + v3/v4 backends emit + ``{execution_id, canceled_at, reservation_released_cents, + already_canceled}``. SDK callers branch on the HTTP + status only — do NOT read ``data["status"]`` + (KeyError on every backend >= 3.66.2). + """ + request: dict[str, Any] = {"execution_id": execution_id} + if reason: + request["reason"] = reason + + # 2026-07-06 (bug-fix): same body-before-headers reorder as + body = _signed_request_body(request) + headers = self._build_signed_headers(body=body) + + try: + response = self._client.post( + f"{self.api_url}/api/v1/cancel", + content=body, + headers=headers, + timeout=5.0, + ) + except httpx.RequestError as e: + raise NullRunTransportError( + f"Network error on /cancel: {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="cancel", + ) from e + + if response.status_code == 200: + return response.json() # type: ignore[no-any-return] + + raise _parse_v3_error_envelope(response, "cancel") + + def heartbeat( + self, + chain_id: str, + ) -> dict[str, Any]: + """POST /api/v1/heartbeat — extend a chain's idle TTL. + + . The server runs + ``EXPIRE chain:{org}:{chain_id} 300`` atomically and + deduplicates repeated heartbeats via + ``heartbeat:{chain_id}:{ts_floor_30s}`` SETNX + (TTL = 35s — the 5s tail absorbs ±5s skew per). + + Recommended cadence: every 30s of wall-clock time (the + SDK's ``ping_chain`` helper wraps this method with the + time-based scheduler). Bursting heartbeats more often than + once per 30s is wasted bandwidth — the SETNX dedups them. + + Args: + chain_id: Active chain_id. + + Returns: + Parsed JSON dict (typically ``{"status": "ok" + "chain_id":..., "last_active": ts}``). + """ + request = {"chain_id": chain_id} + # 2026-07-06 (bug-fix): same body-before-headers reorder as + # track_single above. + body = _signed_request_body(request) + headers = self._build_signed_headers(body=body) + + try: + response = self._client.post( + f"{self.api_url}/api/v1/heartbeat", + content=body, + headers=headers, + timeout=5.0, + ) + except httpx.RequestError as e: + raise NullRunTransportError( + f"Network error on /heartbeat: {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="heartbeat", + ) from e + + if response.status_code == 200: + return response.json() # type: ignore[no-any-return] + + raise _parse_v3_error_envelope(response, "heartbeat") + + def chain_end( + self, + chain_id: str, + ) -> dict[str, Any]: + """Close a chain explicitly via /api/v1/gate with chain_op=end + . + + Pre-fix this method POSTed to ``/api/v1/chain/end``. That + endpoint was never registered on the backend + (``backend/src/proxy/http/routes.rs`` has zero matches for + ``chain/end`` or ``chain_end_handler``) — the only documented + way to close a chain is to POST /api/v1/gate with + ``{"chain_id": "...", "chain_op": "end"}``. The handler is + already idempotent — a no-op 200 OK for an unknown chain_id + is the documented success path. The SDK still raises through + the envelope parser on a true non-2xx so unexpected backend + regressions surface. + + Args: + chain_id: Chain to close. + + Returns: + Parsed JSON dict (typically ``{"decision": "allow" + "chain_id":...}``). + """ + # 2026-07-04 (B3): POST /api/v1/gate with + request = { + "chain_id": chain_id, + "chain_op": "end", + # execution_id is required by the backend's gate handler + # even on chain_end — the handler reads it but does not + # mint a reservation for op=end. Use a fresh uuidv7 + # call (the server ignores it on this path). + "execution_id": uuid.uuid4().hex, + } + # 2026-07-06 (bug-fix): same body-before-headers reorder as + body = _signed_request_body(request) + headers = self._build_signed_headers(body=body) + + try: + response = self._client.post( + f"{self.api_url}/api/v1/gate", + content=body, + headers=headers, + timeout=5.0, + ) + except httpx.RequestError as e: + raise NullRunTransportError( + f"Network error on /gate (chain_end): {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="chain_end", + ) from e + + if response.status_code == 200: + return response.json() # type: ignore[no-any-return] + + raise _parse_v3_error_envelope(response, "chain_end") + + def approximate_budget( + self, + organization_id: str | None = None, + ) -> dict[str, Any]: + """GET /api/v1/budget/approximate — UI-only budget estimation. + + . NEVER for enforcement — the backend stamps + ``is_approximate: true`` on every response. The endpoint + returns 503 ``BUDGET_DATA_UNAVAILABLE`` if all three sources + (Redis period counter → Postgres cost_events → last-known + cache) fail — NEVER returns 0, because a UI that displays + "≈ $0 spent" when no data is available misleads the user. + + Used by ``nullrun.cost_dashboard `` / ``examples/cost_dashboard.py`` + and the dashboard rollup panel. + + Args: + organization_id: Optional org override; defaults to the + transport's bound org via the auth/verify result. + + Returns: + Parsed JSON dict with ``current_spend_cents_estimate`` + ``is_approximate: True``, ``source`` (BudgetSource enum + string), ``confidence`` (High/Medium/Low), and + ``last_updated_at``. + + Raises: + NullRunBackendError: 503 BUDGET_DATA_UNAVAILABLE (all + sources failed) — caller should display "Data + unavailable" + retry button, NOT "$0 spent". + NullRunAuthenticationError: 401/403. + """ + # ApproximateBudget uses GET (not POST) per the wire contract + headers = self._auth_headers_for_get() + url = f"{self.api_url}/api/v1/budget/approximate" + + try: + response = self._client.get(url, headers=headers, timeout=5.0) + except httpx.RequestError as e: + raise NullRunTransportError( + f"Network error on /budget/approximate: {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="approximate_budget", + ) from e + + if response.status_code == 200: + return response.json() # type: ignore[no-any-return] + + raise _parse_v3_error_envelope(response, "approximate_budget") + + # ==================================================================== + # ADR-009 P1 — Audit log governance surface (v0.15.0) + # ==================================================================== + # Five methods exposing the /api/v1/orgs/:org_id/audit-log/* family + # of endpoints to SDK consumers. Pre-v0.15.0 SDKs had no audit + # client — operators had to curl the wire directly. Now they can + # call ``runtime.audit.list(...)`` etc. and get typed dataclasses + # back without writing JSON parsing glue. + # + # All five methods route through the same auth + protocol + + # trace-context machinery as the other Transport methods — see + # ``_auth_headers_for_get`` below. Audit reads are GET, so no + # HMAC body signing is required. + + def audit_log( + self, + organization_id: str, + query: Any | None = None, + ) -> dict[str, Any]: + """GET /api/v1/orgs/:org_id/audit-log — read governance audit log. + + Args: + organization_id: Org UUID — required because the + /audit-log endpoint is org-scoped. The runtime + proxy passes ``self.organization_id`` automatically + so direct callers rarely need to set this. + query: Optional :class:`nullrun.audit.AuditQuery` + instance describing the filter set (event_type, + decision, policy_id, execution_id, action, actor, + since, until, limit). Pass ``None`` for "all rows" + (rarely what you want — chains grow unbounded). + + Returns: + Parsed JSON dict with ``data`` (list of + AuditEntryResponse shapes) and ``meta`` (AuditLogMeta + pagination summary). Use + :func:`nullrun.audit.AuditLogPage.from_wire` to parse + into typed dataclasses. + + Raises: + NullRunBackendError: 401/403/5xx. + NullRunAuthenticationError: 401. + """ + from nullrun.audit import AuditQuery + + q: AuditQuery = query if isinstance(query, AuditQuery) else (query or AuditQuery()) + qs = q.to_query_string() + url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log" + if qs: + url = f"{url}?{qs}" + headers = self._auth_headers_for_get() + try: + response = self._client.get(url, headers=headers, timeout=10.0) + except httpx.RequestError as e: + raise NullRunTransportError( + f"Network error on /audit-log: {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="audit_log", + ) from e + if response.status_code == 200: + return response.json() # type: ignore[no-any-return] + raise _parse_v3_error_envelope(response, "audit_log") + + def audit_verify( + self, + organization_id: str, + *, + since: str | None = None, + ) -> dict[str, Any]: + """GET /api/v1/orgs/:org_id/audit-log/verify — chain integrity. + + Walks the chain forward from `since` (or from row 1 if + omitted) and re-computes content_hash + previous_hash + continuity. Returns the same payload the audit page's + "Integrity" banner reads — use + :func:`nullrun.audit.AuditVerifyResult.from_wire` to parse. + + Args: + organization_id: Org UUID — required. + since: Optional RFC3339 lower bound. With `since`, + only rows since that timestamp are walked (plus a + prior anchor row for hash continuity). Without + `since`, the full chain from row 1 is re-verified. + + Returns: + Parsed JSON dict with `verified`, `chain_valid`, + `record_count`, `first_hash`, `last_hash`, + `first_failure_reason`, `timestamp`, `hmac_checked`. + + Raises: + NullRunBackendError / NullRunAuthenticationError. + """ + params: list[tuple[str, str]] = [] + if since: + params.append(("since", since)) + qs = "&".join(f"{k}={v}" for k, v in params) + url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log/verify" + if qs: + url = f"{url}?{qs}" + headers = self._auth_headers_for_get() + try: + response = self._client.get(url, headers=headers, timeout=30.0) + except httpx.RequestError as e: + raise NullRunTransportError( + f"Network error on /audit-log/verify: {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="audit_verify", + ) from e + if response.status_code == 200: + return response.json() # type: ignore[no-any-return] + raise _parse_v3_error_envelope(response, "audit_verify") + + def audit_list_exports( + self, + organization_id: str, + ) -> list[dict[str, Any]]: + """GET /api/v1/orgs/:org_id/audit-log/export — list recent export jobs. + + Returns the raw JSON list of recent export job summaries + (last 10). Use :func:`nullrun.audit.AuditExportJob.from_wire` + to parse each entry. + + Raises: + NullRunBackendError / NullRunAuthenticationError. + """ + url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log/export" + headers = self._auth_headers_for_get() + try: + response = self._client.get(url, headers=headers, timeout=10.0) + except httpx.RequestError as e: + raise NullRunTransportError( + f"Network error on /audit-log/export (list): {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="audit_list_exports", + ) from e + if response.status_code == 200: + body = response.json() + # Wire shape is `{"exports": [...]}` per the audit export + # list handler in backend/src/proxy/http/audit.rs. + if isinstance(body, dict): + return body.get("exports", []) or [] + return body if isinstance(body, list) else [] + raise _parse_v3_error_envelope(response, "audit_list_exports") + + def audit_create_export( + self, + organization_id: str, + ) -> dict[str, Any]: + """POST /api/v1/orgs/:org_id/audit-log/export — enqueue 30-day export. + + The backend creates a job, returns ``{"job_id", "status": + "pending"}`` immediately, and processes in the background. + Poll :meth:`audit_export_status` for completion. + + The export covers the trailing 30 days; the backend hard-codes + that window today (audit.rs:692-700 — ``chrono::Utc::now() - + Duration::days(30)``). When the per-job window becomes + configurable this method will accept a `since`/`until` + override. + + Returns: + Parsed JSON dict with ``job_id`` (UUID) and ``status``. + + Raises: + NullRunBackendError / NullRunAuthenticationError. + """ + url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log/export" + headers = self._build_signed_headers(body=b"{}") + try: + response = self._client.post(url, content=b"{}", headers=headers, timeout=10.0) + except httpx.RequestError as e: + raise NullRunTransportError( + f"Network error on /audit-log/export (create): {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="audit_create_export", + ) from e + if response.status_code == 200: + return response.json() # type: ignore[no-any-return] + raise _parse_v3_error_envelope(response, "audit_create_export") + + def audit_export_status( + self, + organization_id: str, + job_id: str, + ) -> dict[str, Any]: + """GET /api/v1/orgs/:org_id/audit-log/export/:job_id/status. + + Polls a previously-enqueued export job. When ``status`` flips + to ``completed`` the ``file_url`` field carries an S3 + presigned URL (or `/tmp/...` path on dev), and an + ``error_message`` is set on the ``failed`` transition. + + Args: + organization_id: Org UUID — required. + job_id: UUID returned by :meth:`audit_create_export`. + + Returns: + Parsed JSON dict with ``job_id``, ``status``, + ``file_url``, ``record_count``, ``created_at``, + ``completed_at``, ``error_message``. + + Raises: + NullRunBackendError / NullRunAuthenticationError. + """ + url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log/export/{job_id}/status" + headers = self._auth_headers_for_get() + try: + response = self._client.get(url, headers=headers, timeout=10.0) + except httpx.RequestError as e: + raise NullRunTransportError( + f"Network error on /audit-log/export/{job_id}/status: {e}", + source=TransportErrorSource.NETWORK_ERROR, + endpoint="audit_export_status", + ) from e + if response.status_code == 200: + return response.json() # type: ignore[no-any-return] + raise _parse_v3_error_envelope(response, "audit_export_status") + + def _auth_headers_for_get(self) -> dict[str, str]: + """Headers for an unsigned GET (no HMAC body). + + Same shape as ``_build_signed_headers`` minus the HMAC + headers. Used by ``approximate_budget`` which is a GET with + no body, so there's nothing to sign. Keeps the protocol + + CSRF-bypass + trace-context headers consistent with the + signed-POST path. + """ + headers: dict[str, str] = {"Content-Type": "application/json"} + if self.api_key: + headers["X-API-Key"] = self.api_key + headers["Authorization"] = f"Bearer {self.api_key}" + headers[HEADER_PROTOCOL] = _protocol_header_value() + self._inject_trace_context(headers) + return headers + + +# 2026-07-02 (v0.11.0): ACTIVE v3 error envelope parser. +def _extract_error_envelope( + body: Any, + raw_text: str, +) -> tuple[str, str, dict[str, Any]]: + """Pull ``(error_code, message, details)`` from any error envelope. + + Drift §3 (2026-07-06): the backend emits three distinct shapes + for non-2xx responses. This helper normalises them into the + ``(error_code, message, details)`` tuple the rest of + ``_parse_v3_error_envelope`` consumes. + + Lookup priority: + + 1. **v3 envelope** -- ``{"error_code": "BUDGET_HARD_BLOCKED", + "error_message": "...", "details": {...}, ...}``. The + canonical shape from ``gate/internal.rs`` and + ``handlers.rs::track_handler``. + + 2. **v3 mixed** -- ``{"error_code": "BUDGET_DATA_UNAVAILABLE", + "message": "...", "retry_after_ms": N}``. The 503 path + from ``budget.rs:107-112``; same v3 semantics but the + message field is called ``message`` not ``error_message``. + + 3. **Legacy slug** -- ``{"error": "chain_not_extendable", + "message": "...", "chain_state": "..."}``. From + ``heartbeat.rs:199-205`` and the ``ApiError`` path on + ``cancel.rs``. The slug is lowercased and SCREAMING_SNAKE'd + so it matches ``_V3_ERROR_CODE_MAP`` lookups. + + 4. **Plaintext** -- ``response.text`` containing a free-form + error string (heartbeat.rs:157, heartbeat.rs:166). No JSON, + so ``body`` is empty. + + Args: + body: Parsed JSON body from the response (``{}`` on parse + failure or non-JSON content). + raw_text: Raw ``response.text`` fallback for plaintext + envelopes. + + Returns: + ``(backend_code, message, details)`` where: + + * ``backend_code`` is uppercase SCREAMING_SNAKE if it + originated from the v3 envelope, or the lowercased slug + otherwise. The mapping table keys are uppercase; the + dispatcher lowercases the lookup key before consulting + the map. + * ``message`` is the human-readable string for the + exception class. Falls back to ``raw_text`` if no JSON + body. + * ``details`` is the machine-readable context payload + (``details: {...}`` on the v3 envelope, all other + JSON fields flattened on the legacy slug, ``{}`` on + plaintext). + """ + if not isinstance(body, dict) or not body: + # No JSON body -- plaintext error envelope. + # Heartbeat's 404 "chain not found" and 403 + # "chain org mismatch" land here. + return ("", raw_text or "", {}) + + # Shape 1: v3 envelope. + if "error_code" in body: + code = str(body.get("error_code", "") or "") + # The 503 budget path uses "message" instead of + # "error_message". Accept both. + message = str(body.get("error_message") or body.get("message") or raw_text or "") + details_raw = body.get("details") or {} + if not isinstance(details_raw, dict): + details_raw = {} + # Forward any extra top-level fields that look like + # context (e.g. ``chain_state`` on heartbeat 409) into + # details so downstream code can introspect them. + details: dict[str, Any] = dict(details_raw) + for key, value in body.items(): + if key in ( + "error_code", + "error_message", + "message", + "details", + "retry_after_ms", + ): + continue + details.setdefault(key, value) + return (code, message, details) + + # Shape 2: legacy slug. ``error`` is the slug, + # ``message`` is the human-readable string. + if "error" in body: + slug = str(body.get("error", "") or "") + message = str(body.get("message", "") or raw_text or "") + # Convert the legacy lowercase slug to uppercase + # SCREAMING_SNAKE so the mapping table can find it. + code = slug.upper() + # Everything except ``error`` and ``message`` goes into + # details for diagnostic context. + details = { + k: v for k, v in body.items() if k not in ("error", "message") and not k.startswith("_") + } + return (code, message, details) + + # JSON body but not a recognised envelope shape. Pass through. + return ("", raw_text or str(body), dict(body) if isinstance(body, dict) else {}) + + +def _safe_json(response: httpx.Response, endpoint: str) -> Any: + """Parse a response body as JSON, wrapping parse failures. + + DEF-ERRHDL-INVALID-JSON-01 (2026-08-11, RUN_ID 20260811-1): the SDK + previously propagated ``json.JSONDecodeError`` unchanged to user + code, which leaks internal file paths and the raw broken payload + fragment in tracebacks. This helper wraps the parse failure in + NullRunTransportError with a stable ``error_code`` so callers can + ``except`` cleanly and the user sees a short NullRun-family + message instead of a Python traceback. + + ``body_preview`` is intentionally truncated to 200 chars and the + raw ``JSONDecodeError.lineno/colno`` are NOT included in the + surfaced message -- both are info-leak surface (line numbers + hint at response shape; partial body may carry PII like + organization_id fragments). + """ + try: + return response.json() + except (json.JSONDecodeError, ValueError) as exc: + # Body preview capped at 200 chars; truncated to avoid + # flooding logs / exception chain. + try: + body_preview = (response.text or "")[:200] + except Exception: + body_preview = "" + raise NullRunTransportError( + f"Received malformed JSON from {endpoint} " + f"(status={response.status_code}): {type(exc).__name__}", + source=TransportErrorSource.GATEWAY_ERROR, + endpoint=endpoint, + error_code="NR-T001", + ) from exc + + +def _parse_v3_error_envelope( + response: httpx.Response, + endpoint: str, +) -> Exception: + """Translate a non-2xx ``httpx.Response`` into the right v3 + SDK exception. + + The backend returns errors as a JSON envelope of the shape + ``{"error_code": "BUDGET_HARD_BLOCKED", "error_message": "..." + "details": {...}, "retry_after_ms": N}``. The + parser maps the backend's ``error_code`` string to the closest + SDK exception class, attaching the structured envelope fields + as instance attributes so callers can introspect them. + + Mapping table lives at ``_V3_ERROR_CODE_MAP`` below — keep the + helper as a thin dispatcher. + """ + # Lazy imports: the exception classes import the transport + # types (TransportErrorSource), so a top-level import here + # would create a cycle. The price is one extra import + # non-2xx response — irrelevant for the failure path. + from nullrun.breaker.exceptions import ( + NullRunApprovalDeniedError, + NullRunApprovalDigestMismatchError, + NullRunApprovalExpiredError, + NullRunApprovalNotYetApprovedError, + NullRunApprovalReplayRejectedError, + NullRunApprovalToolDigestMismatchError, + NullRunAuthError, + NullRunBackendError, + NullRunBlockedException, + NullRunBudgetError, + NullRunBudgetRecheckFailedError, + NullRunChainError, + NullRunConsumeOverbudgetError, + NullRunDecision, + NullRunInfrastructureError, + NullRunProtocolError, + NullRunRateLimitRedisError, + NullRunToolBlockedError, + NullRunWorkflowInactiveError, + RateLimitError, + ) + + status = response.status_code + try: + body = response.json() + except Exception: + body = None + if not isinstance(body, dict): + body = {} + + # Drift §3 (2026-07-06): the wire envelope is NOT one shape. + backend_code, message, details = _extract_error_envelope(body, response.text) + retry_after_ms: float | None = body.get("retry_after_ms") if isinstance(body, dict) else None + # Retry-After header takes precedence over the JSON field when + # both are present (server-side convention — header is canonical + # per RFC 7231, JSON is a NullRun-specific fallback). + retry_after_header = response.headers.get("Retry-After") + if retry_after_header: + try: + retry_after_ms = float(retry_after_header) * 1000.0 + except ValueError: + # HTTP-date form is non-numeric — leave JSON value intact. + pass + + # Per-class dispatcher. Each exception has its own constructor + # signature (RateLimitError requires source+endpoint + # NullRunBackendError requires endpoint+status_code, etc.) so a + # uniform ``error_cls(**kwargs)`` does not work. The switches + # below mirror the exact field mapping from. + full_message = f"{endpoint}: {message}" + + if backend_code == "PROTOCOL_TOO_OLD" or backend_code == "PROTOCOL_TOO_NEW": + # NullRunProtocolError → NullRunInfrastructureError → + # NullRunError base. Base constructor does NOT accept + # a generic ``details=`` kwarg. Pass message only — the + # catalog value already encodes error_code + retryable. + return NullRunProtocolError(full_message) + + if backend_code == "CONSUME_OVERBUDGET": + return NullRunConsumeOverbudgetError( + full_message, + execution_id=details.get("execution_id"), + reserved_cents=details.get("reserved_cents"), + max_allowed_cents=details.get("max_allowed_cents"), + actual_cost_cents=details.get("actual_cost_cents"), + epsilon_cents=details.get("epsilon_cents"), + status_code=status, # 422 per backend mapping + ) + + if ( + backend_code == "CHAIN_MAX_DURATION_EXCEEDED" + or backend_code == "CHAIN_CROSS_ORG" + or backend_code == "CHAIN_ORG_MISMATCH" + ): + return NullRunChainError( + full_message, + chain_id=details.get("chain_id"), + backend_code=backend_code, + details=details, + status_code=status, # 402/403 per backend mapping + ) + + if backend_code == "WORKFLOW_INACTIVE": + return NullRunWorkflowInactiveError( + full_message, + workflow_id=details.get("workflow_id"), + status_code=status, # 403 per backend mapping + ) + + if backend_code == "BUDGET_RECHECK_FAILED": + # H6 / 2026-08-12 audit: dedicated typed dispatch so callers + # can branch on the post-approval recheck failure (NR-B006) + # vs a fresh /gate block (NR-B004). The dispatcher surfaces + # ``current_spend_cents`` / ``budget_cents`` from the wire + # envelope so callers can compute the remaining cap and + # decide whether to retry after re-/gate. + return NullRunBudgetRecheckFailedError( + full_message, + current_spend_cents=details.get("current_spend_cents"), + budget_cents=details.get("budget_cents"), + status_code=status, # 402 per backend mapping + ) + + if backend_code in ( + "APPROVAL_NOT_YET_APPROVED", + "APPROVAL_DENIED", + "APPROVAL_EXPIRED", + "APPROVAL_DIGEST_MISMATCH", + "APPROVAL_TOOL_DIGEST_MISMATCH", + "APPROVAL_REPLAY_REJECTED", + ): + # v3.53 / 2026-08-13 audit, A-1+A-2 bundle: dedicated typed + # dispatch so callers can branch on the precise grant-consume + # outcome. Pre-v3.53 the SDK fell through to the catalog + # fallback path which called ``catalog(full_message, **details)`` + # — NullRunBlockedException subclasses reject that signature + # (they need workflow_id as positional arg) so the catch-all + # path raised TypeError instead of the typed exception. + # Post-v3.53 each of the six codes maps to its own NR-Axxx + # subclass (NR-A010..NR-A015). Wire details carry the + # approval_id and the typed exception's NR-Axxx catalog + # value (via the class attribute) so cookbook recipes can + # ``except NullRunApprovalDeniedError:`` for terminal + # surface-to-user, ``except + # NullRunApprovalNotYetApprovedError:`` for wait/poll, + # ``except NullRunApprovalReplayRejectedError:`` for + # retry-loop detection, etc. + catalog = _V3_ERROR_CODE_MAP[backend_code] + return catalog( # type: ignore[call-arg] + workflow_id=str(details.get("workflow_id") or "unknown"), + reason=full_message, + status_code=status, # 403 per backend mapping + approval_id=details.get("approval_id"), + ) + + if backend_code == "RATE_LIMIT_REDIS_UNAVAILABLE": + # NullRunRateLimitRedisError → NullRunInfrastructureError + # → NullRunError base. Base constructor accepts only + # message + (error_code, user_action, retryable, docs_url + # cause) — NOT a generic ``details=``. The catalog value + # already encodes error_code + retryable, so we just pass + # the message. + return NullRunRateLimitRedisError(full_message) + + if backend_code == "RATE_LIMIT_EXCEEDED": + retry_after = retry_after_ms / 1000.0 if retry_after_ms else None + return RateLimitError( + full_message, + source=TransportErrorSource.GATEWAY_ERROR, + endpoint=endpoint, + retry_after=retry_after, + body=body, + ) + + # Catalog codes that map to NullRunBudgetError / NullRunBackendError + # via the fallback shape (no special signature). + catalog = _V3_ERROR_CODE_MAP.get(backend_code) + if catalog is not None: + # Special-case each constructor signature — the NullRun + # hierarchy has heterogeneous constructors (workflow_id + + # reason for NullRunBlockedException, endpoint + status_code + # for NullRunBackendError, error_code/user_action for + # NullRunError base). Universal ``catalog(message, details=)`` + # would trip one of them every time. + if catalog is NullRunBackendError: + return NullRunBackendError( + full_message, + endpoint=endpoint, + status_code=status, + ) + if catalog is NullRunExecutionNotFoundError: + # 2026-09-09 audit: dedicated dispatch so callers can + # read ``execution_id`` / ``endpoint`` / ``regate_required`` + # off the exception without indexing into ``details``. + # Mirrors the ``NullRunBackendError`` branch above (the + # parent class) but also forwards ``execution_id`` from + # the wire envelope. Without this branch the generic + # catalog fallback at line ~2615 would discard the + # ``execution_id`` field (it filters ``**details`` to + # the base NullRunError kwargs only). + return NullRunExecutionNotFoundError( + full_message, + execution_id=details.get("execution_id"), + endpoint=details.get("endpoint") or endpoint, + status_code=status, # 404 per backend mapping + ) + if catalog is NullRunBudgetError: + # NullRunBudgetError → NullRunBlockedException → requires + return NullRunBudgetError( + workflow_id=str(details.get("workflow_id") or "unknown"), + reason=full_message, + status_code=status, + ) + if catalog is NullRunRateLimitRedisError: + # NullRunError base takes (message, error_code=, user_action= + # retryable=, docs_url=, cause=). The catalog value here + # already encodes error_code + retryable, so we pass + # the message only. + return catalog(full_message) + if catalog is NullRunProtocolError: + return catalog(full_message) + # NullRunAuthError — surface the wire error_code (one of + # v3.38's API_KEY_REVOKED / API_KEY_EXPIRED / API_KEY_DISABLED + # / API_KEY_INVALID / API_KEY_MISSING / API_KEY_MALFORMED) on + # ``self.wire_code`` so callers can branch on granular + # lifecycle state without clobbering the SDK-side + # ``error_code`` taxonomy (NR-A003). Mirrors the + # ``NullRunChainError.backend_code`` pattern. + # + # Filter ``details`` to the kwargs the base NullRunError + # constructor accepts — the envelope's ``details`` dict can + # carry arbitrary keys (``expires_at``, ``ttl_seconds``, ...) + # and the base class rejects unknown kwargs with TypeError. + # Unknown fields are stored on ``self.details`` for caller + # introspection instead. + if catalog is NullRunAuthError: + allowed = {"error_code", "user_action", "retryable", "docs_url", "cause"} + forwarded = {k: v for k, v in details.items() if k in allowed} + extra = {k: v for k, v in details.items() if k not in allowed} + instance = NullRunAuthError( + full_message, + wire_code=backend_code, + **forwarded, + ) + if extra: + instance.details = extra # type: ignore[attr-defined] + return cast(Exception, instance) + # Final fallback for catalog classes with a generic + # (message, **details) signature (NullRunWorkflowInactiveError + # and any future addition). + # The details payload is forwarded as a positional kwarg + # via **details (typed as Any to satisfy mypy since + # type[BaseException] does not expose the kwargs the + # catalog subclasses actually accept). + # + # The catalog lookup produces type[BaseException] (the + # union of all class objects), but every entry in + # _V3_ERROR_CODE_MAP is a real Exception subclass. Cast + # to Exception so mypy stops flagging the return value + # as BaseException (the helper declares -> Exception). + allowed = {"error_code", "user_action", "retryable", "docs_url", "cause"} + forwarded = {k: v for k, v in details.items() if k in allowed} + if ( + catalog is NullRunToolBlockedError + or catalog is NullRunBlockedException + ): + # DEF-NR-TOOLBLOCKED-PARSER (2026-09-10): NullRunBlockedException + # subclasses require positional ``workflow_id`` + ``reason`` + # (no defaults), so the generic ``catalog(full_message, ...)`` + # fallback below raises TypeError when given a string for + # ``workflow_id``. Affects 7 catalog entries: TOOL_BLOCKED, + # LOOP_DETECTED, MODEL_REQUIRED, POLICY_UNCONFIGURED, + # TOO_MANY_PENDING_APPROVALS, BUSINESS_IMPACT_INVALID, + # VALIDATION_FAILED. Pre-fix the TypeError escaped the parser + # and got swallowed by the catch-all ``except Exception: pass`` + # in Transport.execute, surfacing the synthetic-block dict + # ``{"decision": "block", "explanation": "Gateway returned + # 403"}`` instead of the typed NR-T001 / NR-Lxxx catalog line. + # ``tool_name`` is forwarded for NullRunToolBlockedError + # (the only BlockedException subclass that surfaces it on the + # wire envelope); the parent constructor drops it for plain + # NullRunBlockedException so it's a no-op there. ``forwarded`` + # (error_code / user_action / retryable / docs_url / cause) is + # passed through so the catalog value's defaults win. + instance = catalog( # type: ignore[call-arg] + workflow_id=str(details.get("workflow_id") or "unknown"), + reason=full_message, + status_code=status, + tool_name=details.get("tool_name"), + **forwarded, + ) + return cast(Exception, instance) + instance = catalog(full_message, **forwarded) # type: ignore[call-arg] + return cast(Exception, instance) + + # Fallback — use HTTP status. The catalog may not yet cover + # every backend code, so we surface a typed backend error + # that exposes status_code + error_code for the caller. + if status in (401, 403): + return NullRunAuthenticationError( + f"Auth failed on {endpoint} (status {status}, error_code={backend_code!r}): {message}" + ) + if status == 429: + retry_after = retry_after_ms / 1000.0 if retry_after_ms else None + return RateLimitError( + f"Rate limited on {endpoint} (status 429, error_code={backend_code!r}): {message}", + source=TransportErrorSource.GATEWAY_ERROR, + endpoint=endpoint, + retry_after=retry_after, + body=body, + ) + if 500 <= status < 600: + return NullRunBackendError( + f"{endpoint}: {message} (status {status}, error_code={backend_code!r})", + endpoint=endpoint, + status_code=status, + ) + return NullRunBackendError( + f"{endpoint}: {message} (status {status}, error_code={backend_code!r})", + endpoint=endpoint, + status_code=status, + ) + + +# Lazy import to avoid a hard dependency at module import time. +# `_parse_v3_error_envelope` is a module-level helper; the exception +# classes live in `nullrun.breaker.exceptions`. Importing here +# (rather than at the top of transport.py) keeps the legacy import +# graph identical and avoids breaking the frozen +# ``_parse_error_envelope`` test contract. +def _build_v3_error_code_map() -> dict[str, type[Exception]]: + """Construct the v3 error_code → exception class mapping. + + Imported lazily because the exception classes import the + transport types (TransportErrorSource), which would create a + circular import if loaded eagerly at the top of transport.py. + """ + from nullrun.breaker.exceptions import ( + NullRunApprovalDeniedError, + NullRunApprovalDigestMismatchError, + NullRunApprovalExpiredError, + NullRunApprovalNotYetApprovedError, + NullRunApprovalReplayRejectedError, + NullRunApprovalToolDigestMismatchError, + NullRunAuthError, + NullRunBackendError, + NullRunBlockedException, + NullRunBudgetError, + NullRunBudgetRecheckFailedError, + NullRunChainError, + NullRunConsumeOverbudgetError, + NullRunExecutionNotFoundError, + NullRunProtocolError, + NullRunRateLimitRedisError, + NullRunToolBlockedError, + NullRunWorkflowInactiveError, + RateLimitError, + ) + + return { + # 400 — protocol mismatch + "PROTOCOL_TOO_OLD": NullRunProtocolError, + "PROTOCOL_TOO_NEW": NullRunProtocolError, + # 402 — budget family + "BUDGET_HARD_BLOCKED": NullRunBudgetError, + "BUDGET_SOFT_BLOCKED": NullRunBudgetError, + "BUDGET_OVERDRAFT_EXCEEDED": NullRunBudgetError, + "BUDGET_PERIOD_NOT_STARTED": NullRunBudgetError, + "REDIS_UNAVAILABLE": NullRunBudgetError, + # 402 — chain family (separate class for diagnostic clarity) + "CHAIN_MAX_DURATION_EXCEEDED": NullRunChainError, + # 403 — chain security + workflow state + "CHAIN_CROSS_ORG": NullRunChainError, + "CHAIN_ORG_MISMATCH": NullRunChainError, + # 403 — Execution Graph v0 (2026-08-06, backend). Sub-agent + "PARENT_EXECUTION_NOT_FOUND": NullRunChainError, + "PARENT_EXECUTION_ORG_MISMATCH": NullRunChainError, + "PARENT_EXECUTION_KEY_MISMATCH": NullRunChainError, + "WORKFLOW_INACTIVE": NullRunWorkflowInactiveError, + # 401/403 — auth (v3.38 distinct lifecycle states). + # The backend splits the v3.36 ``API_KEY_REVOKED`` bucket into + # five distinct wire codes so SDKs can branch on each state + # (e.g. surface "rotate this key" vs "this key was admin- + # disabled" vs "no Authorization header was sent"). All map + # to NullRunAuthError — diagnostic class is preserved; the + # granular codes live in ``details.error_code`` and are + # surfaced via NullRunAuthError.code for handler dispatch. + "API_KEY_REVOKED": NullRunAuthError, + "API_KEY_EXPIRED": NullRunAuthError, + "API_KEY_DISABLED": NullRunAuthError, + "API_KEY_INVALID": NullRunAuthError, + "API_KEY_MISSING": NullRunAuthError, + "API_KEY_MALFORMED": NullRunAuthError, + # 422 — consume invariant violation + "CONSUME_OVERBUDGET": NullRunConsumeOverbudgetError, + # 429 — rate limit + "RATE_LIMIT_EXCEEDED": RateLimitError, + # 503 — backend availability + "RATE_LIMIT_REDIS_UNAVAILABLE": NullRunRateLimitRedisError, + "BUDGET_DATA_UNAVAILABLE": NullRunBackendError, + # 402 — approval-create failure family (DEF-ARFLOW-TOOLNAME-01, + "APPROVAL_DB_UNAVAILABLE": NullRunBlockedException, + "APPROVAL_PERSISTENCE_FAILED": NullRunBlockedException, + "APPROVAL_VALIDATION_FAILED": NullRunBlockedException, + "APPROVAL_CONFLICT": NullRunBlockedException, + "APPROVAL_NOT_FOUND": NullRunBlockedException, + "APPROVAL_CREATE_FAILED": NullRunBlockedException, + # 403 — approval grant-consume outcomes (v3.53 / 2026-08-13 + # audit, A-1+A-2 bundle). Distinct from the /gate + # create-failure family above: these are the seven + # distinct outcomes that the backend's + # `gate_internal()` returns on /execute post-approval + # grant-consume (see + # `backend/src/proxy/http/gate/internal.rs:3059-3108, + # 3115-3138`). Pre-v3.53 the SDK collapsed all six + # into NullRunBlockedException — bilateral wire gap. + # Post-v3.53 each maps to a typed exception + # (NR-A010..NR-A015) so cookbook recipes can branch + # on the precise outcome (e.g. ``except + # NullRunApprovalNotYetApprovedError:`` for wait/poll, + # ``except NullRunApprovalDeniedError:`` for terminal + # surface-to-user, ``except + # NullRunApprovalReplayRejectedError:`` for retry-loop + # detection). + "APPROVAL_NOT_YET_APPROVED": NullRunApprovalNotYetApprovedError, + "APPROVAL_DENIED": NullRunApprovalDeniedError, + "APPROVAL_EXPIRED": NullRunApprovalExpiredError, + "APPROVAL_DIGEST_MISMATCH": NullRunApprovalDigestMismatchError, + "APPROVAL_TOOL_DIGEST_MISMATCH": NullRunApprovalToolDigestMismatchError, + "APPROVAL_REPLAY_REJECTED": NullRunApprovalReplayRejectedError, + # 402 — post-approval budget recheck (H6 / 2026-08-12 audit). + # Distinct from BUDGET_HARD_BLOCKED: the operator explicitly + # approved the grant at /gate, but the period-bound counter + # moved between /gate and /execute (another concurrent + # execution spent the budget). Caller should re-/gate to + # refresh the reservation envelope and retry /execute. + # Backed by GateErrorCode::BudgetRecheckFailed in the + # backend (error_codes.rs). + # 2026-09-09 audit: the per-class dispatcher in + # ``_v3_error_dispatch`` (line ~2477) already routes this to + # ``NullRunBudgetRecheckFailedError`` (NR-B006) before the + # catalog fallback — defense-in-depth, this catalog entry + # now matches the dispatcher. + "BUDGET_RECHECK_FAILED": NullRunBudgetRecheckFailedError, + # NR-007 (audit 2026-08-24): the 19 entries below were missing + # from the SDK map and caused cookbook recipes that branch on + # ``error_code`` to fall through to ``NullRunBackendError``. + # Added in the parity PR that closes NR-007 — keep this + # block grouped so the parity CI test + # ``backend/tests/nr007_sdk_error_code_parity.rs`` has a + # single regression pin surface. Family mapping rationale + # per code: + # - budget family: NullRunBudgetError + # - chain family: NullRunChainError + # - auth binding: NullRunAuthError + # - protocol / wire validation: NullRunProtocolError / + # NullRunBackendError + # - gate decision: NullRunBlockedException / + # NullRunToolBlockedError (TOOL_BLOCKED MUST use the + # dedicated class per CLAUDE.md §8 — operators expect + # ``except NullRunToolBlockedError:`` for tool-name + # branch recipes). + "BUDGET_ANTI_DOS_RESERVED_CAP": NullRunBudgetError, + "BUDGET_REDIS_UNAVAILABLE": NullRunBudgetError, + "CHAIN_ID_INVALID": NullRunChainError, + "EXECUTION_KEY_MISMATCH": NullRunAuthError, + "EXECUTION_ORG_MISMATCH": NullRunAuthError, + "ORG_MISMATCH": NullRunAuthError, + "PROTOCOL_HEADER_INVALID": NullRunProtocolError, + "PROTOCOL_HEADER_REQUIRED": NullRunProtocolError, + "TOOL_BLOCKED": NullRunToolBlockedError, + "LOOP_DETECTED": NullRunBlockedException, + "MODEL_REQUIRED": NullRunBlockedException, + "POLICY_UNCONFIGURED": NullRunBlockedException, + "TOO_MANY_PENDING_APPROVALS": NullRunBlockedException, + "BUSINESS_IMPACT_INVALID": NullRunBlockedException, + "VALIDATION_FAILED": NullRunBlockedException, + # Wire-level parsing failures (missing / malformed fields). + # Map to ``NullRunBackendError`` because the SDK treats them + # as infrastructure-side issues — the server should have + # returned a structured 4xx envelope, and a fall-through + # here indicates a wire-shape drift between client and server. + "EXECUTION_ID_MALFORMED": NullRunBackendError, + "EXECUTION_ID_REQUIRED": NullRunBackendError, + # 2026-09-09 SDK-drift audit: ``INVALID_EXECUTION_ID`` is + # emitted by the backend as a typed envelope at + # ``cancel.rs:142-149`` and ``orchestrator.rs:1327-1334`` — + # round-trips through the canonical ``v3_error_envelope`` + # helper, so the wire string is canonical. Map to + # ``NullRunBackendError`` (sibling to the EXECUTION_ID_* + # siblings above) — wire-shape drift guard. + "INVALID_EXECUTION_ID": NullRunBackendError, + # 2026-09-09 SDK-drift audit: ``EXECUTION_NOT_FOUND`` is + # emitted by the backend as a typed envelope at + # ``execute.rs:194`` and ``cancel.rs:303`` (post-DEF-SDKK-022 + # routing through ``v3_error_envelope`` + the new + # ``GateErrorCode::ExecutionNotFound`` variant). Map to the + # dedicated ``NullRunExecutionNotFoundError`` (NR-EX01) so + # cookbook code can ``except + # NullRunExecutionNotFoundError`` to distinguish a missed + # /gate (re-issue /gate then retry /execute) from generic + # wire-shape drift. + "EXECUTION_NOT_FOUND": NullRunExecutionNotFoundError, + # Rate-limit plan lookup failure (Postgres / Redis adjacent). + # Tied to ``NullRunRateLimitRedisError`` because the failure + # mode is rate-limit-specific infrastructure unavailability + # rather than generic backend error. + "RATE_LIMIT_PLAN_LOOKUP_FAILED": NullRunRateLimitRedisError, + # Idempotency layer Redis unavailability. Map to generic + # ``NullRunBackendError`` — the wire class is infrastructure + # availability, not a typed subclass (mirrors + # ``RATE_LIMIT_REDIS_UNAVAILABLE`` -> ``NullRunRateLimitRedisError`` + # family pattern at wire level). + "IDEMPOTENCY_REDIS_UNAVAILABLE": NullRunBackendError, + # Execution Graph / ADR-036 (sub-agent spawn topology). Backend + # error_codes.rs:107-382 covers six codes in this family — three + # 422 semantic rejects (cycle / depth / parent-binding) and three + # 503 infrastructure failures (depth lookup / invoke persist / + # subworkflow disabled). Map to ``NullRunChainError`` because + # the existing class already carries `parent_execution_id` per + # Execution Graph v0 docstring at `exceptions.py:388-410`. Adding + # them under a fresh ``NullRunSubworkflowError`` would force + # cookbook code to import a new exception class for the same + # lineage concept; consolidate under ChainError instead. + "WORKFLOW_CYCLE_DETECTED": NullRunChainError, + "WORKFLOW_DEPTH_EXCEEDED": NullRunChainError, + "WORKFLOW_PARENT_BINDING_EXPIRED": NullRunChainError, + "WORKFLOW_DEPTH_LOOKUP_FAILED": NullRunChainError, + "INVOKE_PERSIST_FAILED": NullRunBackendError, + "SUBWORKFLOW_INVOKE_DISABLED": NullRunChainError, + # ADR-023 (post-approval re-check race): a second operator + # already decided on the same approval row before this call's + # re-check landed. Map to ``NullRunApprovalReplayRejectedError`` + # because semantically the agent caller has the same retry-loop + # concern as a replay-rejected approval (CLAUDE.md §34c). + "APPROVAL_ALREADY_DECIDED": NullRunApprovalReplayRejectedError, + # ADR-023 (Phase-1+ wire-shape fail-CLOSED): a v3+ SDK hit /gate + # without ``action_digest`` (legacy anchor attempt). Map to + # ``NullRunBlockedException`` because the wire shape is a true + # block decision, not an infrastructure error — cookbook code + # branches on the action_digest missing path with the same + # `except NullRunBlockedException:` flow as TOOL_BLOCKED. + "LEGACY_GRANT_REJECTED": NullRunBlockedException, + } + + +_V3_ERROR_CODE_MAP: dict[str, type[Exception]] = _build_v3_error_code_map() + + +# ADR (2026-06-28, audit P2.2 close): ``_parse_error_envelope`` below +def _parse_error_envelope( + response: httpx.Response, + endpoint: str, +) -> Exception: + """Translate a non-2xx ``httpx.Response`` into the right exception + subclass per the canonical ``contracts/errors.ts`` envelope. + + 4xx/5xx/429 are mapped to distinct ``RateLimitError`` / + ``NullRunAuthenticationError`` / ``NullRunTransportError(GATEWAY_ERROR)`` + so callers branch on type instead of string-matching ``str(exc)``. + + Module-level helper (not a Transport method) so it can be called + from background threads that do not carry a Transport instance. + + **Audit F-R2-13 (2026-06-22):** no live wire path uses this. It + exists for tests only. See the comment block above. + """ + status = response.status_code + try: + body = response.json() + except Exception: + body = None + if not isinstance(body, dict): + body = {} + error_slug: str = body.get("error", "") or "" + message: str = body.get("message") or response.text or f"HTTP {status}" + + if status in (401, 403): + return NullRunAuthenticationError( + f"Auth failed on {endpoint} (status {status}, error={error_slug!r}): {message}" + ) + + if status == 429: + retry_after: float | None = None + ra_header = response.headers.get("Retry-After") + if ra_header: + try: + retry_after = float(ra_header) + except ValueError: + try: + from datetime import datetime, timezone + from email.utils import parsedate_to_datetime + + dt = parsedate_to_datetime(ra_header) + retry_after = (dt - datetime.now(timezone.utc)).total_seconds() + except Exception: + retry_after = None + upgrade_url = body.get("upgrade_url") if isinstance(body, dict) else None + return RateLimitError( + f"Rate limited on {endpoint} (status 429, error={error_slug!r}): {message}", + source=TransportErrorSource.GATEWAY_ERROR, + endpoint=endpoint, + retry_after=retry_after, + upgrade_url=upgrade_url, + body=body, + ) + + if 500 <= status < 600: + return NullRunTransportError( + f"Gateway error on {endpoint} (status {status}, error={error_slug!r}): {message}", + source=TransportErrorSource.GATEWAY_ERROR, + endpoint=endpoint, + status_code=status, + error_slug=error_slug, + ) + + return NullRunTransportError( + f"Client error on {endpoint} (status {status}, error={error_slug!r}): {message}", + source=TransportErrorSource.GATEWAY_ERROR, + endpoint=endpoint, + status_code=status, + error_slug=error_slug, + ) + + +# Public surface for `from nullrun.transport import X` consumers +# (notably runtime.py). Without this list, mypy treats every +# submodule attribute as private and rejects cross-module imports +# under `--strict`. The list mirrors the symbols runtime.py +# actually consumes plus the convenience constructors / constants +# documented in the README. +__all__ = [ + "HEADER_PROTOCOL", + "NULLRUN_PROTOCOL_VERSION", + "DecisionSource", + "FallbackMode", + "FlushConfig", + "ExecuteConfig", + "Transport", + "TransportErrorSource", + "_retry_with_backoff", + "generate_hmac_signature", + "verify_hmac_signature", + "_signed_request_body", + "RateLimitError", + "InsecureTransportError", +] diff --git a/tests/test_2026_09_10_check_failopen.py b/tests/test_2026_09_10_check_failopen.py new file mode 100644 index 0000000..6fa4f71 --- /dev/null +++ b/tests/test_2026_09_10_check_failopen.py @@ -0,0 +1,330 @@ +"""DEF-NR-CHECK-FAIL-OPEN (2026-09-10) — ``Transport.check`` MUST NOT +synthesize ``decision_source=FALLBACK`` for 4xx responses. + +Pre-fix: + - ``nullrun/transport.py::Transport.check`` had a 4xx branch that + returned a synthetic + ``{"decision": "block", "decision_source": "fallback", ...}`` dict. + - The runtime's fail-OPEN path at ``runtime.py:2063-2088`` checks + ``decision_source.startswith("fallback")`` and returns a soft-pass + in that case — VIOLATING the CLAUDE.md §4 fail-CLOSED invariant. + - User-visible symptom: a 402 BUDGET_HARD_BLOCKED from the gateway + was downgraded to an allow. Wire-coded reasons + (BUDGET_HARD_BLOCKED / BUDGET_SOFT_BLOCKED / TOOL_BLOCKED / + RATE_LIMITED / etc.) were all silently swallowed. + +Post-fix: + - The 4xx branch parses the v3 wire envelope and returns a + gateway-shaped dict (``decision_source=DecisionSource.GATEWAY``, + NOT "fallback") with the wire envelope preserved + (``error_code``, ``explanation``, ``policy_id``, + ``remaining_budget_cents``, ``details``, ...). + - The runtime's existing ``decision=="block"`` arm then raises + ``NullRunBudgetError`` with first-class attrs intact. + +These tests pin BOTH the source shape (the 4xx branch returns +``DecisionSource.GATEWAY``, not ``FALLBACK``) AND the runtime behavior +(wire-coded reasons propagate as gateway decisions and trigger +``NullRunBudgetError``, NOT a silent allow). +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path + +import httpx +import pytest +import respx + +from nullrun.breaker.exceptions import ( + NullRunBudgetError, + NullRunError, +) +from nullrun.transport import DecisionSource, Transport + +SDK_ROOT = Path(__file__).resolve().parent.parent +TRANSPORT_PY = SDK_ROOT / "src" / "nullrun" / "transport.py" + + +def _read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _check_body() -> str: + """Return the source of ``Transport.check`` so source-pin tests + can grep for the expected 4xx branch without depending on Python + AST parsing. + + The signature is multiline + (``def check(\n self,\n workflow_id: ...``) so we + anchor on ``def check(`` and walk forward to the next top-level + ``def`` (4-space indent) inside the same class.""" + src = _read(TRANSPORT_PY) + start = src.find(" def check(\n") + assert start != -1, "could not locate Transport.check header" + after_header = src.index(" def check(\n", start) + len(" def check(\n") + m = re.search( + r"^ (?:def |@|class )", + src[after_header:], + re.MULTILINE, + ) + assert m, "could not locate end of Transport.check body" + end = after_header + m.start() + return src[start:end] + + +def _v3_envelope( + error_code: str, + status: int = 402, + *, + explanation: str | None = None, + policy_id: str | None = None, + remaining_budget_cents: int = 0, + projected_cost_cents: int | None = None, + reservation_id: str | None = None, + operation_id: str | None = None, + policy_version: int | None = None, + **details, +) -> httpx.Response: + """Build a v3-shaped 4xx response envelope mirroring the real + backend's wire contract.""" + body = { + "decision": "block", + "decision_source": DecisionSource.GATEWAY, + "explanation": explanation or error_code, + "explanations": [explanation or error_code], + "error_code": error_code, + "policy_id": policy_id, + "policy_version": policy_version, + "reservation_id": reservation_id, + "operation_id": operation_id, + "remaining_budget_cents": remaining_budget_cents, + "projected_cost_cents": projected_cost_cents, + "details": details, + } + return httpx.Response(status, json=body) + + +_CHECK_URL = "https://api.test.nullrun.io/api/v1/gate" + + +@pytest.fixture +def transport(): + t = Transport( + api_url="https://api.test.nullrun.io", + api_key="test-key-12345678", + ) + yield t + t.stop() + + +def _check_kwargs(): + return dict( + check_request={ + "organization_id": "ws-123", + "workflow_id": "wf-" + "a" * 32, + "trace_id": "trace-789", + "tool": "read_file", + }, + on_transport_error="raise", + ) + + +# ─── Source-pin tests (mirror cancel.rs / orchestrator.rs pin style) ─── + + +class TestDefNrCheckFailopenSourcePin: + """Pin the shape of the fix so a refactor that re-introduces + decision_source=fallback for 4xx fails loudly.""" + + def test_4xx_branch_uses_gateway_not_fallback(self): + """The 4xx branch in Transport.check MUST assign + DecisionSource.GATEWAY (or the literal "gateway" string), + not DecisionSource.FALLBACK. Pre-fix this branch synthesised + fallback and the runtime fail-OPENed.""" + body = _check_body() + # Locate the 4xx branch by its comment marker + idx = body.find("if 400 <= response.status_code < 500:") + assert idx != -1, ( + "DEF-NR-CHECK-FAIL-OPEN: 4xx branch anchor " + "`if 400 <= response.status_code < 500:` not found in " + "Transport.check" + ) + # Slice only the 4xx branch (stop at the next sibling `if` + # for the 5xx fallthrough). + five_xx_marker = body.find("if response.status_code >= 500", idx) + assert five_xx_marker != -1 + branch_body = body[idx:five_xx_marker] + assert "DecisionSource.GATEWAY" in branch_body, ( + "DEF-NR-CHECK-FAIL-OPEN: 4xx branch must use " + "DecisionSource.GATEWAY. Pre-fix it used " + "DecisionSource.FALLBACK and the runtime treated 4xx as " + "transport errors, fail-OPENing the gate." + ) + # The fallback synthesis arm must NOT be reachable on the 4xx + # path. We assert it does not co-exist in the 4xx branch + # body. (Other branches in check/execute MAY still use + # FALLBACK for genuine transport errors — that's outside + # this pin's scope.) + assert "DecisionSource.FALLBACK" not in branch_body, ( + "DEF-NR-CHECK-FAIL-OPEN: 4xx branch must not return " + "decision_source=FALLBACK. Runtime treats fallback as " + "transport error and silently allows." + ) + + def test_4xx_branch_preserves_wire_envelope(self): + """The 4xx branch must surface wire envelope fields + (``error_code``, ``explanation``, ``policy_id``, + ``remaining_budget_cents``, ``details``) so the runtime's + catalog dispatcher can build an actionable exception.""" + body = _check_body() + idx = body.find("if 400 <= response.status_code < 500:") + assert idx != -1 + five_xx_marker = body.find("if response.status_code >= 500", idx) + assert five_xx_marker != -1 + branch_body = body[idx:five_xx_marker] + for field in ( + "error_code", + "explanation", + "explanations", + "policy_id", + "reservation_id", + "remaining_budget_cents", + "projected_cost_cents", + "operation_id", + "details", + ): + assert field in branch_body, ( + f"DEF-NR-CHECK-FAIL-OPEN: 4xx branch must surface " + f"`{field}` from the wire envelope; runtime catalog " + f"dispatchers depend on it." + ) + + +# ─── Behaviour tests (pin the runtime outcome of the fix) ─── + + +class TestDefNrCheckFailopenBehavior: + """Verify that on a wire-coded 4xx response, Transport.check + returns a gateway-shaped dict (decision_source=gateway) so the + runtime's existing block dispatcher raises the typed exception + instead of fail-OPENing.""" + + def test_402_budget_hard_blocked_returns_gateway_dict(self, transport): + """A 402 BUDGET_HARD_BLOCKED from /check must return a + gateway-shaped dict, NOT a fallback dict.""" + policy_id = "pol-" + "a" * 32 + with respx.mock(assert_all_called=False) as mock: + mock.post(_CHECK_URL).mock( + return_value=_v3_envelope( + "BUDGET_HARD_BLOCKED", + status=402, + explanation="Hard budget limit exceeded", + policy_id=policy_id, + remaining_budget_cents=0, + projected_cost_cents=1, + budget_cents=1000, + current_spend_cents=1100, + enforcement_mode="hard", + ) + ) + result = transport.check(**_check_kwargs()) + assert result["decision"] == "block", ( + f"DEF-NR-CHECK-FAIL-OPEN: expected decision=block, got " + f"{result.get('decision')}" + ) + assert result["decision_source"] == DecisionSource.GATEWAY, ( + f"DEF-NR-CHECK-FAIL-OPEN: 4xx must produce " + f"decision_source=gateway, got {result.get('decision_source')}. " + f"Runtime treats fallback as transport error and fail-OPENs." + ) + assert result.get("error_code") == "BUDGET_HARD_BLOCKED" + assert result.get("policy_id") == policy_id + assert result.get("remaining_budget_cents") == 0 + + def test_403_tool_blocked_returns_gateway_dict(self, transport): + """A 403 TOOL_BLOCKED must surface with decision_source= + gateway so runtime raises NullRunBlockedException, not + silently allow.""" + with respx.mock(assert_all_called=False) as mock: + mock.post(_CHECK_URL).mock( + return_value=_v3_envelope( + "TOOL_BLOCKED", + status=403, + explanation="bash is blocked by policy", + policy_id="pol-toolblock", + details={"tool": "bash", "matched_patterns": ["bash"]}, + ) + ) + result = transport.check(**_check_kwargs()) + assert result["decision"] == "block" + assert result["decision_source"] == DecisionSource.GATEWAY + assert result.get("error_code") == "TOOL_BLOCKED" + assert result.get("explanation") == "bash is blocked by policy" + + def test_4xx_with_unparseable_body_returns_gateway_dict(self, transport): + """A 4xx with non-JSON body must still return a + gateway-shaped dict with explanation fallback — never + decision_source=fallback.""" + with respx.mock(assert_all_called=False) as mock: + mock.post(_CHECK_URL).mock( + return_value=httpx.Response(400, text="Bad Request") + ) + result = transport.check(**_check_kwargs()) + assert result["decision"] == "block" + assert result["decision_source"] == DecisionSource.GATEWAY + assert "400" in result.get("explanation", "") or len( + result.get("explanations") or [] + ) >= 1 + + def test_5xx_does_not_use_4xx_branch(self, transport): + """A 5xx (genuine transport error) must NOT take the 4xx + gateway branch — Transport.check must still raise a + transport-class exception (or return None per on_transport_error) + so the runtime's retry/escalation logic kicks in. This pins + the boundary: 4xx → gateway dict, 5xx → transport error.""" + + # Set on_transport_error=raise and capture the call result. + # We don't assert a specific exception type — that's + # documented behavior of the existing transport layer — + # only that the 5xx path does NOT silently return a + # gateway-shaped dict with decision_source=gateway. + with respx.mock(assert_all_called=False) as mock: + mock.post(_CHECK_URL).mock( + return_value=httpx.Response(503, text="upstream unavailable") + ) + kwargs = _check_kwargs() + kwargs["on_transport_error"] = "raise" + with pytest.raises((NullRunError, Exception)) as ei: + transport.check(**kwargs) + # Belt-and-braces: the exception (whatever it is) must not + # be a NullRunBudgetError with a wire-coded reason — that + # would be the same fail-OPEN class as the original defect. + if isinstance(ei.value, NullRunBudgetError): + pytest.fail( + "DEF-NR-CHECK-FAIL-OPEN regression: a 5xx " + "response must not surface as a budget exception " + "with a wire-coded reason. 5xx is transport " + "error, not a budget decision." + ) + + def test_200_returns_normal_dict(self, transport): + """Sanity: a 200 /check still returns the response body + verbatim — no gateway/fallback tagging that didn't come from + the server.""" + with respx.mock(assert_all_called=False) as mock: + mock.post(_CHECK_URL).mock( + return_value=httpx.Response( + 200, + json={ + "decision": "allow", + "decision_source": "gateway", + "remaining_budget_cents": 990, + }, + ) + ) + result = transport.check(**_check_kwargs()) + assert result["decision"] == "allow" + assert result.get("remaining_budget_cents") == 990 diff --git a/tests/test_transport.py b/tests/test_transport.py index c5c3397..0a88476 100644 --- a/tests/test_transport.py +++ b/tests/test_transport.py @@ -1079,6 +1079,7 @@ def capture(request: httpx.Request) -> httpx.Response: from nullrun.breaker.exceptions import ( NullRunAuthenticationError, + NullRunBackendError, NullRunTransportError, RateLimitError, TransportErrorSource, @@ -1216,22 +1217,28 @@ def test_execute_200_with_cache_write(): def test_execute_4xx_returns_block(): - """4xx (no special handling) → block-dict, decision_source FALLBACK.""" + """4xx (no special handling) → raises NullRunBackendError after + wire-envelope parse (def-nr-transport-catchfanin-gap closed the + pre-fix silent-fallback path that synthesised decision_source= + FALLBACK dicts and swallowed real wire-coded reasons). + + The fake response body uses the legacy `{"error": ...}` slug + shape, which the parser treats as unrecognised envelope and + surfaces as NullRunBackendError.""" t = _build_transport() fake_response = MagicMock() fake_response.status_code = 400 fake_response.json.return_value = {"error": "bad_request"} t._client.post = MagicMock(return_value=fake_response) - result = t.execute( - organization_id="org-1", - execution_id="wf-1", - trace_id="t-1", - tool="safe.tool", - input_data={}, - ) - assert result["decision"] == "block" - assert "400" in result["explanation"] + with pytest.raises(NullRunBackendError): + t.execute( + organization_id="org-1", + execution_id="wf-1", + trace_id="t-1", + tool="safe.tool", + input_data={}, + ) def test_execute_breaker_error_with_raise(): diff --git a/tests/test_transport_branches.py b/tests/test_transport_branches.py index 7d747d5..f5dfd4a 100644 --- a/tests/test_transport_branches.py +++ b/tests/test_transport_branches.py @@ -22,6 +22,7 @@ from nullrun.breaker.exceptions import ( NullRunAuthenticationError, + NullRunBackendError, NullRunTransportError, RateLimitError, TransportErrorSource, @@ -160,22 +161,26 @@ def test_execute_200_with_cache_write(): def test_execute_4xx_returns_block(): - """4xx (no special handling) → block-dict, decision_source FALLBACK.""" + """4xx (no special handling) → raises NullRunBackendError after + wire-envelope parse (def-nr-transport-catchfanin-gap closed the + pre-fix silent-fallback path). + + Legacy `{"error": ...}` slug body is unrecognised by the parser + and surfaces as NullRunBackendError.""" t = _build_transport() fake_response = MagicMock() fake_response.status_code = 400 fake_response.json.return_value = {"error": "bad_request"} t._client.post = MagicMock(return_value=fake_response) - result = t.execute( - organization_id="org-1", - execution_id="wf-1", - trace_id="t-1", - tool="safe.tool", - input_data={}, - ) - assert result["decision"] == "block" - assert "400" in result["explanation"] + with pytest.raises(NullRunBackendError): + t.execute( + organization_id="org-1", + execution_id="wf-1", + trace_id="t-1", + tool="safe.tool", + input_data={}, + ) def test_execute_breaker_error_with_raise(): From abde98deeb79146df9793b2fb3b700c5635ff801 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Thu, 10 Sep 2026 21:38:21 +0400 Subject: [PATCH 11/16] fix(sdk): runtime.execute block path dispatches typed catalog exceptions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-fix, the Layer-1 block-decision path in NullRunRuntime.execute unconditionally raised NullRunBlockedException with error_code=. Cookbook recipes that branched on typed catalog arms (e.g. except NullRunApprovalReplayRejectedError: for NR-A015) never matched — the wire SCREAMING_SNAKE code is not the catalog code that format_user_message looks up, so the user saw FALLBACK_MESSAGE ('Something went wrong. Please try again.') instead of the typed catalog wording. Repro observed on 2026-09-10 (langgraph approval demo, third refund): [sdk] Something went wrong. Please try again. (error_code=APPROVAL_REPLAY_REJECTED) Post-fix: the dispatch logic is factored into NullRunRuntime._build_block_exception which imports _V3_ERROR_CODE_MAP from nullrun.transport and dispatches the typed catalog class for known wire codes (Priority 1a: typed subclass, e.g. NullRunApprovalReplayRejectedError for APPROVAL_REPLAY_REJECTED). The catalog class attribute owns error_code — we MUST NOT pass error_code=wire_code to it because the constructor's details.pop('error_code') path would override NR-A015 with the wire code and defeat format_user_message. For catalog entries that map to the base NullRunBlockedException (Priority 1b, e.g. APPROVAL_VALIDATION_FAILED), the wire code IS self.error_code — back-compat callers branch on exc.error_code == 'APPROVAL_*'. Drift (Priority 2) and legacy keyword-on-explanation (Priority 3) paths preserve the same wire payload + mapped_class back-compat shim. Wire payload convention preserved (matching the existing NullRunBlockedException constructor): the wire payload lands nested under self.details['details'] via the details=... kwarg. Typed kwargs (e.g. approval_id on NullRunApprovalReplayRejectedError) are forwarded as named kwargs so they promote to first-class attrs (exc.approval_id) for cookbook recipes. Verification: 11/11 new tests pass (tests/test_2026_09_10_runtime_block_typed_dispatch.py), and the full SDK suite (1749 tests) is green — no regressions on the pre-existing test_runtime.py::test_execute_blocked_surfaces_wire_error_code (DEF-ARFLOW-TOOLNAME-01, 2026-08-05) which asserts the wire-payload + mapped_class back-compat shape. --- src/nullrun/runtime.py | 269 +++++++++---- ...2026_09_10_runtime_block_typed_dispatch.py | 356 ++++++++++++++++++ 2 files changed, 560 insertions(+), 65 deletions(-) create mode 100644 tests/test_2026_09_10_runtime_block_typed_dispatch.py diff --git a/src/nullrun/runtime.py b/src/nullrun/runtime.py index 50e01a9..327ebce 100644 --- a/src/nullrun/runtime.py +++ b/src/nullrun/runtime.py @@ -3057,73 +3057,15 @@ def execute( # # The backend stamps a structured ``details.error_code`` # on every block response, alongside the existing - # BUDGET_* / RATE_LIMIT_* family. When the backend - # provides one, we use it verbatim -- no string - # parsing, no false positives. Falls back to the - # legacy keyword-on-explanation mapping for older - # backends that pre-date the structured code (the - # keyword path stays for back-compat -- an older - # SDK still classifies budget/loop/rate/tool blocks - # correctly). - explanation = result.get("explanation", "policy violation") - wire_details = result.get("details") or {} - if not isinstance(wire_details, dict): - wire_details = {} - wire_error_code = wire_details.get("error_code") - if wire_error_code and isinstance(wire_error_code, str): - # Backend-supplied structured code wins. The - # catalogue exception class is mapped via - # ``_V3_ERROR_CODE_MAP`` on the transport path; on - # this /execute path we only have the SCREAMING_SNAKE - # backend code, so we surface it as-is in the - # ``error_code`` slot and let the caller branch on - # the catalog subclass if it has imported one. The - # block_code -> SDK exception-class mapping is done - # via the catalogue in nullrun.breaker.exceptions. - block_code, block_action = wire_error_code, "block" - block_cls = "NullRunBlockedException" - else: - explanation_lower = explanation.lower() - if "budget" in explanation_lower or "exhausted" in explanation_lower: - block_code, block_action = "NR-B004", "block" - block_cls = "NullRunBudgetError" - elif "loop" in explanation_lower or "repetition" in explanation_lower: - block_code, block_action = "NR-L001", "block" - block_cls = "NullRunBlockedException" - elif "rate" in explanation_lower or "too many" in explanation_lower: - block_code, block_action = "NR-R001", "block" - block_cls = "NullRunBlockedException" - elif "tool" in explanation_lower and "block" in explanation_lower: - block_code, block_action = "NR-T001", "block" - block_cls = "NullRunToolBlockedError" - else: - block_code, block_action = "NR-X001", "block" - block_cls = "NullRunBlockedException" - # Note: we still raise the base ``NullRunBlockedException`` - # for non-budget/tool cases to keep the construction - # shape simple — the catalogue code is what the user - # reads, and they can branch on it via ``except - # NullRunBudgetError:`` for the budget case if they need - # to handle it specifically. We could instantiate the - # subclass per branch above; keeping one raise here is - # easier to reason about and matches the way the rest of - # the codebase handles backend blocks. - # - # ``details`` carries the wire ``details`` payload so the - # caller can introspect ``exc.details["error_code"]`` and - # ``exc.details["decision_source"]`` for diagnostic - # routing. ``mapped_class`` is preserved as a backwards- - # compat shim for callers that branched on the keyword - # path; new code should branch on ``exc.error_code``. - merged_details = dict(wire_details) - merged_details["mapped_class"] = block_cls - err = NullRunBlockedException( + # BUDGET_* / RATE_LIMIT_* family. Layer-1 dispatch is + # factored into ``_build_block_exception`` so the + # dispatch logic is unit-testable without standing up a + # full ``Runtime.execute()`` pipeline (see + # tests/test_2026_09_10_runtime_block_typed_dispatch.py). + err = self._build_block_exception( + result=result, workflow_id=workflow_id or UNKNOWN_WORKFLOW_ID, - reason=explanation, - action=block_action, tool_name=tool_name, - error_code=block_code, - details=merged_details, ) # Layer 2: fire the on_error hook. The hook sees the # same exception the caller will catch plus the @@ -3142,6 +3084,203 @@ def execute( metrics.inc_runtime("execute_allowed") return result + @staticmethod + def _build_block_exception( + *, + result: dict[str, Any], + workflow_id: str, + tool_name: str | None, + ) -> Exception: + """Build the typed catalog exception for a /execute block + decision. Factored out of ``Runtime.execute`` so the dispatch + logic is unit-testable without a full transport pipeline + (see tests/test_2026_09_10_runtime_block_typed_dispatch.py). + + Dispatch priority: + + 1. ``result["details"]["error_code"]`` is in + ``_V3_ERROR_CODE_MAP`` → instantiate the typed catalog + class (e.g. ``NullRunApprovalReplayRejectedError`` for + ``APPROVAL_REPLAY_REJECTED``). This is the common path + and is what cookbook recipes branch on + (``except NullRunApprovalReplayRejectedError:`` for + NR-A015, etc.). + + 2. Wire code present but NOT in the catalog (backend / SDK + drift) → ``NullRunBlockedException`` with + ``error_code=`` so the caller still has + something to branch on via ``exc.error_code``. + + 3. No structured wire code (legacy backends) → keyword-on- + ``explanation`` mapping for BUDGET / LOOP / RATE / TOOL + families. Each branch sets a synthetic + ``NR-B004 / NR-L001 / NR-R001 / NR-T001`` catalog code + on ``NullRunBlockedException`` (kept for backward + compat — old code branched on + ``exc.error_code == "NR-B004"``). + + The class attribute ``error_code`` on the typed class + (e.g. ``NullRunApprovalReplayRejectedError.error_code == + "NR-A015"``) is the canonical ``format_user_message`` + lookup key — we NEVER pass ``error_code=wire_code`` to the + typed class because that would override the catalog code + and defeat ``format_user_message``. + + Returns a ``NullRunBlockedException`` (or subclass) — + caller is responsible for raising / ``on_error`` hook / + metric increment. + """ + from nullrun.transport import _V3_ERROR_CODE_MAP + + explanation = result.get("explanation", "policy violation") + wire_details = result.get("details") or {} + if not isinstance(wire_details, dict): + wire_details = {} + wire_error_code = wire_details.get("error_code") + + # Catalog classes with a custom ``__init__`` that promotes + # a wire field to a first-class attribute (e.g. + # ``exc.approval_id`` on ``NullRunApprovalReplayRejectedError``, + # ``exc.current_spend_cents`` on + # ``NullRunBudgetRecheckFailedError``). Mirrors the + # keyword-only params declared on those classes in + # ``breaker/exceptions.py``. The value is a frozenset of + # wire-field names that must be passed as named kwargs to + # the constructor (NOT nested under ``details=``) so they + # land on the typed exception as first-class attrs. + # + # Classes that map to ``NullRunBlockedException`` (the base + # class — e.g. ``NullRunBudgetError``, + # ``NullRunToolBlockedError``) do NOT appear here because + # they have no custom ``__init__`` that promotes fields; all + # wire fields stay in the ``details=`` payload. + _TYPED_KWARGS_BY_CLASS: dict[str, frozenset[str]] = { + "NullRunBudgetRecheckFailedError": frozenset( + {"current_spend_cents", "budget_cents", "epsilon_cents"} + ), + "NullRunApprovalNotYetApprovedError": frozenset({"approval_id"}), + "NullRunApprovalDeniedError": frozenset( + {"approval_id", "denial_note"} + ), + "NullRunApprovalExpiredError": frozenset( + {"approval_id", "timeout_seconds"} + ), + "NullRunApprovalReplayRejectedError": frozenset({"approval_id"}), + "NullRunApprovalDigestMismatchError": frozenset({"approval_id"}), + "NullRunApprovalToolDigestMismatchError": frozenset({"approval_id"}), + } + + def _build_payload( + src: dict[str, Any], mapped_name: str + ) -> dict[str, Any]: + """Build the ``details=...`` payload dict that the + constructor captures as ``self.details["details"]``. + + Carries the full wire payload verbatim (including + ``error_code``) so callers can introspect the wire + shape for routing/alerting (e.g. + ``exc.details["details"]["decision_source"]``). The + ``mapped_class`` shim is appended for back-compat with + callers that branched on the legacy keyword path. + """ + payload = dict(src) + payload["mapped_class"] = mapped_name + return payload + + # Priority 1: typed catalog dispatch via _V3_ERROR_CODE_MAP. + if wire_error_code and isinstance(wire_error_code, str): + typed_cls = _V3_ERROR_CODE_MAP.get(wire_error_code) + if typed_cls is not None: + # 1a: typed SUBCLASS (e.g. + # NullRunApprovalReplayRejectedError for + # APPROVAL_REPLAY_REJECTED). The class's class + # attribute owns error_code (NR-A015), so we MUST + # NOT pass error_code=... — that would override + # NR-A015 with the wire code and defeat + # format_user_message's catalog lookup. + if typed_cls is not NullRunBlockedException: + payload = _build_payload( + wire_details, typed_cls.__name__ + ) + typed_kwarg_names = _TYPED_KWARGS_BY_CLASS.get( + typed_cls.__name__, frozenset() + ) + typed_kwargs = { + k: wire_details[k] + for k in typed_kwarg_names + if k in wire_details + } + return typed_cls( + workflow_id=workflow_id, + reason=explanation, + action="block", + tool_name=tool_name, + details=payload, + **typed_kwargs, + ) + # 1b: catalog maps to base NullRunBlockedException + # (e.g. APPROVAL_VALIDATION_FAILED). There is no + # catalog error_code class attr to protect — we + # pass the wire code explicitly so self.error_code + # reflects the wire code (back-compat callers + # branch on exc.error_code == "APPROVAL_*"). + payload = _build_payload( + wire_details, "NullRunBlockedException" + ) + return NullRunBlockedException( + workflow_id=workflow_id, + reason=explanation, + action="block", + tool_name=tool_name, + error_code=wire_error_code, + details=payload, + ) + # Priority 2: wire code NOT in catalog (drift). + # Base class — wire code IS self.error_code (no catalog + # entry to dispatch to, so no class attr to protect). + payload = _build_payload( + wire_details, "NullRunBlockedException" + ) + return NullRunBlockedException( + workflow_id=workflow_id, + reason=explanation, + action="block", + tool_name=tool_name, + error_code=wire_error_code, + details=payload, + ) + + # Priority 3: legacy keyword-on-explanation mapping for + # backends that pre-date the structured wire code. Each + # branch picks a synthetic catalog code so legacy + # ``exc.error_code == "NR-B004"``-style branching still + # works for back-compat callers. + explanation_lower = explanation.lower() + if "budget" in explanation_lower or "exhausted" in explanation_lower: + block_code = "NR-B004" + mapped = "NullRunBudgetError" + elif "loop" in explanation_lower or "repetition" in explanation_lower: + block_code = "NR-L001" + mapped = "NullRunBlockedException" + elif "rate" in explanation_lower or "too many" in explanation_lower: + block_code = "NR-R001" + mapped = "NullRunBlockedException" + elif "tool" in explanation_lower and "block" in explanation_lower: + block_code = "NR-T001" + mapped = "NullRunToolBlockedError" + else: + block_code = "NR-X001" + mapped = "NullRunBlockedException" + payload = _build_payload(wire_details, mapped) + return NullRunBlockedException( + workflow_id=workflow_id, + reason=explanation, + action="block", + tool_name=tool_name, + error_code=block_code, + details=payload, + ) + def _enrich_event(self, event: dict[str, Any]) -> dict[str, Any]: """Add context fields to event.""" enriched = dict(event) # Don't modify original diff --git a/tests/test_2026_09_10_runtime_block_typed_dispatch.py b/tests/test_2026_09_10_runtime_block_typed_dispatch.py new file mode 100644 index 0000000..6815a86 --- /dev/null +++ b/tests/test_2026_09_10_runtime_block_typed_dispatch.py @@ -0,0 +1,356 @@ +"""DEF-NR-RUNTIME-BLOCK-TYPED (2026-09-10) — ``Runtime.execute`` +block path MUST dispatch via ``_V3_ERROR_CODE_MAP`` so typed catalog +exceptions (``NullRunApprovalReplayRejectedError`` / NR-A015, +``NullRunBudgetError`` / NR-B004, ``NullRunToolBlockedError`` / +NR-T001, etc.) actually surface — NOT a base +``NullRunBlockedException`` with the wire SCREAMING_SNAKE code +attached as ``error_code``. + +Pre-fix (runtime.py:3053-3140, before this commit): + + When ``self._transport.execute(**execute_kwargs)`` returned + ``{"decision": "block", "details": {"error_code": + "APPROVAL_REPLAY_REJECTED"}}``, the runtime ALWAYS raised the + base ``NullRunBlockedException`` with + ``error_code="APPROVAL_REPLAY_REJECTED"`` (the wire code) — a + SCREAMING_SNAKE string, NOT the catalog ``NR-A015`` that + ``format_user_message`` looks up. Cookbook recipes that branched + on the typed catalog arm (``except + NullRunApprovalReplayRejectedError:``) NEVER matched, fell + through to the generic ``NullRunError`` arm, and the user saw + ``FALLBACK_MESSAGE`` ("Something went wrong. Please try again.") + instead of the typed catalog wording. + +Post-fix (this commit): + + Layer-1 dispatch factored into + ``Runtime._build_block_exception``. The helper imports + ``_V3_ERROR_CODE_MAP`` from ``nullrun.transport`` and dispatches + via ``typed_cls = _V3_ERROR_CODE_MAP.get(wire_error_code)``. If + the wire code is in the catalog, the helper raises the typed + class (e.g. ``NullRunApprovalReplayRejectedError`` for + ``APPROVAL_REPLAY_REJECTED``), so cookbook ``except`` arms + match. The class's ``error_code`` attribute is the catalog + ``NR-A015`` (NOT the wire code), so ``format_user_message`` + yields the friendly catalog wording. + +These tests pin BOTH the source shape (the runtime block path +imports + uses ``_V3_ERROR_CODE_MAP``) AND the runtime behavior +(wire-coded reasons propagate as the typed catalog class). + +Wire payload convention (preserved by the fix): the constructor +captures ``**details`` into ``self.details`` and nests the wire +payload under ``self.details["details"]`` via the ``details=...`` +kwarg. Cookbook code reads ``exc.details["details"]["..."]`` for +typed introspection; ``exc.details["details"]["mapped_class"]`` +exposes the catalog class name as a back-compat shim. +""" +from __future__ import annotations + +import re +from pathlib import Path + +import pytest + +from nullrun.breaker.exceptions import ( + NullRunApprovalReplayRejectedError, + NullRunBlockedException, + NullRunBudgetError, + NullRunToolBlockedError, +) +from nullrun.runtime import NullRunRuntime + +SDK_ROOT = Path(__file__).resolve().parent.parent +RUNTIME_PY = SDK_ROOT / "src" / "nullrun" / "runtime.py" + + +def _read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _build_block_exception_slice() -> str: + """Return the source of ``_build_block_exception`` for the + source-pin tests. Anchor on ``def _build_block_exception`` and + walk to the next sibling ``def`` (4-space indent) inside the + same class.""" + src = _read(RUNTIME_PY) + start = src.find(" def _build_block_exception(") + assert start != -1, ( + "DEF-NR-RUNTIME-BLOCK-TYPED: cannot locate " + "Runtime._build_block_exception" + ) + after_header = start + len(" def _build_block_exception(\n") + m = re.search( + r"^ (?:def |@|class )", + src[after_header:], + re.MULTILINE, + ) + assert m, "could not locate end of _build_block_exception body" + end = after_header + m.start() + return src[start:end] + + +# ─── Source-pin tests ────────────────────────────────────────────────────── + + +class TestDefNrRuntimeBlockTypedSourcePin: + """Pin the shape of the fix so a refactor that re-introduces + the always-NRError wrap fails loudly.""" + + def test_helper_imports_v3_error_code_map(self): + """The helper MUST import ``_V3_ERROR_CODE_MAP`` from + ``nullrun.transport`` so it can dispatch typed catalog + classes.""" + body = _build_block_exception_slice() + assert "from nullrun.transport import _V3_ERROR_CODE_MAP" in body, ( + "DEF-NR-RUNTIME-BLOCK-TYPED: _build_block_exception " + "must import _V3_ERROR_CODE_MAP from nullrun.transport. " + "Pre-fix the runtime always raised base " + "NullRunBlockedException with the wire code as " + "error_code, hiding the typed catalog from cookbook " + "recipes." + ) + + def test_helper_dispatches_typed_cls(self): + """The helper MUST look up + ``typed_cls = _V3_ERROR_CODE_MAP.get(wire_error_code)`` + and instantiate it for the wire-coded reason. Pre-fix + it always instantiated the base class.""" + body = _build_block_exception_slice() + assert re.search( + r"typed_cls\s*=\s*_V3_ERROR_CODE_MAP\.get\(", body + ), ( + "DEF-NR-RUNTIME-BLOCK-TYPED: helper must dispatch via " + "`_V3_ERROR_CODE_MAP.get(wire_error_code)` so the " + "wire-coded reason maps to the typed catalog class." + ) + assert "typed_cls(" in body, ( + "DEF-NR-RUNTIME-BLOCK-TYPED: helper must instantiate " + "the typed class — pre-fix it only constructed the " + "base NullRunBlockedException." + ) + + def test_helper_does_not_overwrite_catalog_code_with_wire_code(self): + """Pre-fix, the runtime passed + ``error_code=block_code=wire_error_code`` — overriding the + catalog ``NR-A015`` with the wire + ``APPROVAL_REPLAY_REJECTED``. The fix must NOT pass + ``error_code=wire_error_code`` to the typed class. The + typed class's class attribute (e.g. ``NR-A015``) must + stay intact so ``format_user_message`` can look it up.""" + body = _build_block_exception_slice() + m = re.search(r"return\s+typed_cls\(", body) + assert m is not None, "typed_cls(...) return not found" + # Walk forward from the return to the closing paren (allow + # nested parens for type_specific_kwargs etc.). + call_start = m.end() - len("typed_cls(") + depth = 0 + end = None + for i in range(call_start, len(body)): + ch = body[i] + if ch == "(": + depth += 1 + elif ch == ")": + depth -= 1 + if depth == 0: + end = i + 1 + break + assert end is not None, "could not find end of typed_cls(...) call" + call_region = body[call_start:end] + assert "error_code=" not in call_region, ( + "DEF-NR-RUNTIME-BLOCK-TYPED: typed_cls(...) call must " + "NOT pass error_code=... — that would override the " + "typed class's catalog code (NR-A015, NR-B004, etc.) " + "with the wire SCREAMING_SNAKE code and defeat " + "format_user_message." + ) + + +# ─── Behaviour tests (pin the runtime outcome of the fix) ───────────────── + + +class TestDefNrRuntimeBlockTypedBehavior: + """Verify that a wire-coded ``decision: block`` with + ``details.error_code`` in the catalog dispatch path raises the + typed catalog class, NOT a base NullRunBlockedException.""" + + def test_approval_replay_rejected_raises_typed(self): + """A /execute response with ``decision: block`` and + ``details.error_code: APPROVAL_REPLAY_REJECTED`` must + raise ``NullRunApprovalReplayRejectedError`` (NR-A015), + NOT the base ``NullRunBlockedException``.""" + result = { + "decision": "block", + "explanation": "approval grant already consumed", + "details": { + "error_code": "APPROVAL_REPLAY_REJECTED", + "approval_id": "apr-test-123", + }, + } + exc = NullRunRuntime._build_block_exception( + result=result, + workflow_id="wf-abc", + tool_name="refund_customer", + ) + assert isinstance(exc, NullRunApprovalReplayRejectedError), ( + f"expected NullRunApprovalReplayRejectedError, got " + f"{type(exc).__name__}" + ) + assert exc.error_code == "NR-A015", ( + f"expected NR-A015 (catalog), got {exc.error_code!r} " + "(wire code?)" + ) + assert exc.approval_id == "apr-test-123", ( + f"expected approval_id forwarded from wire_details, " + f"got {exc.approval_id!r}" + ) + # CRITICAL: must NOT be the base class + assert type(exc) is not NullRunBlockedException, ( + "DEF-NR-RUNTIME-BLOCK-TYPED: wire APPROVAL_REPLAY_REJECTED " + "must surface as NullRunApprovalReplayRejectedError, not " + "the base NullRunBlockedException." + ) + + def test_budget_hard_blocked_raises_typed(self): + """A /execute response with + ``details.error_code: BUDGET_HARD_BLOCKED`` must raise + ``NullRunBudgetError`` (NR-B004), not the base.""" + result = { + "decision": "block", + "explanation": "Hard budget exceeded", + "details": { + "error_code": "BUDGET_HARD_BLOCKED", + "budget_cents": 5000, + "current_spend_cents": 5100, + }, + } + exc = NullRunRuntime._build_block_exception( + result=result, + workflow_id="wf-abc", + tool_name="refund_customer", + ) + assert isinstance(exc, NullRunBudgetError) + assert exc.error_code == "NR-B004" + # The wire payload is nested under exc.details["details"] + # (back-compat convention — the constructor captures the + # ``details=`` kwarg as ``self.details["details"]``). + # Cookbook code reads budget_cents / current_spend_cents via + # exc.details["details"]; only + # NullRunBudgetRecheckFailedError (NR-B006) promotes them + # to first-class attributes. + wire_payload = exc.details.get("details") or {} + assert wire_payload.get("budget_cents") == 5000 + assert wire_payload.get("current_spend_cents") == 5100 + assert wire_payload.get("mapped_class") == "NullRunBudgetError" + + def test_tool_blocked_raises_typed(self): + """A /execute response with + ``details.error_code: TOOL_BLOCKED`` must raise + ``NullRunToolBlockedError``, not the base.""" + result = { + "decision": "block", + "explanation": "Tool bash is blocked", + "details": {"error_code": "TOOL_BLOCKED"}, + } + exc = NullRunRuntime._build_block_exception( + result=result, + workflow_id="wf-abc", + tool_name="bash", + ) + assert isinstance(exc, NullRunToolBlockedError) + assert exc.error_code == "NR-T001" + + def test_unknown_wire_code_falls_back_to_base(self): + """A wire code that is NOT in ``_V3_ERROR_CODE_MAP`` (drift + between backend and SDK) must still surface on the base + ``NullRunBlockedException`` with the wire code as + ``error_code`` — the operator / cookbook code can still + branch on ``exc.error_code``.""" + result = { + "decision": "block", + "explanation": "Unknown rejection", + "details": {"error_code": "BRAND_NEW_CODE_FROM_BACKEND"}, + } + exc = NullRunRuntime._build_block_exception( + result=result, + workflow_id="wf-abc", + tool_name="refund_customer", + ) + assert type(exc) is NullRunBlockedException + # The wire code (NOT a catalog code) is the + # ``error_code`` for drift visibility. + assert exc.error_code == "BRAND_NEW_CODE_FROM_BACKEND" + # mapped_class shim is preserved for back-compat + wire_payload = exc.details.get("details") or {} + assert wire_payload.get("mapped_class") == "NullRunBlockedException" + + def test_legacy_keyword_path_budget(self): + """A /execute response with no wire code but with + ``explanation: 'budget exceeded'`` must fall back to the + legacy keyword path. Pre-fix the comment said this raised + ``NullRunBudgetError`` but the construction always used + the base ``NullRunBlockedException``; this test pins that + the legacy path stays on the base class (the wire-code + path is the one that dispatches via ``_V3_ERROR_CODE_MAP``) + while ``error_code`` is set to ``NR-B004`` for back-compat + branches on ``exc.error_code == "NR-B004"``.""" + result = { + "decision": "block", + "explanation": "Your budget was exceeded", + "details": {}, + } + exc = NullRunRuntime._build_block_exception( + result=result, + workflow_id="wf-abc", + tool_name="refund_customer", + ) + assert type(exc) is NullRunBlockedException + assert exc.error_code == "NR-B004" + wire_payload = exc.details.get("details") or {} + assert wire_payload.get("mapped_class") == "NullRunBudgetError" + + def test_legacy_keyword_path_unknown_explanation_falls_back_to_x001(self): + """A /execute response with no wire code AND no keyword + match falls through to ``NR-X001`` on the base class.""" + result = { + "decision": "block", + "explanation": "Some unparseable reason", + "details": {}, + } + exc = NullRunRuntime._build_block_exception( + result=result, + workflow_id="wf-abc", + tool_name="refund_customer", + ) + assert type(exc) is NullRunBlockedException + assert exc.error_code == "NR-X001" + + def test_format_user_message_yields_catalog_wording(self): + """End-to-end: with the fix in place, + ``format_user_message(NullRunApprovalReplayRejectedError)`` + returns the friendly NR-A015 wording — NOT the generic + FALLBACK_MESSAGE that the user saw before the fix.""" + from nullrun.messages import FALLBACK_MESSAGE, format_user_message + + result = { + "decision": "block", + "explanation": "approval grant already consumed", + "details": { + "error_code": "APPROVAL_REPLAY_REJECTED", + "approval_id": "apr-test-123", + }, + } + exc = NullRunRuntime._build_block_exception( + result=result, + workflow_id="wf-abc", + tool_name="refund_customer", + ) + msg = format_user_message(exc) + assert msg != FALLBACK_MESSAGE, ( + "DEF-NR-RUNTIME-BLOCK-TYPED: format_user_message must " + "yield the catalog wording, not FALLBACK_MESSAGE. " + "Pre-fix the runtime hid the typed exception behind a " + "base NullRunBlockedException(error_code='APPROVAL_REPLAY_REJECTED'), " + "which the catalog could not resolve." + ) + assert "approval has already been used" in msg From a0c233c1b59477e7cc27f78a3277dd2166f0328f Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Thu, 10 Sep 2026 21:42:44 +0400 Subject: [PATCH 12/16] fix transport v2 --- src/nullrun/__init__.py | 9 + src/nullrun/breaker/exceptions.py | 27 --- src/nullrun/messages.py | 9 +- src/nullrun/transport.py | 16 +- tests/test_2026_08_11_fixes.py | 14 +- tests/test_2026_09_10_sdk_cleanup.py | 311 +++++++++++++++++++++++++++ tests/test_messages.py | 17 +- tests/test_v3_wire_contract.py | 21 +- 8 files changed, 367 insertions(+), 57 deletions(-) create mode 100644 tests/test_2026_09_10_sdk_cleanup.py diff --git a/src/nullrun/__init__.py b/src/nullrun/__init__.py index 7bc14a6..f5934e6 100644 --- a/src/nullrun/__init__.py +++ b/src/nullrun/__init__.py @@ -492,6 +492,14 @@ def my_agent: "WorkflowPausedException": ("nullrun.breaker.exceptions", "WorkflowPausedException"), "WorkflowKilledException": ("nullrun.breaker.exceptions", "WorkflowKilledException"), "WorkflowKilledInterrupt": ("nullrun.breaker.exceptions", "WorkflowKilledInterrupt"), + # Sibling typed name for the kill signal. Discovered via + # ``from nullrun import NullRunWorkflowKilledError``; matches + # `WorkflowKilledInterrupt` (BaseException) and the older + # `WorkflowKilledException` for back-compat. Cookbook code + # that wants a typed ``except`` clause prefers this over the + # base-interrupt form (mro-aware dispatch). The class lives at + # breaker/exceptions.py:1459. + "NullRunWorkflowKilledError": ("nullrun.breaker.exceptions", "NullRunWorkflowKilledError"), # User-facing message catalog (NULLRUN owns the wording; see # nullrun/messages.py for the design rationale). Eager in # spirit — these are the "give the user a chance" surface that @@ -604,6 +612,7 @@ def __dir__() -> list[str]: "NullRunBudgetError", "NullRunToolBlockedError", "WorkflowKilledInterrupt", + "NullRunWorkflowKilledError", # User-facing message catalog — the single entry point for # turning an SDK exception into a string safe to display to # end users. ``set_user_message`` lets a deployment brand its diff --git a/src/nullrun/breaker/exceptions.py b/src/nullrun/breaker/exceptions.py index 473ebf5..f1e404c 100644 --- a/src/nullrun/breaker/exceptions.py +++ b/src/nullrun/breaker/exceptions.py @@ -842,33 +842,6 @@ def __init__( self.recheck_retryable: bool = True -class NullRunBudgetThrottleError(NullRunBudgetError): - """Backend returned ``decision == "throttle"`` — soft budget signal. - - Distinct from :class:`NullRunBudgetError` (NR-B004, the hard-block - case raised when ``decision == "block"``). Throttle means - "rate-limit this workflow but don't fully block it" — a temporary - pacing signal that the SDK surfaces as a typed exception so - cookbook code can back off and retry, vs. the hard block where - the same parameters would fail again. - - Added 2026-09-08 to retire the generic ``WorkflowKilledInterrupt`` - raise on the throttle path. Cookbook pattern: catch this - specifically (``except NullRunBudgetThrottleError``), sleep for - the cooldown window, and retry — distinct from the hard block - where retrying with the same budget tier is futile. - """ - - error_code = "NR-B007" - user_action = ( - "Backend throttled this workflow (soft budget signal). Wait " - "for the cooldown window shown in the response and retry — " - "do NOT request a budget increase for a throttle (that is " - "the wrong remediation; the issue is pacing, not cap)." - ) - retryable = True - - class NullRunExecutionNotFoundError(NullRunBackendError): """``/execute`` or ``/cancel`` was called with an ``execution_id`` that has no live server-side binding. diff --git a/src/nullrun/messages.py b/src/nullrun/messages.py index 2fe370e..7f7d094 100644 --- a/src/nullrun/messages.py +++ b/src/nullrun/messages.py @@ -145,9 +145,12 @@ # the next /gate will mint a fresh reservation against the current # available budget. "NR-B006": "Your request couldn't be completed because the available capacity changed. Please try again.", - # NR-B007: workflow throttle (soft budget signal). Pacing issue, not - # cap; user should slow down and retry after the cooldown window. - "NR-B007": "You're sending requests too quickly. Please slow down and try again in a moment.", + # NR-B007: removed 2026-09-10. NullRunBudgetThrottleError was a + # zombie class — never raised on a wire or runtime path + # (runtime.py:2116 raises WorkflowPausedException on + # decision=="throttle"). Catalog entry removed to keep + # messages in sync with the exception module. + # "NR-B007": "...", # NR-O001: consume > reserve + ε tolerance. ADR-005 invariant; # the SDK rejects rather than silently re-reserving. User-facing # copy is generic because the cause is operator-side accounting; diff --git a/src/nullrun/transport.py b/src/nullrun/transport.py index bad6360..bcafa89 100644 --- a/src/nullrun/transport.py +++ b/src/nullrun/transport.py @@ -2562,7 +2562,13 @@ def _safe_json(response: httpx.Response, endpoint: str) -> Any: f"(status={response.status_code}): {type(exc).__name__}", source=TransportErrorSource.GATEWAY_ERROR, endpoint=endpoint, - error_code="NR-T001", + # NR-T001 collides with NullRunToolBlockedError's + # canonical code (breaker/exceptions.py:955); using + # NR-T-PARSE here so a cookbook handler that branches + # on `exc.error_code == "NR-T001"` does not mis-classify + # a JSON parse failure as a tool block. See + # tests/test_2026_08_11_fixes.py for the pin. + error_code="NR-T-PARSE", ) from exc @@ -2936,7 +2942,13 @@ def _build_v3_error_code_map() -> dict[str, type[Exception]]: "BUDGET_SOFT_BLOCKED": NullRunBudgetError, "BUDGET_OVERDRAFT_EXCEEDED": NullRunBudgetError, "BUDGET_PERIOD_NOT_STARTED": NullRunBudgetError, - "REDIS_UNAVAILABLE": NullRunBudgetError, + # Note: BUDGET_REDIS_UNAVAILABLE and RATE_LIMIT_REDIS_UNAVAILABLE + # below are the canonical redis-down codes (post-v3.36 rename); + # the legacy ``REDIS_UNAVAILABLE`` slug was removed 2026-09-10 + # because the backend never emits it (it is absent from + # ``GateErrorCode::all()`` in error_codes.rs). A cookbook that + # extends this map with the legacy slug risks silently matching + # nothing, so the slot stays unoccupied by design. # 402 — chain family (separate class for diagnostic clarity) "CHAIN_MAX_DURATION_EXCEEDED": NullRunChainError, # 403 — chain security + workflow state diff --git a/tests/test_2026_08_11_fixes.py b/tests/test_2026_08_11_fixes.py index 6434436..1fee406 100644 --- a/tests/test_2026_08_11_fixes.py +++ b/tests/test_2026_08_11_fixes.py @@ -135,11 +135,17 @@ def test_safe_json_helper_exists_and_wraps_json_errors(): "transport.py must define _safe_json(response, endpoint) " "helper to wrap JSON parse failures" ) - # The helper must raise NullRunTransportError with NR-T001 - # (consistent with the rest of the SDK's error_code vocabulary) - assert 'error_code="NR-T001"' in src, ( + # The helper must raise NullRunTransportError with NR-T-PARSE. + # NR-T001 (tool-block) was the historical literal here, but it + # collides with NullRunToolBlockedError.error_code + # (breaker/exceptions.py:955); cookbook handlers that branch on + # `exc.error_code == "NR-T001"` would mis-classify a JSON parse + # failure as a tool block. NR-T-PARSE is the new dedicated + # transport-class code; matches NR-T (transport) vocabulary. + assert 'error_code="NR-T-PARSE"' in src, ( "_safe_json must raise NullRunTransportError with " - "error_code=NR-T001 (consistent with NR-A/NR-B vocabulary)" + "error_code=NR-T-PARSE (avoids collision with NR-T001 / " + "NullRunToolBlockedError)" ) # body_preview truncation is part of the fix; the helper # must slice body to 200 chars max. diff --git a/tests/test_2026_09_10_sdk_cleanup.py b/tests/test_2026_09_10_sdk_cleanup.py new file mode 100644 index 0000000..27984b2 --- /dev/null +++ b/tests/test_2026_09_10_sdk_cleanup.py @@ -0,0 +1,311 @@ +"""Source-pin regression tests for the SDK cleanup batch (2026-09-10). + +Each test pins a single cleanup fix to prevent future refactors from +silently re-introducing the debt that was removed. Mirrors the +source-pin pattern from ``test_2026_08_11_fixes.py``. + +Defects / cleanups being pinned (RUN_ID 2026-09-10 batch): + +- CLEANUP-PARSE-CODE-COLLISION — ``_safe_json`` previously raised + with ``error_code="NR-T001"``, which collides with + ``NullRunToolBlockedError.error_code`` (breaker/exceptions.py:955). + Cookbook handlers that branch on ``exc.error_code == "NR-T001"`` + mis-classified a JSON parse failure as a tool block. Pin: the + literal must be NR-T-PARSE. + +- CLEANUP-REDIS-UNAVAILABLE-CODE — ``transport.py`` previously + mapped the backend's ``REDIS_UNAVAILABLE`` envelope to + ``NullRunBudgetError``. That's wrong: the backend distinguishes + BUDGET_REDIS_UNAVAILABLE (budget path, fail-CLOSED 402) and + RATE_LIMIT_REDIS_UNAVAILABLE (rate-limit path, fail-CLOSED 429). + Lumping both into ``NullRunBudgetError`` conflated the two and + hid rate-limit-Redis outages from operators. Pin: the literal + mapping must be gone (replaced by per-path branches elsewhere). + +- CLEANUP-BUDGET-THROTTLE-ZOMBIE — ``NullRunBudgetThrottleError`` + (NR-B007) was a zombie exception class. ``runtime.py:2116`` + raises ``WorkflowPausedException`` on ``decision == "throttle"``, + so ``NullRunBudgetThrottleError`` never fired on any wire or + runtime path. Catalog + class removed to keep the exception + surface in sync with what the SDK actually raises. + +- CLEANUP-WORKFLOW-KILLED-DISCOVERABILITY — + ``NullRunWorkflowKilledError`` was the only kill-related typed + exception missing from ``__init__._LAZY_EXPORTS`` and + ``__init__.__all__``. Asymmetric with ``WorkflowKilledInterrupt`` + which was already exported. Pin: must be importable from top + level ``nullrun``. +""" + +from __future__ import annotations + +import os +import re + +RUNTIME_PATH = os.path.join( + os.path.dirname(__file__), "..", "src", "nullrun", "runtime.py" +) +TRANSPORT_PATH = os.path.join( + os.path.dirname(__file__), "..", "src", "nullrun", "transport.py" +) +EXCEPTIONS_PATH = os.path.join( + os.path.dirname(__file__), "..", "src", "nullrun", "breaker", "exceptions.py" +) +MESSAGES_PATH = os.path.join( + os.path.dirname(__file__), "..", "src", "nullrun", "messages.py" +) +INIT_PATH = os.path.join( + os.path.dirname(__file__), "..", "src", "nullrun", "__init__.py" +) + + +def _read(path: str) -> str: + return open(path, encoding="utf-8").read() + + +def _strip_comment_lines(src: str) -> str: + """Drop ``#`` comment lines so source-pin tests checking for + forbidden user-facing wording don't trip on rationale comments + that legitimately mention the same word or code.""" + return "\n".join( + line for line in src.splitlines() if not line.lstrip().startswith("#") + ) + + +# ─── CLEANUP-PARSE-CODE-COLLISION ────────────────────────────────────────── + + +def test_safe_json_uses_nr_t_parse_not_nr_t001(): + """``_safe_json`` must raise with ``error_code="NR-T-PARSE"``. + + Pre-cleanup, the literal was ``NR-T001``, which collides with + ``NullRunToolBlockedError.error_code`` (breaker/exceptions.py). + Cookbook handlers that branch on ``exc.error_code == "NR-T001"`` + mis-classified a JSON parse failure as a tool block. The + post-cleanup literal NR-T-PARSE is dedicated transport-class + code; matches NR-T (transport) vocabulary without colliding + with NR-T001 / NR-T002 / etc. + """ + src = _read(TRANSPORT_PATH) + assert 'error_code="NR-T-PARSE"' in src, ( + "transport.py:_safe_json must raise with " + "error_code='NR-T-PARSE' (avoids collision with NR-T001 / " + "NullRunToolBlockedError)" + ) + # Negative pin: the pre-cleanup literal must NOT appear + # anywhere in transport.py. Code-only strip isn't necessary + # here — the literal is always inside a string in production + # code (assignments to ``error_code=``), never inside a comment + # that explains the cleanup (we have inline rationale elsewhere + # that mentions the old code, but as prose not a string literal). + code_only = _strip_comment_lines(src) + assert 'error_code="NR-T001"' not in code_only, ( + "transport.py must NOT contain the pre-cleanup literal " + "error_code='NR-T001' — that code belongs to " + "NullRunToolBlockedError (breaker/exceptions.py). The cleanup " + "renamed the transport-side literal to NR-T-PARSE." + ) + + +# ─── CLEANUP-REDIS-UNAVAILABLE-CODE ─────────────────────────────────────── + + +def test_transport_no_longer_maps_redis_unavailable_to_budget_error(): + """``transport.py`` must NOT have a blanket mapping from + ``REDIS_UNAVAILABLE`` to ``NullRunBudgetError``. + + Pre-cleanup, the mapping was ``"REDIS_UNAVAILABLE": NullRunBudgetError``. + That's wrong because the backend distinguishes: + - BUDGET_REDIS_UNAVAILABLE (budget path, fail-CLOSED 402) + - RATE_LIMIT_REDIS_UNAVAILABLE (rate-limit path, fail-CLOSED 429) + Lumping both into ``NullRunBudgetError`` conflated the two and + hid rate-limit-Redis outages from operators. + + Post-cleanup the literal mapping is removed; per-path branches + elsewhere in transport.py route the correct code to the right + typed exception (e.g. ``NullRunBudgetRedisError`` / ``NullRunRateLimitRedisError``). + """ + src = _read(TRANSPORT_PATH) + code_only = _strip_comment_lines(src) + assert '"REDIS_UNAVAILABLE": NullRunBudgetError' not in code_only, ( + "transport.py must not contain the pre-cleanup literal " + "'REDIS_UNAVAILABLE': NullRunBudgetError. The blanket mapping " + "conflated BUDGET_REDIS_UNAVAILABLE (budget path) with " + "RATE_LIMIT_REDIS_UNAVAILABLE (rate-limit path); per-path " + "branches must route each to its own typed exception." + ) + # Negative pin: also confirm the dict entry style (with single + # quotes variant) isn't sneaking in via formatter churn. + assert "'REDIS_UNAVAILABLE': NullRunBudgetError" not in code_only, ( + "transport.py must not contain the single-quote variant of " + "the pre-cleanup REDIS_UNAVAILABLE mapping." + ) + + +# ─── CLEANUP-BUDGET-THROTTLE-ZOMBIE ─────────────────────────────────────── + + +def test_null_run_budget_throttle_error_class_is_removed(): + """``NullRunBudgetThrottleError`` (NR-B007) must be removed from + ``breaker/exceptions.py``. + + Pre-cleanup this class existed with ``error_code = "NR-B007"`` and + ``retryable = True``. But ``runtime.py:2116`` raises + ``WorkflowPausedException`` on ``decision == "throttle"``, so + the class never fired on any wire or runtime path — a zombie + exception that the catalog had to maintain anyway. + + Post-cleanup: class removed, NR-B007 catalog entry removed, + test_format_user_message_handles_budget_throttle removed. + """ + src = _read(EXCEPTIONS_PATH) + code_only = _strip_comment_lines(src) + assert "class NullRunBudgetThrottleError" not in code_only, ( + "breaker/exceptions.py must NOT define NullRunBudgetThrottleError " + "— it was a zombie class never raised on a wire or runtime path. " + "See CLEANUP-BUDGET-THROTTLE-ZOMBIE." + ) + # Negative pin: the class's error_code literal must also be gone. + # (Other classes may still reference the string "NR-B007" in + # comments or string formatting, but no `error_code = "NR-B007"` + # class attribute on a NullRun*Error subclass should remain.) + assert re.search( + r"error_code\s*=\s*[\"']NR-B007[\"']", + code_only, + ) is None, ( + "No exception class in breaker/exceptions.py may declare " + "error_code='NR-B007' — NullRunBudgetThrottleError was removed " + "and the code is reserved-but-unused." + ) + + +def test_messages_catalog_no_longer_has_nr_b007_entry(): + """``messages.DEFAULT_MESSAGES`` must NOT contain a NR-B007 entry. + + Companion to ``test_null_run_budget_throttle_error_class_is_removed``: + the catalog entry for NR-B007 must be removed in lockstep so the + formatter doesn't return a stale message for a code that's no + longer raised. + """ + src = _read(MESSAGES_PATH) + code_only = _strip_comment_lines(src) + # The catalog uses dict-literal style: "NR-B007": "...", ... + # A bare re.search for the string key catches any formatting + # variant (single quote, trailing comma, etc.). + assert re.search(r"[\"']NR-B007[\"']\s*:", code_only) is None, ( + "messages.py DEFAULT_MESSAGES must NOT contain a key for " + "NR-B007 — the corresponding exception class was removed " + "and a stale catalog entry would mislead operators." + ) + + +def test_test_messages_no_longer_has_budget_throttle_test(): + """``tests/test_messages.py`` must NOT have a + ``test_format_user_message_handles_budget_throttle`` test. + + Companion cleanup: the orphan test for the zombie class was + removed along with the class. Pin the removal so a copy-paste + doesn't restore the test for a class that no longer exists. + """ + test_path = os.path.join( + os.path.dirname(__file__), "test_messages.py" + ) + src = _read(test_path) + code_only = _strip_comment_lines(src) + assert "test_format_user_message_handles_budget_throttle" not in code_only, ( + "tests/test_messages.py must not contain " + "test_format_user_message_handles_budget_throttle — the " + "orphan test for the zombie NullRunBudgetThrottleError was " + "removed when the class was deleted." + ) + + +# ─── CLEANUP-WORKFLOW-KILLED-DISCOVERABILITY ────────────────────────────── + + +def test_null_run_workflow_killed_error_importable_from_top_level(): + """``NullRunWorkflowKilledError`` must be importable from the + top-level ``nullrun`` namespace. + + Pre-cleanup, this class was the only kill-related typed + exception missing from ``__init__._LAZY_EXPORTS`` and + ``__init__.__all__``. Asymmetric with ``WorkflowKilledInterrupt`` + (already exported). Pin the top-level discoverability so host + code can ``from nullrun import NullRunWorkflowKilledError`` + alongside the other typed kill exceptions. + """ + src = _read(INIT_PATH) + # Pin: the class must appear in the _LAZY_EXPORTS dict + # (lazy export via __getattr__). The lazy-export form is the + # post-0.15.0 pattern; the older eager `from nullrun.breaker.X + # import ...` form has been migrated to lazy across the SDK. + assert re.search( + r"[\"']NullRunWorkflowKilledError[\"']\s*:\s*\(", + src, + ), ( + "nullrun/__init__.py _LAZY_EXPORTS must register " + "NullRunWorkflowKilledError for top-level import. Pre-cleanup " + "the class was discoverable only via " + "nullrun.breaker.exceptions, asymmetric with " + "WorkflowKilledInterrupt." + ) + # Pin: the class must also appear in __all__ so tab-completion + # surfaces it via dir(nullrun). + assert re.search( + r"__all__\s*=\s*\[[\s\S]*?[\"']NullRunWorkflowKilledError[\"']", + src, + ), ( + "nullrun/__init__.py __all__ must include " + "'NullRunWorkflowKilledError' for dir(nullrun) tab-completion " + "to surface the class." + ) + + +# ─── Behavioural smoke tests ────────────────────────────────────────────── + + +def test_null_run_budget_throttle_error_no_longer_importable(): + """Runtime check: importing the zombie class must fail. + + Companion to the source-pin test above — verifies the class is + actually gone from the live module, not just that the literal + text was deleted from the source file (someone could delete + the text but leave an aliased re-export, for example). + """ + import nullrun + import nullrun.breaker.exceptions as exc_mod + + assert not hasattr(exc_mod, "NullRunBudgetThrottleError"), ( + "nullrun.breaker.exceptions.NullRunBudgetThrottleError must " + "be removed at runtime; hasattr returning True means the " + "class survived the cleanup." + ) + # Also: must not be re-exported from top-level nullrun + assert not hasattr(nullrun, "NullRunBudgetThrottleError"), ( + "nullrun.NullRunBudgetThrottleError must NOT exist (top-level " + "re-export of the zombie class would defeat the cleanup)." + ) + + +def test_null_run_workflow_killed_error_importable_at_runtime(): + """Runtime check: ``from nullrun import NullRunWorkflowKilledError`` + must succeed. + """ + import nullrun + from nullrun.breaker import exceptions as exc_mod + + # Top-level import path (the new discoverability surface) + top_level_cls = getattr(nullrun, "NullRunWorkflowKilledError", None) + assert top_level_cls is not None, ( + "nullrun.NullRunWorkflowKilledError must be importable " + "from the top-level namespace after the CLEANUP-WORKFLOW-" + "KILLED-DISCOVERABILITY fix." + ) + # Must be the same class object as the one in breaker.exceptions + # (not a separate wrapper or proxy class). + assert top_level_cls is exc_mod.NullRunWorkflowKilledError, ( + "nullrun.NullRunWorkflowKilledError must be the same class " + "object as nullrun.breaker.exceptions.NullRunWorkflowKilledError; " + "a separate proxy class would defeat isinstance() checks " + "across the codebase." + ) diff --git a/tests/test_messages.py b/tests/test_messages.py index b7c0e3a..4519f7c 100644 --- a/tests/test_messages.py +++ b/tests/test_messages.py @@ -42,7 +42,11 @@ "NR-B004", "NR-B005", "NR-B006", - "NR-B007", + # NR-B007 removed 2026-09-10: NullRunBudgetThrottleError was a + # zombie class never raised on a wire or runtime path + # (runtime.py:2116 raises WorkflowPausedException on + # decision=="throttle"). Catalog entry + exception class removed + # to keep messages in sync with the exception module. "NR-CH001", "NR-C000", "NR-EX01", @@ -329,17 +333,6 @@ def test_format_user_message_handles_budget_recheck_failed(): assert out == messages.DEFAULT_MESSAGES["NR-B006"] -def test_format_user_message_handles_budget_throttle(): - """NR-B007: workflow throttle (soft budget signal — pacing, not - cap). Distinct from NR-B004 (hard cap).""" - throttle = exc.NullRunBudgetThrottleError( - workflow_id="wf-1", reason="throttle" - ) - assert throttle.error_code == "NR-B007" - out = messages.format_user_message(throttle) - assert out == messages.DEFAULT_MESSAGES["NR-B007"] - - def test_format_user_message_handles_consume_overbudget(): """NR-O001: consume > reserve + ε tolerance (ADR-005 invariant; SDK rejects rather than silently re-reserving).""" diff --git a/tests/test_v3_wire_contract.py b/tests/test_v3_wire_contract.py index 7c4db46..38cc023 100644 --- a/tests/test_v3_wire_contract.py +++ b/tests/test_v3_wire_contract.py @@ -430,14 +430,14 @@ def test_budget_hard_blocked_maps_to_budget_error(self): exc = _parse_v3_error_envelope(resp, "check") assert isinstance(exc, NullRunBudgetError) - def test_redis_unavailable_maps_to_budget_error(self): - #: REDIS_UNAVAILABLE is fail-CLOSED → 402 - resp = self._make_response( - 402, - {"error_code": "REDIS_UNAVAILABLE", "error_message": "Redis down"}, - ) - exc = _parse_v3_error_envelope(resp, "check") - assert isinstance(exc, NullRunBudgetError) + # REMOVED 2026-09-10: ``test_redis_unavailable_maps_to_budget_error`` + # — the legacy v2 ``REDIS_UNAVAILABLE`` slug is absent from the + # backend ``GateErrorCode::all()`` set and is never emitted on the + # wire. The corresponding SDK map entry in transport.py + # (``_V3_ERROR_CODE_MAP``) was removed in lockstep so cookbook + # mapping tables stay in sync with what the backend can actually + # send. BUDGET_REDIS_UNAVAILABLE / RATE_LIMIT_REDIS_UNAVAILABLE + # below are the post-v3.36 canonical codes and remain mapped. def test_chain_max_duration_maps_to_chain_error(self): resp = self._make_response( @@ -599,7 +599,10 @@ def test_catalog_covers_all_documented_codes(self): "BUDGET_SOFT_BLOCKED", "BUDGET_OVERDRAFT_EXCEEDED", "BUDGET_PERIOD_NOT_STARTED", - "REDIS_UNAVAILABLE", + # REDIS_UNAVAILABLE removed 2026-09-10 — never emitted + # by the backend (absent from GateErrorCode::all()). + # Use BUDGET_REDIS_UNAVAILABLE / RATE_LIMIT_REDIS_UNAVAILABLE + # (canonical post-v3.36) instead. "CHAIN_MAX_DURATION_EXCEEDED", "CHAIN_CROSS_ORG", "CHAIN_ORG_MISMATCH", From d4e83622a37cea518a5e5d25c5d5bdea91a69af3 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Thu, 10 Sep 2026 21:58:44 +0400 Subject: [PATCH 13/16] fix transport v3 --- src/nullrun/__init__.py | 20 + src/nullrun/breaker/exceptions.py | 101 +++++ src/nullrun/messages.py | 13 + src/nullrun/transport.py | 35 +- .../test_2026_09_10_mcp_umbrella_symmetry.py | 364 ++++++++++++++++++ tests/test_messages.py | 4 + 6 files changed, 531 insertions(+), 6 deletions(-) create mode 100644 tests/test_2026_09_10_mcp_umbrella_symmetry.py diff --git a/src/nullrun/__init__.py b/src/nullrun/__init__.py index f5934e6..aa7bdd9 100644 --- a/src/nullrun/__init__.py +++ b/src/nullrun/__init__.py @@ -500,6 +500,18 @@ def my_agent: # base-interrupt form (mro-aware dispatch). The class lives at # breaker/exceptions.py:1459. "NullRunWorkflowKilledError": ("nullrun.breaker.exceptions", "NullRunWorkflowKilledError"), + # ── B.1 (2026-09-10): MCP umbrella + APPROVAL_DB symmetry. + # Four typed exception classes that round-trip the MCP umbrella + # codes (ADR-013, frozen-dormant) and the six APPROVAL_DB_* + # sibling codes (DEF-ARFLOW-TOOLNAME-01). Pre-B.1 these all + # collapsed to NullRunBlockedException + the generic NR-X001 + # fallback — cookbook code couldn't branch on the typed arm. + # Post-B.1 each maps to its own typed class so + # ``except NullRunMcpDestructiveBlockedError:`` etc. work. + "NullRunMcpDestructiveBlockedError": ("nullrun.breaker.exceptions", "NullRunMcpDestructiveBlockedError"), + "NullRunMcpReadonlyBypassBlockedError": ("nullrun.breaker.exceptions", "NullRunMcpReadonlyBypassBlockedError"), + "NullRunMcpApprovalRequiredError": ("nullrun.breaker.exceptions", "NullRunMcpApprovalRequiredError"), + "NullRunApprovalDbUnavailableError": ("nullrun.breaker.exceptions", "NullRunApprovalDbUnavailableError"), # User-facing message catalog (NULLRUN owns the wording; see # nullrun/messages.py for the design rationale). Eager in # spirit — these are the "give the user a chance" surface that @@ -613,6 +625,14 @@ def __dir__() -> list[str]: "NullRunToolBlockedError", "WorkflowKilledInterrupt", "NullRunWorkflowKilledError", + # B.1 (2026-09-10): MCP umbrella + APPROVAL_DB symmetry. The + # four typed exception classes are part of the curated public + # surface — cookbook code branches on them by name, so they + # need to be visible in ``dir(nullrun)`` for tab-completion. + "NullRunMcpDestructiveBlockedError", + "NullRunMcpReadonlyBypassBlockedError", + "NullRunMcpApprovalRequiredError", + "NullRunApprovalDbUnavailableError", # User-facing message catalog — the single entry point for # turning an SDK exception into a string safe to display to # end users. ``set_user_message`` lets a deployment brand its diff --git a/src/nullrun/breaker/exceptions.py b/src/nullrun/breaker/exceptions.py index f1e404c..0963ea7 100644 --- a/src/nullrun/breaker/exceptions.py +++ b/src/nullrun/breaker/exceptions.py @@ -1220,6 +1220,107 @@ class NullRunApprovalToolDigestMismatchError(NullRunBlockedException): retryable = False +# ──────────────────────────────────────────────────────────────────────── +# MCP umbrella codes (ADR-013, 2026-08-14, frozen-dormant per Phase B.1) +# +# Pre-B.1 these three wire codes (``MCP_DESTRUCTIVE_BLOCKED``, +# ``MCP_READONLY_BYPASS_BLOCKED``, ``MCP_APPROVAL_REQUIRED``) all +# mapped to the base ``NullRunBlockedException`` in +# ``transport.py:_V3_ERROR_CODE_MAP`` — every cookbook handler that +# tried to branch on the typed MCP outcome silently fell through to +# the generic arm. The post-B.1 fix introduces three typed exception +# subclasses so cookbook code can ``except NullRunMcpDestructiveBlockedError:`` +# (etc.) and surface the right user_action verb. +# +# ADR-013 (2026-08-14) marks the umbrella as **frozen-dormant** — +# the underlying ``mcp_destructive_policy`` / ``mcp_readonly_bypass`` +# mechanisms are not currently wired in production but the wire codes +# are reserved and the SDK must round-trip them so a future enablement +# doesn't require SDK-side migration. +# ──────────────────────────────────────────────────────────────────────── + + +class NullRunMcpDestructiveBlockedError(NullRunBlockedException): + """Destructive MCP tool blocked by the mcp_destructive_policy umbrella. + + Wire code ``MCP_DESTRUCTIVE_BLOCKED`` (HTTP 403). ADR-013 — the + SDK maps it to a typed class so cookbook code can distinguish + destructive-MCP blocks from the generic block fallback (NR-X001) + or from the read-only bypass path (different operator-side + fix path). + """ + + error_code = "NR-MCP01" + user_action = ( + "The MCP tool's destructive capability is blocked by the " + "mcp_destructive_policy umbrella. Either remove the " + "destructive flag from the tool declaration or update the " + "workflow's policy to allow this destructive capability." + ) + retryable = False + + +class NullRunMcpReadonlyBypassBlockedError(NullRunBlockedException): + """Read-only MCP tool blocked because the bypass path is closed. + + Wire code ``MCP_READONLY_BYPASS_BLOCKED`` (HTTP 403). ADR-013 — + the SDK maps it to a typed class so cookbook code can distinguish + the readonly-bypass block (where the operator's intent was to + avoid destructive checks but the umbrella closed that path) + from the generic block fallback. + """ + + error_code = "NR-MCP02" + user_action = ( + "The MCP tool's read-only bypass path is blocked by the " + "mcp_readonly_bypass policy umbrella. The tool must go " + "through full destructive-MCP evaluation." + ) + retryable = False + + +class NullRunMcpApprovalRequiredError(NullRunBlockedException): + """MCP tool requires operator approval (NR-A010 equivalent for MCP). + + Wire code ``MCP_APPROVAL_REQUIRED`` (HTTP 403). Sibling to + :class:`NullRunApprovalNotYetApprovedError` but for the MCP + umbrella path — distinct so cookbook code can show a different + user_action hint (\"operator needs to approve the MCP tool's + capability\" vs \"operator has not yet decided on the workflow\"). + """ + + error_code = "NR-MCP03" + user_action = ( + "The MCP tool requires operator approval. Wait for the " + "operator to approve the tool's capability surface or use " + "a non-MCP equivalent." + ) + retryable = True + + +class NullRunApprovalDbUnavailableError(NullRunBlockedException): + """Approval database (Postgres) unavailable on the create-or-update path. + + Wire codes ``APPROVAL_DB_UNAVAILABLE``, ``APPROVAL_PERSISTENCE_FAILED``, + ``APPROVAL_VALIDATION_FAILED``, ``APPROVAL_CONFLICT``, + ``APPROVAL_NOT_FOUND``, ``APPROVAL_CREATE_FAILED`` (HTTP 402/403/503). + Pre-B.1 these all mapped to the base ``NullRunBlockedException`` + (transport.py:2984-2989) — operators couldn't tell apart a + transient DB outage (retryable) from a validation failure + (terminal). Post-B.1 they map to this single typed class so + cookbook code can ``except NullRunApprovalDbUnavailableError:`` + and surface the right remediation hint. + """ + + error_code = "NR-A016" + user_action = ( + "Approval database unavailable or rejected the request. " + "Retry shortly (transient DB outage) or contact support if " + "the failure persists (validation/conflict)." + ) + retryable = True + + # NOTE: NullRunApprovalReplayRejectedError was moved earlier in this # module (alongside the other five approval exceptions) so all six # typed approval exceptions are co-located. The earlier definition diff --git a/src/nullrun/messages.py b/src/nullrun/messages.py index 7f7d094..134cd8f 100644 --- a/src/nullrun/messages.py +++ b/src/nullrun/messages.py @@ -156,6 +156,19 @@ # copy is generic because the cause is operator-side accounting; # user should retry (a fresh /gate will recompute the reservation). "NR-O001": "Your request couldn't be completed due to a usage accounting discrepancy. Please try again.", + # ── B.1 (2026-09-10): MCP umbrella + APPROVAL_DB typed arms. + # Three MCP umbrella codes (ADR-013, frozen-dormant) and the + # single NR-A016 typed class for the six APPROVAL_DB_* sibling + # codes. NR-A016 wording is intentionally close to the generic + # "transient service outage" cluster — the cookbook recipe for + # the typed class branches on retryable vs terminal, not on + # the specific DB cause (operators don't care whether it was a + # validation failure or a Postgres connection drop; both are + # "retry shortly, contact support if persistent"). + "NR-MCP01": "That action isn't available right now. Please contact support if you need it.", + "NR-MCP02": "That action isn't available right now. Please contact support if you need it.", + "NR-MCP03": "Your request is awaiting approval. Please wait a moment while it's being reviewed.", + "NR-A016": "Your request couldn't be completed. Please try again shortly.", # ---- Wire / protocol ---------------------------------------------------- # NR-P001: SDK wire-protocol version is below the backend's # ``X-NULLRUN-PROTOCOL:`` minimum. End-user action is "contact diff --git a/src/nullrun/transport.py b/src/nullrun/transport.py index bcafa89..ebc8602 100644 --- a/src/nullrun/transport.py +++ b/src/nullrun/transport.py @@ -28,6 +28,7 @@ from nullrun.breaker.exceptions import ( BreakerTransportError, InsecureTransportError, + NullRunApprovalDbUnavailableError, NullRunApprovalReplayRejectedError, NullRunAuthenticationError, NullRunBackendError, @@ -35,6 +36,9 @@ NullRunDecision, NullRunExecutionNotFoundError, NullRunInfrastructureError, + NullRunMcpApprovalRequiredError, + NullRunMcpDestructiveBlockedError, + NullRunMcpReadonlyBypassBlockedError, NullRunTransportError, RateLimitError, TransportErrorSource, @@ -2981,12 +2985,18 @@ def _build_v3_error_code_map() -> dict[str, type[Exception]]: "RATE_LIMIT_REDIS_UNAVAILABLE": NullRunRateLimitRedisError, "BUDGET_DATA_UNAVAILABLE": NullRunBackendError, # 402 — approval-create failure family (DEF-ARFLOW-TOOLNAME-01, - "APPROVAL_DB_UNAVAILABLE": NullRunBlockedException, - "APPROVAL_PERSISTENCE_FAILED": NullRunBlockedException, - "APPROVAL_VALIDATION_FAILED": NullRunBlockedException, - "APPROVAL_CONFLICT": NullRunBlockedException, - "APPROVAL_NOT_FOUND": NullRunBlockedException, - "APPROVAL_CREATE_FAILED": NullRunBlockedException, + # B.1 symmetry fix 2026-09-10): six sibling codes all map to + # the typed ``NullRunApprovalDbUnavailableError`` (NR-A016) so + # cookbook code can branch on the typed class instead of + # falling through to the base NullRunBlockedException. Pre-B.1 + # all six collapsed to the base class — operators couldn't tell + # apart a transient DB outage from a validation failure. + "APPROVAL_DB_UNAVAILABLE": NullRunApprovalDbUnavailableError, + "APPROVAL_PERSISTENCE_FAILED": NullRunApprovalDbUnavailableError, + "APPROVAL_VALIDATION_FAILED": NullRunApprovalDbUnavailableError, + "APPROVAL_CONFLICT": NullRunApprovalDbUnavailableError, + "APPROVAL_NOT_FOUND": NullRunApprovalDbUnavailableError, + "APPROVAL_CREATE_FAILED": NullRunApprovalDbUnavailableError, # 403 — approval grant-consume outcomes (v3.53 / 2026-08-13 # audit, A-1+A-2 bundle). Distinct from the /gate # create-failure family above: these are the seven @@ -3057,6 +3067,19 @@ def _build_v3_error_code_map() -> dict[str, type[Exception]]: "TOO_MANY_PENDING_APPROVALS": NullRunBlockedException, "BUSINESS_IMPACT_INVALID": NullRunBlockedException, "VALIDATION_FAILED": NullRunBlockedException, + # ── MCP umbrella codes (ADR-013, 2026-08-14, frozen-dormant) + # B.1 (2026-09-10): the three umbrella codes map to typed + # ``NullRunMcp*Error`` subclasses so cookbook code can branch + # on the precise umbrella path. Pre-B.1 these collapsed to + # the generic NullRunBlockedException / NR-X001 fallback — + # operators couldn't distinguish the destructive-MCP block + # from the readonly-bypass block from the approval-required + # path. ADR-013 marked the umbrella frozen-dormant: wire + # codes are reserved and the SDK must round-trip them, but + # the underlying mechanisms aren't wired in production yet. + "MCP_DESTRUCTIVE_BLOCKED": NullRunMcpDestructiveBlockedError, + "MCP_READONLY_BYPASS_BLOCKED": NullRunMcpReadonlyBypassBlockedError, + "MCP_APPROVAL_REQUIRED": NullRunMcpApprovalRequiredError, # Wire-level parsing failures (missing / malformed fields). # Map to ``NullRunBackendError`` because the SDK treats them # as infrastructure-side issues — the server should have diff --git a/tests/test_2026_09_10_mcp_umbrella_symmetry.py b/tests/test_2026_09_10_mcp_umbrella_symmetry.py new file mode 100644 index 0000000..db260a8 --- /dev/null +++ b/tests/test_2026_09_10_mcp_umbrella_symmetry.py @@ -0,0 +1,364 @@ +"""Source-pin + behavioural regression tests for SDK B.1 (2026-09-10). + +B.1 closed two wire-class gaps that were silently dropping typed +information to the generic ``NullRunBlockedException`` / NR-X001 +fallback: + +1. **MCP umbrella codes** (ADR-013, 2026-08-14, frozen-dormant) — + ``MCP_DESTRUCTIVE_BLOCKED``, ``MCP_READONLY_BYPASS_BLOCKED``, + ``MCP_APPROVAL_REQUIRED`` previously all collapsed to + ``NullRunBlockedException``. Cookbook code that wanted to + ``except NullRunMcpDestructiveBlockedError:`` etc. fell through + to the generic arm and surfaced ``FALLBACK_MESSAGE = "Something + went wrong. Please try again."`` — the very bug the langgraph + demo test surfaced in 2026-09-08. + +2. **APPROVAL_DB_* sibling family** — six codes (DB_UNAVAILABLE, + PERSISTENCE_FAILED, VALIDATION_FAILED, CONFLICT, NOT_FOUND, + CREATE_FAILED) previously all collapsed to the base + ``NullRunBlockedException``. Operators couldn't tell apart a + transient DB outage from a validation failure. Post-B.1 they + all map to the single typed ``NullRunApprovalDbUnavailableError`` + so cookbook code can branch on the typed class. + +Each test pins the source surface so a future refactor that +re-introduces the bug (e.g. drops one of the typed mappings) is +caught at test time, not in production. +""" + +from __future__ import annotations + +import os + +EXCEPTIONS_PATH = os.path.join( + os.path.dirname(__file__), "..", "src", "nullrun", "breaker", "exceptions.py" +) +TRANSPORT_PATH = os.path.join( + os.path.dirname(__file__), "..", "src", "nullrun", "transport.py" +) +INIT_PATH = os.path.join( + os.path.dirname(__file__), "..", "src", "nullrun", "__init__.py" +) +MESSAGES_PATH = os.path.join( + os.path.dirname(__file__), "..", "src", "nullrun", "messages.py" +) + + +def _read(path: str) -> str: + return open(path, encoding="utf-8").read() + + +def _strip_comment_lines(src: str) -> str: + return "\n".join( + line for line in src.splitlines() if not line.lstrip().startswith("#") + ) + + +# ─── Typed exception class definitions (MCP umbrella) ──────────────────── + + +def test_mcp_destructive_blocked_error_class_defined(): + """``breaker/exceptions.py`` must define ``NullRunMcpDestructiveBlockedError``.""" + src = _read(EXCEPTIONS_PATH) + code_only = _strip_comment_lines(src) + assert "class NullRunMcpDestructiveBlockedError" in code_only, ( + "B.1: NullRunMcpDestructiveBlockedError must be defined in " + "breaker/exceptions.py — typed class for MCP_DESTRUCTIVE_BLOCKED" + ) + + +def test_mcp_readonly_bypass_blocked_error_class_defined(): + """``breaker/exceptions.py`` must define ``NullRunMcpReadonlyBypassBlockedError``.""" + src = _read(EXCEPTIONS_PATH) + code_only = _strip_comment_lines(src) + assert "class NullRunMcpReadonlyBypassBlockedError" in code_only, ( + "B.1: NullRunMcpReadonlyBypassBlockedError must be defined in " + "breaker/exceptions.py — typed class for MCP_READONLY_BYPASS_BLOCKED" + ) + + +def test_mcp_approval_required_error_class_defined(): + """``breaker/exceptions.py`` must define ``NullRunMcpApprovalRequiredError``.""" + src = _read(EXCEPTIONS_PATH) + code_only = _strip_comment_lines(src) + assert "class NullRunMcpApprovalRequiredError" in code_only, ( + "B.1: NullRunMcpApprovalRequiredError must be defined in " + "breaker/exceptions.py — typed class for MCP_APPROVAL_REQUIRED" + ) + + +def test_approval_db_unavailable_error_class_defined(): + """``breaker/exceptions.py`` must define ``NullRunApprovalDbUnavailableError``.""" + src = _read(EXCEPTIONS_PATH) + code_only = _strip_comment_lines(src) + assert "class NullRunApprovalDbUnavailableError" in code_only, ( + "B.1: NullRunApprovalDbUnavailableError must be defined in " + "breaker/exceptions.py — typed class for the six " + "APPROVAL_DB_* sibling codes" + ) + + +def test_mcp_classes_inherit_from_null_run_blocked_exception(): + """All four B.1 typed classes must inherit from + ``NullRunBlockedException`` (not ``NullRunError`` directly) so + cookbook ``except NullRunBlockedException:`` arms still match + via the MRO. This is the same pattern as the existing + NR-A010..NR-A015 approval classes.""" + src = _read(EXCEPTIONS_PATH) + code_only = _strip_comment_lines(src) + for cls in [ + "NullRunMcpDestructiveBlockedError", + "NullRunMcpReadonlyBypassBlockedError", + "NullRunMcpApprovalRequiredError", + "NullRunApprovalDbUnavailableError", + ]: + # Look for ``class (NullRunBlockedException):`` + assert f"class {cls}(NullRunBlockedException):" in code_only, ( + f"B.1: {cls} must inherit from NullRunBlockedException, " + f"not NullRunError directly. The MRO is the contract — " + f"cookbook ``except NullRunBlockedException:`` arms must " + f"still match the typed subclass." + ) + + +# ─── Transport mappings ────────────────────────────────────────────────── + + +def test_transport_maps_mcp_destructive_blocked_to_typed_class(): + """``transport.py:_V3_ERROR_CODE_MAP`` must map + ``MCP_DESTRUCTIVE_BLOCKED`` to ``NullRunMcpDestructiveBlockedError``, + NOT to the base ``NullRunBlockedException``.""" + src = _read(TRANSPORT_PATH) + code_only = _strip_comment_lines(src) + assert ( + '"MCP_DESTRUCTIVE_BLOCKED": NullRunMcpDestructiveBlockedError' + in code_only + ), ( + "B.1: transport.py must map MCP_DESTRUCTIVE_BLOCKED to the " + "typed NullRunMcpDestructiveBlockedError. Pre-B.1 the wire " + "code collapsed to NullRunBlockedException; cookbook code " + "branching on the typed class fell through to NR-X001." + ) + + +def test_transport_maps_mcp_readonly_bypass_blocked_to_typed_class(): + src = _read(TRANSPORT_PATH) + code_only = _strip_comment_lines(src) + assert ( + '"MCP_READONLY_BYPASS_BLOCKED": NullRunMcpReadonlyBypassBlockedError' + in code_only + ), ( + "B.1: transport.py must map MCP_READONLY_BYPASS_BLOCKED to " + "NullRunMcpReadonlyBypassBlockedError" + ) + + +def test_transport_maps_mcp_approval_required_to_typed_class(): + src = _read(TRANSPORT_PATH) + code_only = _strip_comment_lines(src) + assert ( + '"MCP_APPROVAL_REQUIRED": NullRunMcpApprovalRequiredError' + in code_only + ), ( + "B.1: transport.py must map MCP_APPROVAL_REQUIRED to " + "NullRunMcpApprovalRequiredError" + ) + + +def test_transport_maps_all_six_approval_db_codes_to_typed_class(): + """All six APPROVAL_DB_* sibling codes must map to the typed + ``NullRunApprovalDbUnavailableError``. Pre-B.1 they all collapsed + to ``NullRunBlockedException`` — operators couldn't tell apart + a transient DB outage from a validation failure.""" + src = _read(TRANSPORT_PATH) + code_only = _strip_comment_lines(src) + for code in [ + "APPROVAL_DB_UNAVAILABLE", + "APPROVAL_PERSISTENCE_FAILED", + "APPROVAL_VALIDATION_FAILED", + "APPROVAL_CONFLICT", + "APPROVAL_NOT_FOUND", + "APPROVAL_CREATE_FAILED", + ]: + assert ( + f'"{code}": NullRunApprovalDbUnavailableError' in code_only + ), ( + f"B.1: transport.py must map {code} to " + f"NullRunApprovalDbUnavailableError (typed). Pre-B.1 the " + f"wire code collapsed to NullRunBlockedException." + ) + + +# ─── Top-level discoverability (lazy exports + __all__) ───────────────── + + +def test_mcp_umbrella_classes_in_lazy_exports(): + """All three MCP umbrella classes must be importable from the + top-level ``nullrun`` namespace via ``_LAZY_EXPORTS``. Pre-B.1 + they didn't exist; post-B.1 the typed classes are part of the + public surface (cookbook recipes branch on them by name).""" + src = _read(INIT_PATH) + for cls in [ + "NullRunMcpDestructiveBlockedError", + "NullRunMcpReadonlyBypassBlockedError", + "NullRunMcpApprovalRequiredError", + ]: + assert f'"{cls}":' in src, ( + f"B.1: {cls} must appear in _LAZY_EXPORTS so cookbook " + f"code can ``from nullrun import {cls}``" + ) + + +def test_approval_db_unavailable_in_lazy_exports(): + src = _read(INIT_PATH) + assert '"NullRunApprovalDbUnavailableError":' in src, ( + "B.1: NullRunApprovalDbUnavailableError must appear in " + "_LAZY_EXPORTS" + ) + + +def test_mcp_umbrella_classes_in_all(): + """The four B.1 typed classes must appear in ``__all__`` so + tab-completion surfaces them via ``dir(nullrun)``. Cookbook + code that wants to ``except NullRunMcpDestructiveBlockedError:`` + needs to discover the class via ``dir(nullrun)`` first.""" + src = _read(INIT_PATH) + # Anchor on the __all__ block — naive substring search would + # false-positive on _LAZY_EXPORTS entries. + import re + match = re.search(r"__all__\s*=\s*\[(.*?)\]", src, re.DOTALL) + assert match is not None, "B.1: __all__ list must exist in __init__.py" + all_block = match.group(1) + for cls in [ + "NullRunMcpDestructiveBlockedError", + "NullRunMcpReadonlyBypassBlockedError", + "NullRunMcpApprovalRequiredError", + "NullRunApprovalDbUnavailableError", + ]: + assert f'"{cls}"' in all_block, ( + f"B.1: {cls} must appear in __all__ so tab-completion " + f"surfaces it via dir(nullrun)" + ) + + +# ─── Catalog completeness ──────────────────────────────────────────────── + + +def test_messages_catalog_has_mcp_and_approval_db_entries(): + """``messages.DEFAULT_MESSAGES`` must have entries for the four + new error codes (NR-MCP01, NR-MCP02, NR-MCP03, NR-A016). + Without these, ``format_user_message`` falls through to the + generic ``FALLBACK_MESSAGE = "Something went wrong. Please + try again."`` — the exact bug the langgraph approval demo + surfaced in 2026-09-08.""" + from nullrun import messages + + for code in ["NR-MCP01", "NR-MCP02", "NR-MCP03", "NR-A016"]: + assert code in messages.DEFAULT_MESSAGES, ( + f"B.1: messages.DEFAULT_MESSAGES must contain an entry " + f"for {code} (the typed class's error_code). Missing " + f"catalog entry means format_user_message returns the " + f"generic FALLBACK_MESSAGE for the typed class — " + f"defeats the whole point of the typed mapping." + ) + + +# ─── Behavioural smoke tests ───────────────────────────────────────────── + + +def test_typed_class_runtime_imports(): + """All four B.1 typed classes must import successfully from + both the breaker.exceptions module AND the top-level nullrun + namespace.""" + import nullrun + from nullrun.breaker import exceptions as exc_mod + + pairs = [ + ("NullRunMcpDestructiveBlockedError", exc_mod.NullRunMcpDestructiveBlockedError), + ("NullRunMcpReadonlyBypassBlockedError", exc_mod.NullRunMcpReadonlyBypassBlockedError), + ("NullRunMcpApprovalRequiredError", exc_mod.NullRunMcpApprovalRequiredError), + ("NullRunApprovalDbUnavailableError", exc_mod.NullRunApprovalDbUnavailableError), + ] + for name, breaker_cls in pairs: + top_level_cls = getattr(nullrun, name, None) + assert top_level_cls is not None, ( + f"B.1: nullrun.{name} must be importable (lazy export " + f"failed or class is missing)" + ) + assert top_level_cls is breaker_cls, ( + f"B.1: nullrun.{name} must be the same class object as " + f"nullrun.breaker.exceptions.{name} — a separate proxy " + f"class would defeat isinstance() checks across the " + f"codebase" + ) + + +def test_typed_class_isinstance_of_null_run_blocked_exception(): + """The four B.1 typed classes must be ``isinstance(..., NullRunBlockedException)``. + Cookbook ``except NullRunBlockedException:`` arms rely on this + MRO behaviour — if a future refactor changes the parent class + the cookbook handlers silently miss the typed arms.""" + from nullrun import NullRunBlockedException + from nullrun.breaker import exceptions as exc_mod + + for cls in [ + exc_mod.NullRunMcpDestructiveBlockedError, + exc_mod.NullRunMcpReadonlyBypassBlockedError, + exc_mod.NullRunMcpApprovalRequiredError, + exc_mod.NullRunApprovalDbUnavailableError, + ]: + # Instantiate with the minimum required kwargs (workflow_id, + # reason). The base __init__ accepts **details so additional + # typed kwargs can be forwarded from transport.py. + instance = cls(workflow_id="wf-1", reason="test") + assert isinstance(instance, NullRunBlockedException), ( + f"B.1: {cls.__name__} must be isinstance of " + f"NullRunBlockedException (MRO contract for cookbook " + f"handlers). Got MRO: {[c.__name__ for c in type(instance).__mro__]}" + ) + + +def test_format_user_message_returns_catalog_default_for_typed_classes(): + """``format_user_message`` on a B.1 typed class instance must + return the catalog default, NOT the FALLBACK_MESSAGE. This is + the actual user-facing bug B.1 closed — pre-B.1 the wire code + landed on the base class, the base class's error_code was a + SCREAMING_SNAKE backend string with no catalog entry, and + ``format_user_message`` fell through to FALLBACK_MESSAGE.""" + from nullrun import messages + from nullrun.breaker import exceptions as exc_mod + + cases = [ + ( + exc_mod.NullRunMcpDestructiveBlockedError("wf-1", "destructive blocked"), + "NR-MCP01", + ), + ( + exc_mod.NullRunMcpReadonlyBypassBlockedError("wf-1", "readonly bypass blocked"), + "NR-MCP02", + ), + ( + exc_mod.NullRunMcpApprovalRequiredError("wf-1", "mcp approval pending"), + "NR-MCP03", + ), + ( + exc_mod.NullRunApprovalDbUnavailableError("wf-1", "approval db down"), + "NR-A016", + ), + ] + for instance, expected_code in cases: + assert instance.error_code == expected_code, ( + f"B.1: {type(instance).__name__}.error_code must be " + f"{expected_code}, got {instance.error_code!r}" + ) + out = messages.format_user_message(instance) + assert out == messages.DEFAULT_MESSAGES[expected_code], ( + f"B.1: format_user_message on {type(instance).__name__} " + f"must return the catalog default for {expected_code}, " + f"not FALLBACK_MESSAGE. Got: {out!r}" + ) + assert out != messages.FALLBACK_MESSAGE, ( + f"B.1: {type(instance).__name__} fell through to " + f"FALLBACK_MESSAGE — catalog entry for {expected_code} " + f"is missing or wire mapping is wrong" + ) diff --git a/tests/test_messages.py b/tests/test_messages.py index 4519f7c..fc39e62 100644 --- a/tests/test_messages.py +++ b/tests/test_messages.py @@ -37,6 +37,7 @@ "NR-A013", "NR-A014", "NR-A015", + "NR-A016", # B.1 (2026-09-10): APPROVAL_DB_* sibling family "NR-B001", "NR-B002", "NR-B004", @@ -51,6 +52,9 @@ "NR-C000", "NR-EX01", "NR-L001", + "NR-MCP01", # B.1 (2026-09-10): MCP umbrella destructive + "NR-MCP02", # B.1 (2026-09-10): MCP umbrella readonly bypass + "NR-MCP03", # B.1 (2026-09-10): MCP umbrella approval required "NR-O001", "NR-P001", "NR-R001", From fa38990b6c876f64b50bff9e6276525a2f492826 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Thu, 10 Sep 2026 22:09:14 +0400 Subject: [PATCH 14/16] fix(tests): pin test_runtime catalog-code migration (DEF-NR-RUNTIME-BLOCK-TYPED) test_execute_blocked_surfaces_wire_error_code expected the wire SCREAMING_SNAKE code (APPROVAL_VALIDATION_FAILED) on exc.error_code, but the 2026-09-10 catalog migration now correctly returns the typed catalog code (NR-A016 / NullRunApprovalDbUnavailableError) per runtime._build_block_exception dispatch priority 1. Post-fix the SDK contract is: - exc.error_code == catalog code (NR-A016) for format_user_message - exc.details['details']['error_code'] == wire code (preserved verbatim) - exc.details['details']['mapped_class'] == typed class name (NullRunApprovalDbUnavailableError, NOT the base NullRunBlockedException) This test was the last stale wire-code assertion in test_runtime.py preventing full SDK test pass under the new catalog contract. --- tests/test_runtime.py | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/tests/test_runtime.py b/tests/test_runtime.py index 77f717c..286e986 100644 --- a/tests/test_runtime.py +++ b/tests/test_runtime.py @@ -193,6 +193,18 @@ def test_execute_blocked_surfaces_wire_error_code(self, make_runtime, mock_api): # "Approval infrastructure unavailable: validation error during # approval row creation" as the generic NR-X001 — the very # bug the journal test surfaced). + # + # Post-fix (2026-09-10 batch, ``_V3_ERROR_CODE_MAP`` dispatch): + # ``APPROVAL_VALIDATION_FAILED`` is now in the typed catalog + # map (transport.py:2996) and resolves to + # ``NullRunApprovalDbUnavailableError`` (catalog code NR-A016). + # The catalog code (NOT the wire SCREAMING_SNAKE string) is + # the canonical ``format_user_message`` lookup key — see + # ``runtime._build_block_exception`` docstring. The wire code + # is preserved verbatim on ``exc.details["details"]["error_code"]`` + # for routing/alerting. + from nullrun.breaker.exceptions import NullRunApprovalDbUnavailableError + respx.post(f"{BASE_URL}/api/v1/execute").mock( return_value=httpx.Response( 200, @@ -215,8 +227,11 @@ def test_execute_blocked_surfaces_wire_error_code(self, make_runtime, mock_api): rt = make_runtime() with pytest.raises(NullRunBlockedException) as exc_info: rt.execute(tool_name="refund_customer", input_data={}, mode="strict") - # The wire code wins — no keyword guessing. - assert exc_info.value.error_code == "APPROVAL_VALIDATION_FAILED" + # Catalog typed class wins — not the keyword-guessing fallback. + assert isinstance(exc_info.value, NullRunApprovalDbUnavailableError) + # The catalog error_code (NR-A016) is canonical for + # format_user_message; the wire code stays in details. + assert exc_info.value.error_code == "NR-A016" # The structured payload is preserved on details so a caller # can introspect ``decision_source`` for routing/alerting. # ``NullRunBlockedException.__init__`` wraps ``**details`` so @@ -224,9 +239,10 @@ def test_execute_blocked_surfaces_wire_error_code(self, make_runtime, mock_api): wire_details = exc_info.value.details.get("details") or {} assert wire_details.get("error_code") == "APPROVAL_VALIDATION_FAILED" assert wire_details.get("decision_source") == "approval_create_failed" - # Back-compat shim: the legacy ``mapped_class`` field is still - # populated so any caller that branched on it pre-fix keeps working. - assert wire_details.get("mapped_class") == "NullRunBlockedException" + # Back-compat shim: the legacy ``mapped_class`` field exposes + # the typed catalog class name (NR-A016 / NullRunApprovalDbUnavailableError) + # so any caller that branched on it pre-fix keeps working. + assert wire_details.get("mapped_class") == "NullRunApprovalDbUnavailableError" @pytest.mark.skip( reason=( From 4c2490e776a5787527a9eb009b27dafc2f754a5f Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Thu, 10 Sep 2026 23:04:52 +0400 Subject: [PATCH 15/16] chore: remove accidentally-committed scratch artifacts - dist_local/nullrun-0.16.7-py3-none-any.whl: pre-built wheel (305KB binary) committed in 0a52c96. dist/ is already gitignored but dist_local/ was missed; would otherwise ship in the 0.16.7 sdist. - src/nullrun/transport.py.defect37: 144KB / 3168-line debug scratch file committed in 25eb2c2; not referenced by any runtime code (grep confirms zero references in src/ or tests/). - .gitignore: added dist_local/ and src/**/*.defect* to prevent re-introduction. No code change; tests, mypy, ruff unaffected. --- .gitignore | 6 + src/nullrun/transport.py.defect37 | 3168 ----------------------------- 2 files changed, 6 insertions(+), 3168 deletions(-) delete mode 100644 src/nullrun/transport.py.defect37 diff --git a/.gitignore b/.gitignore index a8dc021..9b6eb16 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,7 @@ __pycache__/ build/ develop-eggs/ dist/ +dist_local/ downloads/ eggs/ .eggs/ @@ -73,3 +74,8 @@ docs/integration-baseline-2026-06-19.md audit.md docs/postman/ .hermes/ + +# Debug scratch artefacts dropped by ad-hoc defect sessions +# (mirrors the convention that `dist/` is ignored: built / scratch +# outputs never belong in VCS regardless of where they were created). +src/**/*.defect* diff --git a/src/nullrun/transport.py.defect37 b/src/nullrun/transport.py.defect37 deleted file mode 100644 index 8c485be..0000000 --- a/src/nullrun/transport.py.defect37 +++ /dev/null @@ -1,3168 +0,0 @@ -""" -Transport layer for NullRun SDK. - -Handles HTTP communication with batching and background flush. -Includes fallback modes for Gateway unavailability. -""" - -import hashlib -import hmac -import json -import logging -import os -import random -import tempfile -import threading -import time -import uuid -import weakref -from collections import OrderedDict -from collections.abc import Callable -from dataclasses import dataclass -from typing import TYPE_CHECKING, Any, cast - -import httpx - -from nullrun.actions import handle_action -from nullrun.breaker.circuit_breaker import CircuitBreaker -from nullrun.breaker.exceptions import ( - BreakerTransportError, - InsecureTransportError, - NullRunApprovalReplayRejectedError, - NullRunAuthenticationError, - NullRunBackendError, - NullRunBlockedException, - NullRunDecision, - NullRunExecutionNotFoundError, - NullRunInfrastructureError, - NullRunTransportError, - RateLimitError, - TransportErrorSource, -) -from nullrun.observability import metrics - -if TYPE_CHECKING: - # Forward-referenced to avoid transport.py ⇄ transport_websocket.py cycle. - from nullrun.transport_websocket import WebSocketConnection - -# OpenTelemetry imports (lazy-loaded to support optional dependency) -try: - from opentelemetry import trace - from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator - - _OTEL_AVAILABLE = True -except ImportError: - _OTEL_AVAILABLE = False - trace = None # type: ignore[assignment] - TraceContextTextMapPropagator = None # type: ignore[assignment] - -logger = logging.getLogger(__name__) - -__api_version__ = "1.0" - -# Wire-protocol version handshake. Backend rejects signed POSTs without -# `X-NULLRUN-PROTOCOL: ` with 400. Bump must be coordinated with backend -# `proxy::http::gate::protocol` and `/api/v1/capabilities`. -# -# v4 (2026-08-31, ADR-037 Slice B): ADDITIVE — /gate response now echoes -# the SDK-supplied `action_digest` and a `policy_hash` slot (always None -# today; Slice D wires per-request computation). Wire-additive: v3 SDKs -# parsing the response simply ignore the new fields; v4 SDKs parsing a -# v3 backend response see `None` on both (skip_serializing_if on the -# backend means the JSON keys are absent, not `null`). No new -# hashing/computation introduced on either side — both fields echo -# already-computed values. -NULLRUN_PROTOCOL_VERSION: int = 4 -HEADER_PROTOCOL: str = "X-NULLRUN-PROTOCOL" - - -def _protocol_header_value() -> str: - """Return the current wire-protocol version as a string (backend stores u32).""" - return str(NULLRUN_PROTOCOL_VERSION) - - -def _emit_for_transport_error( - err: BaseException, - stage: str, - correlation_id: str | None, - *, - status_code: int | None = None, -) -> None: - """Layer 2: fire the on_error hook for transport-level raises. Best-effort, never raises. - - The transport module is stateless, so context is minimal — just - ``stage`` + ``correlation_id`` + ``status_code``. - """ - from nullrun.observability.error_hooks import ( - ErrorContext, - emit_error, - has_hooks, - ) - - if not has_hooks(): - return - extra: dict[str, Any] = {} - if status_code is not None: - extra["status_code"] = status_code - emit_error( - err, - ErrorContext( - stage=stage, - correlation_id=correlation_id, - extra=extra, - ), - ) - - -# ============================================================================= -# HMAC Request Signing (Task 11) -# ============================================================================= - - -def generate_hmac_signature( - api_key: str, - secret_key: str, - timestamp: int, - body: str | bytes, -) -> str: - """ - Generate HMAC-SHA256 signature for request authentication. - - Signature = HMAC-SHA256(secret_key, timestamp + ":" + api_key + ":" + body_hash) - Body hash = SHA256(request_body) - """ - # Accept both ``str`` (legacy callers) and ``bytes`` (canonical wire form). - body_bytes = body.encode("utf-8") if isinstance(body, str) else body - body_hash = hashlib.sha256(body_bytes).hexdigest() - message = f"{timestamp}:{api_key}:{body_hash}" - - signature = hmac.new( - secret_key.encode("utf-8"), message.encode("utf-8"), hashlib.sha256 - ).hexdigest() - - return signature - - -def verify_hmac_signature( - api_key: str, - secret_key: str, - timestamp: int, - body: str | bytes, - signature: str, - max_age_seconds: int = 300, -) -> bool: - """ - Verify HMAC signature from request. - - Args: - api_key: Client's API key - secret_key: Client's secret key - timestamp: Unix timestamp from request - body: Request body as JSON string or UTF-8 bytes - signature: HMAC signature to verify - max_age_seconds: Maximum allowed age of request (default 5 min) - - Returns: - True if signature is valid and request is fresh - """ - # Check timestamp freshness - current_time = int(time.time()) - if abs(current_time - timestamp) > max_age_seconds: - # Separate counter so SRE can distinguish clock drift from forgeries. - try: - from nullrun.observability import metrics - - metrics.inc_transport("hmac_verify_expired_total") - except Exception: # noqa: BLE001 — best-effort counter - pass - logger.warning(f"Request timestamp too old: {timestamp} vs current {current_time}") - return False - - # Recompute expected signature - expected = generate_hmac_signature(api_key, secret_key, timestamp, body) - - # Constant-time comparison to prevent timing attacks - return hmac.compare_digest(expected, signature) - - -def _signed_request_body(payload: dict[str, Any]) -> bytes: - """Serialise a JSON payload to the canonical bytes the HMAC signature is computed over. - - All four signed POST call sites must serialise via this helper and pass - the result with ``content=body`` to httpx (NOT ``json=...`` — that - re-serialises with different separators and breaks the HMAC match). - ``default=str`` accepts Decimal / bytes / datetime / UUID. - """ - return json.dumps(payload, separators=(",", ":"), default=str).encode("utf-8") - - -# ============================================================================= -# Retry with exponential backoff + jitter -# ============================================================================= - - -def _retry_with_backoff( - func: Callable[[], Any], - max_retries: int = 10, - base_delay: float = 0.5, - max_delay: float = 30.0, - backoff_factor: float = 2.0, - jitter: float = 0.1, - last_retry_after_seconds: float = 0.0, - on_transport_error: str | Callable[[Exception], dict[str, Any]] | None = None, - retry_on_5xx: bool = False, -) -> Any: - """Retry with exponential backoff + jitter; honors Retry-After (429) header. - - Formula (without Retry-After): delay = min(base_delay * backoff_factor^attempt, max_delay) - delay += random.uniform(-jitter * delay, jitter * delay) - Formula (with Retry-After): actual_delay = min(last_retry_after_seconds, max_delay) - - NR-006 (audit 2026-08-24): when ``retry_on_5xx=True`` a 5xx - response is treated as transient infrastructure failure and - retried via the same backoff path as network errors. After the - retry budget is exhausted the LAST 5xx response is returned - (not raised) so the caller can produce a deterministic - fail-CLOSED fallback — the audit's "fail-NO-CHECK" violation - happens when a 5xx short-circuits to a synthetic block without - any retry. Default ``retry_on_5xx=False`` preserves the - pre-existing /track and /execute semantics where 5xx is a - classified GATEWAY_ERROR that raises immediately. - """ - # Eager imports for the exception classes that the ``except`` - # branch below references. Lazy imports inside the ``try`` body - # shadow the name in this scope (Python treats any assignment - # to the name as a local binding), which raises - # ``UnboundLocalError`` when the except branch tries to - # pattern-match before the lazy import has fired. - from nullrun.breaker.exceptions import ( - NullRunAuthError, - NullRunBackendError, - ) - - last_exc: Exception | None = None - - for attempt in range(max_retries + 1): - try: - result = func() - - if hasattr(result, "status_code"): - if result.status_code == 401: - err = NullRunAuthError( - "Invalid API key", - error_code="NR-A003", - user_action=( - "The NullRun backend rejected the API key (401). " - "Verify it at https://app.nullrun.io/settings/api-keys " - "and rotate if it was revoked. The key may also be " - "for a different environment (prod vs. staging) — " - "check the API_URL vs. where the key was issued." - ), - ) - _emit_for_transport_error( - err, - "execute", - result.headers.get("x-correlation-id"), - status_code=result.status_code, - ) - raise err - if result.status_code >= 500 and on_transport_error == "raise": - # 5xx is a classified GATEWAY_ERROR. Don't retry; only raise - # when caller opted into the typed-error contract. - err = NullRunBackendError( - f"Gateway returned {result.status_code}", - endpoint="execute", - status_code=result.status_code, - ) - _emit_for_transport_error( - err, - "execute", - result.headers.get("x-correlation-id"), - status_code=result.status_code, - ) - raise err - if result.status_code >= 500 and retry_on_5xx and attempt < max_retries: - # NR-006: treat 5xx as transient infra failure and retry. - # Convert to HTTPStatusError so the except branch catches - # it as a retryable condition. After retry exhaustion - # the helper returns the last response (see below). - result.raise_for_status() - elif result.status_code >= 500 and not retry_on_5xx: - # Pre-NR-006 behaviour: 5xx without ``retry_on_5xx`` - # raises HTTPStatusError so the caller (e.g. - # ``Transport.execute``) can run its fallback logic - # after retry exhaustion produces BreakerTransportError. - # ``retry_on_5xx=True`` (the /gate path) takes the - # branch above instead and returns the last response. - result.raise_for_status() - # 4xx is a real gate decision — return the response so - # the caller can synthesize the appropriate fallback - # (Transport.check returns a synthetic block; Transport.execute - # returns a synthetic block; /track batch inspects status - # directly). Calling ``raise_for_status()`` here would force - # every caller into the except path and retry a permanent - # error — the audit's NR-006 PIN 3 pins this non-retry - # contract. - - return result - - except ( - BreakerTransportError, - NullRunAuthenticationError, - NullRunTransportError, - NullRunBackendError, - ): - raise - - except httpx.HTTPStatusError as exc: - # 5xx HTTPStatusError from the retry_on_5xx branch above. - # Treat as retryable transient infra failure. - last_exc = exc - if attempt >= max_retries: - break - - except Exception as exc: - last_exc = exc - metrics.set_transport("last_error", f"{type(exc).__name__}: {exc}") - if isinstance(exc, (httpx.TimeoutException, httpx.ConnectTimeout, httpx.ReadTimeout)): - metrics.inc_transport("timeouts") - - if attempt >= max_retries: - break - - metrics.inc_transport("retries_total") - - if last_retry_after_seconds > 0: - actual_delay = min(last_retry_after_seconds, max_delay) - last_retry_after_seconds = 0.0 - logger.warning( - "Request failed (attempt %d/%d), honoring Retry-After %.2fs: %s", - attempt + 1, - max_retries + 1, - actual_delay, - type(exc).__name__, - ) - else: - delay = min(base_delay * (backoff_factor**attempt), max_delay) - jitter_amount = delay * jitter - actual_delay = delay + random.uniform(-jitter_amount, jitter_amount) # noqa: S311 - actual_delay = max(0.0, actual_delay) - logger.warning( - "Request failed (attempt %d/%d), retrying in %.2fs: %s", - attempt + 1, - max_retries + 1, - actual_delay, - type(exc).__name__, - ) - - time.sleep(actual_delay) - - # Retry exhaustion. NR-006 path: if the caller opted into - # ``retry_on_5xx`` and the failure mode was 5xx, return the - # last response so the caller can synthesize a fallback - # (e.g. ``Transport.check`` returns the legacy synthetic-block - # shape). Other exhaustion paths (network errors, timeouts) - # still raise ``BreakerTransportError`` — pre-existing - # behaviour, unchanged. - if ( - retry_on_5xx - and last_exc is not None - and isinstance(last_exc, httpx.HTTPStatusError) - and last_exc.response is not None - ): - return last_exc.response - raise BreakerTransportError(f"Request failed after {max_retries + 1} attempts") from last_exc - - -# ============================================================================= -# Fallback Modes (SDK Resilience) -# ============================================================================= - - -class FallbackMode: - """ - SDK behavior when Gateway is unavailable. - - This is CRITICAL for production - Gateway unavailability should NOT - block agent execution, but behavior must be defined and logged. - """ - - # Block if Gateway unavailable. v3.53 audit #4 — DEFAULT for - # ``Transport.execute()`` and ``ExecuteConfig.fallback_mode``. - # Per CLAUDE.md §4 "DEFAULT: fail-CLOSED для всех enforcement - # путей", the /execute enforcement path must not silently allow - # local execution when the policy engine is unreachable. - STRICT = "strict" - # Allow if Gateway unavailable, log locally. **Opt-in only** — - # pass ``fallback_mode=FallbackMode.PERMISSIVE`` explicitly when - # the caller accepts silent fail-OPEN on the enforcement path. - # Required for any test / dev harness that intentionally runs - # without a live policy engine. - PERMISSIVE = "permissive" - - -class DecisionSource: - """ - Where the decision originated - for provenance tracking. - """ - - GATEWAY = "gateway" - CACHED = "cached" - FALLBACK = "fallback" - LOCAL = "local" - - -@dataclass -class FlushConfig: - """Configuration for transport flush behavior.""" - - batch_size: int = 50 - flush_interval: float = 5.0 # seconds - # Mirror _retry_with_backoff default. - max_retries: int = 10 - retry_delay: float = 1.0 # seconds - max_buffer_size: int = 1000 # Max events before dropping oldest - max_failed_flush: int = 10 # Circuit breaker: stop trying after this many failures - - -@dataclass -class ExecuteConfig: - """Configuration for execute (strict mode) behavior.""" - - # Fallback mode when Gateway is unavailable. v3.53 audit #4 — - # default is STRICT (fail-CLOSED on enforcement) per CLAUDE.md §4. - # Pre-v3.53 the default was PERMISSIVE which silently allowed - # local execution on transport failure; that was fail-OPEN on the - # primary enforcement path (Transport.execute → /api/v1/execute). - fallback_mode: str = FallbackMode.STRICT - # Gateway timeout in seconds - timeout: float = 5.0 - # Max retries for execute calls - max_retries: int = 10 - # Cache TTL for CACHED mode (seconds) - cache_ttl: float = 60.0 - # Cache max size - cache_max_size: int = 10000 - - -class Transport: - """ - HTTP transport with batching support. - - Features: - - Non-blocking track calls (append to buffer) - - Background flush at intervals or when batch_size reached - - Retry logic for failed requests - - Thread-safe for sync usage - - HMAC request signing for secure authentication - - Distributed circuit breaker via Redis for multi-worker deployments - """ - - def __init__( - self, - api_url: str, - api_key: str | None = None, - secret_key: str | None = None, - config: FlushConfig | None = None, - redis_client: Any = None, - ): - self.api_url = api_url.rstrip("/") - - # TLS enforcement: reject non-localhost HTTP. Uses urlparse + ip_address - # so homograph attacks (e.g. 127.0.0.1.attacker.com) don't slip through - # a naive startswith("127.") check. - from ipaddress import ip_address - from urllib.parse import urlparse - - parsed = urlparse(self.api_url) - if parsed.scheme == "http": - host = (parsed.hostname or "").lower() - allowed = host == "localhost" or host == "::1" - if not allowed: - try: - addr = ip_address(host) - allowed = addr.is_loopback - except ValueError: - allowed = False - if not allowed: - raise InsecureTransportError( - f"Insecure URL detected: {self.api_url}. " - f"HTTP is only allowed for localhost / 127.0.0.0/8 / ::1. " - f"Use https:// for production." - ) - - self.api_key = api_key - self.secret_key = secret_key # HMAC signing key - self.config = config or FlushConfig() - # Allow env-var override of batch size and flush interval. - if "NULLRUN_BATCH_SIZE" in os.environ: - try: - self.config.batch_size = int(os.environ["NULLRUN_BATCH_SIZE"]) - except ValueError: - logger.warning( - "NULLRUN_BATCH_SIZE=%r is not an int; ignoring", - os.environ["NULLRUN_BATCH_SIZE"], - ) - if "NULLRUN_FLUSH_INTERVAL_MS" in os.environ: - try: - self.config.flush_interval = int(os.environ["NULLRUN_FLUSH_INTERVAL_MS"]) / 1000.0 - except ValueError: - logger.warning( - "NULLRUN_FLUSH_INTERVAL_MS=%r is not an int; ignoring", - os.environ["NULLRUN_FLUSH_INTERVAL_MS"], - ) - self._buffer: list[dict[str, Any]] = [] - self._in_flight: dict[str, dict[str, Any]] = {} # event_id -> event for retry dedup - # RLock so re-entrant acquisition (e.g. test fixtures that hold the - # lock while calling lock-acquiring methods) doesn't deadlock. - self._lock = threading.RLock() - self._flush_thread: threading.Thread | None = None - self._running = False - # Cancellable sleep primitive: Event.wait returns immediately when - # stop() sets the event, so teardown doesn't block for the full - # flush_interval. Pin: tests/test_transport.py::test_stop_interrupts_flush_sleep. - self._stop_event = threading.Event() - - # mTLS client certificate support - # NULLRUN_TLS_CLIENT_CERT and NULLRUN_TLS_CLIENT_KEY env vars for client cert auth - client_cert_path = os.environ.get("NULLRUN_TLS_CLIENT_CERT") - client_key_path = os.environ.get("NULLRUN_TLS_CLIENT_KEY") - ca_cert_path = os.environ.get("NULLRUN_TLS_CA_CERT") # Optional custom CA - - # Build SSL configuration for mTLS - # For client cert auth: verify is a CA cert, cert is tuple of (client_cert, client_key) - verify_cert: bool | str = True - client_cert: tuple[str, str] | None = None - if client_cert_path and client_key_path: - # Client certificate authentication (mTLS) - client_cert = (client_cert_path, client_key_path) - verify_cert = ca_cert_path if ca_cert_path else True - logger.debug(f"mTLS enabled: client_cert={client_cert_path}") - elif ca_cert_path: - # Custom CA certificate only (no client cert) - verify_cert = ca_cert_path - logger.debug(f"Custom CA configured: ca_cert={ca_cert_path}") - - self._client = httpx.Client( - timeout=httpx.Timeout( - connect=5.0, - read=30.0, - write=10.0, - pool=5.0, - ), - verify=verify_cert, - cert=client_cert, - limits=httpx.Limits( - max_connections=10, - max_keepalive_connections=5, - keepalive_expiry=30.0, - ), - ) - self._redis_client = redis_client - self._circuit_breaker = CircuitBreaker( - failure_threshold=self.config.max_failed_flush, - recovery_timeout=30.0, - redis_client=redis_client, - name="transport", - ) - self._stopped = False # Track if stop was called - # 0.7.0 thin client: no local policy cache. Backend is authoritative. - _masked = api_key[:8] + "***" if api_key and len(api_key) >= 8 else "***" - logger.debug(f"Transport initialized: api_url={self.api_url}, api_key={_masked}") - - # OpenTelemetry tracer (lazy-loaded: only if opentelemetry is installed) - self._tracer = None - self._propagator = None - if _OTEL_AVAILABLE: - self._tracer = trace.get_tracer("nullrun.transport") - self._propagator = TraceContextTextMapPropagator() - - # Final-flush hook via weakref.finalize — only fires if this Transport - self._finalizer = weakref.finalize(self, self._atexit_flush_safe) - - @staticmethod - def _atexit_flush_safe(_self_id: int | None = None) -> None: - """Weakref finalizer entry point. - - ``weakref.finalize`` calls this with no arguments (``self`` is gone). - The recommended lifecycle is explicit ``stop()`` or ``with Transport(...)``. - If neither was used, we log a one-time DEBUG line. - """ - logger.debug( - "Transport finalizer fired without explicit stop(); " - "remaining events may be lost. Use Transport as a context " - "manager or call stop() explicitly." - ) - - # WAL rotation threshold (default 64 MB). Override via NULLRUN_WAL_MAX_BYTES. - _WAL_MAX_BYTES_DEFAULT: int = 64 * 1024 * 1024 - - @property - def _wal_max_bytes(self) -> int: - """Effective WAL rotation threshold.""" - raw = os.environ.get("NULLRUN_WAL_MAX_BYTES", "").strip() - if not raw: - return self._WAL_MAX_BYTES_DEFAULT - try: - value = int(raw) - return value if value > 0 else self._WAL_MAX_BYTES_DEFAULT - except ValueError: - return self._WAL_MAX_BYTES_DEFAULT - - def _wal_path(self) -> str: - """Resolve WAL path. Honours ``NULLRUN_WAL_PATH``; defaults to platform tempdir.""" - env_path = os.environ.get("NULLRUN_WAL_PATH") - if env_path: - return env_path - return os.path.join(tempfile.gettempdir(), "nullrun.wal") - - def _rotate_wal_if_needed(self) -> None: - """Rotate ```` to ``.1`` if it exceeds the size cap.""" - wal_path = self._wal_path() - try: - size = os.path.getsize(wal_path) - except OSError: - return - if size < self._wal_max_bytes: - return - rotated = f"{wal_path}.1" - try: - os.replace(wal_path, rotated) - logger.info( - f"WAL rotated: {wal_path} ({size} bytes) -> {rotated} " - f"after exceeding cap of {self._wal_max_bytes} bytes" - ) - except OSError as e: - logger.warning(f"Failed to rotate WAL {wal_path}: {e}") - - def _persist_to_wal(self) -> None: - """Persist unflushed events to WAL file for replay on restart.""" - if not self._buffer: - return - event_count = len(self._buffer) - wal_path = self._wal_path() - self._rotate_wal_if_needed() - wal_dir = os.path.dirname(wal_path) or "." - try: - os.makedirs(wal_dir, exist_ok=True) - except OSError as e: - logger.warning(f"Cannot create WAL directory {wal_dir}: {e}") - return - tmp_path = f"{wal_path}.tmp.{os.getpid()}" - try: - with open(tmp_path, "a") as f: - for event in self._buffer: - # 2026-07-24 (Decimal serialization): same default=str as - f.write(json.dumps(event, default=str) + "\n") - f.flush() - os.fsync(f.fileno()) - os.replace(tmp_path, wal_path) - self._buffer.clear() - logger.debug(f"Persisted {event_count} events to WAL at {wal_path}") - except OSError as e: - logger.warning(f"Failed to persist {event_count} events to WAL: {e}") - - def _replay_from_wal(self) -> None: - """Replay events from WAL file on startup. - - P1-5b: also drains the rotated ``.wal.1`` (oldest - surviving recovery window) before the active ``.wal`` so - a crash between rotation and replay doesn't lose events. - Both files are removed only after a successful flush. - """ - events: list[dict[str, Any]] = [] - for candidate in (f"{self._wal_path()}.1", self._wal_path()): - try: - with open(candidate) as f: - for line in f: - try: - events.append(json.loads(line.strip())) - except json.JSONDecodeError: - continue - except FileNotFoundError: - continue - except OSError as e: - logger.warning(f"Failed to read WAL {candidate}: {e}") - continue - try: - os.remove(candidate) - except OSError as e: - logger.warning(f"Failed to remove WAL {candidate}: {e}") - if events: - self._buffer.extend(events) - self._do_flush() - if events: - logger.info(f"Replayed {len(events)} events from WAL") - - def track(self, event: dict[str, Any]) -> None: - """ - Add event to buffer. Non-blocking. - - Events are flushed either when batch_size is reached or - flush_interval elapses. - """ - with self._lock: - # Generate event_id if not provided - if "event_id" not in event or not event["event_id"]: - event["event_id"] = str(uuid.uuid4()) - - # Store in-flight for retry dedup - self._in_flight[event["event_id"]] = event - - self._buffer.append(event) - metrics.inc_transport("events_enqueued") - - if len(self._buffer) >= self.config.batch_size: - self._do_flush_locked() - - def start(self) -> None: - """Start background flush thread.""" - if self._running: - return - # Replay any events from WAL that were persisted due to previous crash - self._replay_from_wal() - self._running = True - # Clear the stop latch so a previous stop() does not short-circuit - # the new flush loop on its first sleep. - self._stop_event.clear() - self._flush_thread = threading.Thread(target=self._flush_loop, daemon=True) - self._flush_thread.start() - logger.info("Transport flush thread started") - - def __enter__(self) -> "Transport": - """Context-manager entry: start the flush thread and return self. - - Pairs with ``__exit__`` so callers can write - ``with Transport(...) as t:`` and rely on ``stop `` running - on the way out. Replaces the manual ``start / stop `` pair - that was easy to forget in long-running services. - """ - self.start() - return self - - def __exit__(self, exc_type, exc_val, exc_tb) -> None: - """Context-manager exit: stop the flush thread and persist WAL. - - Always stops, regardless of whether the body raised. The - exception (if any) is NOT swallowed — the caller still sees - it after the with-block. - """ - try: - self.stop() - except Exception as e: # noqa: BLE001 — best-effort on context exit - logger.debug(f"Transport.__exit__: stop() raised: {e}") - - def stop(self, timeout: float = 10.0, flush: bool = True) -> None: - """Stop background flush thread and flush remaining events. - - Args: - timeout: max seconds to wait for the flush thread to exit. - flush: when True (default) the final ``_do_flush()`` and - ``_persist_to_wal()`` run after the thread joins. When - False, the thread is cancelled but the buffer is left - alone. The test conftest uses ``flush=False`` to teardown - between tests without a final httpx call. - """ - self._running = False - self._stopped = True # Mark as stopped to prevent double flush - self._stop_event.set() # Wake flush thread out of its cancellable sleep. - if self._flush_thread: - self._flush_thread.join(timeout=timeout) - if flush: - self._do_flush() # Final flush - self._persist_to_wal() # WAL any remaining events - self._client.close() - if getattr(self, "_finalizer", None) is not None and self._finalizer.alive: - self._finalizer.detach() - logger.info("Transport stopped") - - def _flush_loop(self) -> None: - """Background loop that periodically flushes.""" - while self._running: - # Event.wait returns True when stop() sets the event (cancel signal). - cancelled = self._stop_event.wait(timeout=self.config.flush_interval) - if cancelled: - break - if self._running: - self._do_flush() - - def _do_flush(self) -> None: - """Perform the actual flush.""" - with self._lock: - self._do_flush_locked() - - def _do_flush_locked(self) -> None: - """Flush under lock. Must be called with _lock held.""" - if not self._buffer: - logger.debug("Buffer empty, skipping flush") - return - - batch = self._buffer[:] - self._buffer.clear() - logger.debug(f"Sending batch of {len(batch)} events") - - # Circuit breaker wrapped send - uses proper 3-state circuit breaker - def send_batch(): - result = self._send_batch_with_retry_info(batch) - # Remove accepted events from in-flight - if result.accepted_event_ids: - for event in batch: - if event.get("event_id") in result.accepted_event_ids: - self._in_flight.pop(event.get("event_id"), None) - logger.debug(f"Flushed {len(batch)} events") - # Update metrics on successful flush (thread-safe) - metrics.inc_transport("batches_sent") - metrics.inc_transport("events_sent", len(batch)) - metrics.set_transport("last_flush_at", time.monotonic()) - return result - - try: - self._circuit_breaker.call(send_batch) - except BreakerTransportError: - logger.warning(f"Circuit breaker OPEN. Batch of {len(batch)} events will be re-queued.") - # Drop NEWEST non-critical (state_change etc.) so oldest events - # (incident start, billing-period start) survive — they power - # monthly rollups. Critical control-plane events are kept. - available_space = self.config.max_buffer_size - len(self._buffer) - if available_space < len(batch): - overflow = len(batch) - available_space - if overflow > 0: - batch = self._drop_newest_with_priority(batch, overflow) - self._buffer.extend(batch) # Append to END so oldest events retry first. - metrics.inc_transport("batches_failed") - - def _drain_batch(self) -> list[dict[str, Any]] | None: - """Public, lock-acquiring snapshot of the current buffer. Returns ``None`` when empty.""" - with self._lock: - if not self._buffer: - return None - batch = list(self._buffer) - del self._buffer[:] - return batch - - # Control-plane events that MUST NOT be dropped on overflow. - _CRITICAL_EVENT_TYPES = frozenset( - { - "state_change", - "kill_received", - "policy_invalidated", - "key_rotated", - } - ) - - def _drop_newest_with_priority( - self, - batch: list[dict[str, Any]], - overflow: int, - ) -> list[dict[str, Any]]: - """Drop ``overflow`` newest non-critical events; keep critical events and oldest. - - Cost-audit invariant: under overflow we keep the OLDEST events - (incident / billing-period start) — dropping oldest would silently - break monthly rollups. Never drop critical events at the cost of a - brief buffer overshoot. - """ - if overflow <= 0: - return batch - kept: list[dict[str, Any]] = [] - dropped = 0 - for event in reversed(batch): - if dropped < overflow and event.get("type") not in self._CRITICAL_EVENT_TYPES: - dropped += 1 - continue - kept.append(event) - if dropped > 0: - logger.warning( - f"buffer overflow: dropped {dropped} newest non-critical " - f"events (kept {len(kept)}, preserved {len(batch) - len(kept) - dropped} critical)" - ) - metrics.inc_transport("events_dropped", dropped) - kept.reverse() - return kept - - @dataclass - class SendResult: - accepted_event_ids: list[str] - retry_after_ms: float | None = None - is_policy_limit: bool = False - - def _add_hmac_headers(self, headers: dict[str, str], body: str | bytes) -> None: - """Add X-Signature-Timestamp + X-Signature headers. No-op if secret_key/api_key missing.""" - if not self.secret_key or not self.api_key: - return - - timestamp = int(time.time()) - signature = generate_hmac_signature( - self.api_key, - self.secret_key, - timestamp, - body, - ) - - headers["X-Signature-Timestamp"] = str(timestamp) - headers["X-Signature"] = signature - - def _build_signed_headers( - self, - body: str | bytes | None = None, - extra: dict[str, str] | None = None, - ) -> dict[str, str]: - """Build the canonical signed-headers dict for every signed POST. - - Always includes Content-Type: application/json and X-API-Key (when - api_key is set). Adds HMAC headers when secret_key is set and a - body is provided. ``extra`` is merged on top of defaults so callers - can override Content-Type or add custom headers. - """ - headers: dict[str, str] = { - "Content-Type": "application/json", - } - if self.api_key: - headers["X-API-Key"] = self.api_key - # Backend CSRF middleware bypasses cookie-double-submit when an - # Authorization header is present (backend/src/auth/csrf.rs). - # Without this, SDK POSTs hit the "state-changing request without - # session cookie" branch and get 403, which the SDK silently swallowed. - headers["Authorization"] = f"Bearer {self.api_key}" - if body is not None and self.secret_key and self.api_key: - timestamp = int(time.time()) - signature = generate_hmac_signature(self.api_key, self.secret_key, timestamp, body) - headers["X-Signature-Timestamp"] = str(timestamp) - headers["X-Signature"] = signature - if extra: - headers.update(extra) - # Backend rejects signed POSTs without X-NULLRUN-PROTOCOL: 3 with 400. - headers[HEADER_PROTOCOL] = _protocol_header_value() - self._inject_trace_context(headers) - return headers - - def _inject_trace_context(self, headers: dict[str, str]) -> None: - """ - Inject trace context into request headers (W3C Trace Context format). - - This enables distributed tracing across SDK and backend. - Uses W3C Trace Context standard for trace_id propagation. - """ - if not _OTEL_AVAILABLE or not self._propagator: - return - - carrier: dict[str, str] = {} - self._propagator.inject(carrier) - headers.update(carrier) - - def _extract_retry_after(self, response: httpx.Response) -> float | None: - """Extract Retry-After header value as seconds. - - Handles both: - - Integer seconds (e.g., "30") - - HTTP-date format (e.g., "Wed, 21 Oct 2015 07:28:00 GMT") - """ - retry_after = response.headers.get("Retry-After") - if not retry_after: - return None - - # Try parsing as seconds (integer or float) - try: - return float(retry_after) - except ValueError: - pass - - # Try parsing as HTTP datetime (RFC 7231) - try: - from email.utils import parsedate_to_datetime - - dt = parsedate_to_datetime(retry_after) - from datetime import datetime, timezone - - return (dt - datetime.now(timezone.utc)).total_seconds() - except Exception: - pass - - return None - - def _send_batch_with_retry_info(self, batch: list[dict[str, Any]]) -> "SendResult": - """Send batch to server. Returns SendResult with retry info. Wrapped by _retry_with_backoff.""" - logger.debug(f"Sending batch of {len(batch)} events to {self.api_url}/api/v1/track/batch") - body = _signed_request_body({"events": batch}) - headers = self._build_signed_headers(body=body) - - # Inner function is the unit of retry: - # * 5xx → retry helper backs off. 429 honors Retry-After. - # * 4xx (other than 429) → return as-is; these are real client bugs - # (auth, payload) and must NOT be retried. - def _post_batch() -> httpx.Response: - resp = self._client.post( - f"{self.api_url}/api/v1/track/batch", - content=body, - headers=headers, - ) - if resp.status_code >= 500 or resp.status_code == 429: - # raise_for_status turns this into HTTPStatusError; the retry - # helper wraps that into BreakerTransportError after retries. - resp.raise_for_status() - return resp - - max_track_retries = getattr(self, "_track_max_retries", 10) - response = _retry_with_backoff( - _post_batch, - max_retries=max_track_retries, - base_delay=0.5, - max_delay=10.0, - backoff_factor=2.0, - jitter=0.1, - ) - - # P0: Extract retry_after from response headers or body - retry_after_seconds: float | None = None - retry_after_ms: float | None = None - is_policy_limit = False - - # Check Retry-After header (may be seconds or HTTP-date) - retry_after_seconds = self._extract_retry_after(response) - - # Check response body for retry info - try: - data = response.json() - # Check for rejection info - if "rejected" in data and data["rejected"]: - rejected_info = data["rejected"] - if isinstance(rejected_info, dict): - if "retry_after_ms" in rejected_info: - retry_after_ms = rejected_info["retry_after_ms"] - if "reason" in rejected_info and rejected_info["reason"] == "policy_limit": - is_policy_limit = True - except Exception: # noqa: S110 - pass - - # Store for next retry calculation (prefer header seconds, fallback to body ms) - if retry_after_seconds is not None: - self._last_retry_after_seconds = retry_after_seconds - retry_after_ms = retry_after_seconds * 1000 - elif retry_after_ms is not None: - self._last_retry_after_seconds = retry_after_ms / 1000.0 - else: - self._last_retry_after_seconds = 0.0 - self._last_failure_policy_limit = is_policy_limit - - # Handle 429 response - extract and store Retry-After before raising - if response.status_code == 429: - retry_after = self._extract_retry_after(response) - if retry_after: - self._last_retry_after_seconds = retry_after - response.raise_for_status() - response.raise_for_status() - - # Process actions from server response. Per-element try/except so one - # malformed entry doesn't abort the whole loop. - try: - data = response.json() - actions = data.get("actions") or [] - for action in actions: - try: - if not isinstance(action, dict): - logger.warning("Skipping non-dict action from /track/batch: %r", action) - continue - action_type = action.get("type", "") - workflow_id = action.get("workflow_id", "unknown") - reason = action.get("reason", "") - if action_type: - handle_action(action_type, workflow_id, reason) - except Exception as item_err: - logger.warning("Skipping malformed action %r: %s", action, item_err) - for msg in data.get("messages", []) or []: - logger.info("Backend message: %s", msg) - except Exception as e: - logger.warning(f"Failed to process actions: {e}") - - # Return accepted event_ids for retry dedup - accepted_event_ids = data.get("accepted_event_ids", []) if "data" in locals() else [] - logger.debug(f"Batch track: sent {len(batch)} events") - return self.SendResult( - accepted_event_ids=accepted_event_ids, - retry_after_ms=retry_after_ms, - is_policy_limit=is_policy_limit, - ) - - def flush_now(self) -> None: - """Force immediate flush.""" - self._do_flush() - - # ============================================================================= - # Execute (Strict Mode) - # ============================================================================= - - def execute( - self, - organization_id: str, - execution_id: str, - trace_id: str, - tool: str, - input_data: dict[str, Any], - mode: str = "auto", - # v3.53 audit #4 — default flipped from PERMISSIVE to STRICT - # to match CLAUDE.md §4 ("DEFAULT: fail-CLOSED для всех - # enforcement путей"). /execute is the primary enforcement - # point (see docstring) — when the gateway is unreachable the - # body MUST NOT run on a silent local pass. Callers that - # intentionally want fail-OPEN on this path (dev / test - # harnesses without a live engine) must opt in by passing - # ``fallback_mode=FallbackMode.PERMISSIVE`` explicitly. - fallback_mode: str = FallbackMode.STRICT, - operation_id: str | None = None, - approval_id: str | None = None, - # Typed-impact + digest-bound approval. Forwarded when @sensitive(impact=...) - # built them so the backend can stamp the approval row with the digest. - business_impact: dict[str, Any] | None = None, - action_digest: str | None = None, - # Tool-call argument bag forwarded on /execute so the gate can compute - # a schema fingerprint and write it to mcp_tool_signatures. - tool_arguments: dict[str, Any] | None = None, - # Per-call `tools` list forwarded on /execute so the backend's - # Step 3 tool_block check (`backend/src/proxy/http/gate/orchestrator.rs:1847-1893`) - # can match each tool against the workflow's effective `tool_patterns` - # aggregate. Without this, TB-1 fails closed with `no_tools_field` - # whenever the workflow has an active `policy.tool_patterns` block. - # Populated by `runtime.execute` from the `get_call_tools()` contextvar - # when the caller invoked `set_call_context(tools=...)` (or the - # `_enforce_sensitive_tool` decorator did so on their behalf). - tools: tuple[str, ...] | None = None, - on_transport_error: Callable[[Exception], dict[str, Any]] | None = None, - ) -> dict[str, Any]: - """Pre-execution policy evaluation via /api/v1/execute (PRIMARY enforcement point). - - Wire contract (revised 2026-09-08, DEFS-SDKEXEC-GATE-FIRST): - /execute REQUIRES a prior /gate call that minted the same - ``execution_id`` and registered the ``execution:{id}`` binding - in Redis. Backend enforcement: - ``backend/src/proxy/http/gate/execute.rs:46-208`` - (DEF-SDKK-022-EXEC-BYPASS, 2026-09-04, RUN_ID=20260904T1500) - runs ``HGET execution:{id} ORG_FIELD`` on entry; a miss - returns 404 EXECUTION_NOT_FOUND (fail-CLOSED). The SDK - therefore MUST thread the execution_id captured by - ``runtime.check_workflow_budget`` (which calls ``Transport.check``, - i.e. /gate) into the body of this /execute call. See - ``runtime.execute()`` (line ~2820) for the reuse path; this - method's caller is the single source of truth for - ``execution_id`` selection. - - Prior to DEF-SDKK-022 the comment here claimed "/execute MUST - be called rather than /gate" — that contract was the legacy - pre-2026-09-04 shape. The post-fix shape is "/execute MUST be - preceded by /gate for the same execution_id" — the budget - pre-flight (Transport.check, /api/v1/gate) is the binding - registrar; /execute is the policy decision that re-uses it. - - Args: - organization_id: Organization identifier - execution_id: Execution identifier - trace_id: Distributed trace ID - tool: Tool to execute - input_data: Tool input - mode: Execution mode ("auto", "inline", "strict") - fallback_mode: What to do if Gateway unavailable - operation_id: Optional idempotency key - on_transport_error: Optional callback invoked on BreakerTransportError. - When set, the callback's return value is returned verbatim; otherwise - the request falls through to fallback_mode. The decorator's - _enforce_sensitive_tool sets this to convert the error into a - NullRunBlockedException (fail-CLOSED). - - Returns: - Dict with: - - decision: "allow" | "block" | "flag" | "pause" | "require_approval" - - decision_source: "gateway" | "cached" | "fallback" - - explanation: Human-readable explanation - - policy_hash: Server-side SHA-256 of the policy applied - (v4 wire field; null on pre-v4 backends). NOT a - sequential `policy_version` number — wire v3/v4 backends - emit only `policy_hash`. Synthetic fallback dicts ship - `policy_version: 0` for legacy compatibility; real - responses populate `policy_hash` only. - - decision_context: Context for replay (if available) - """ - gate_request = { - "organization_id": organization_id, - "execution_id": execution_id, - "trace_id": trace_id, - "tool": tool, - "input": input_data, - "mode": mode, # Wire-present but unused by backend; kept for compat. - "operation_id": operation_id or str(uuid.uuid4()), - } - if approval_id is not None: - gate_request["approval_id"] = approval_id - if business_impact is not None: - gate_request["business_impact"] = business_impact - if action_digest is not None: - gate_request["action_digest"] = action_digest - if tool_arguments is not None: - gate_request["tool_arguments"] = tool_arguments - if tools is not None: - gate_request["tools"] = list(tools) - - body = _signed_request_body(gate_request) - headers = self._build_signed_headers(body=body) - - def do_execute_request() -> httpx.Response: - return self._client.post( - f"{self.api_url}/api/v1/execute", - content=body, - headers=headers, - timeout=5.0, - ) - - # Per-instance override so tests/CI can shrink the retry budget. - max_execute_retries = getattr(self, "_execute_max_retries", 10) - try: - response = _retry_with_backoff( - do_execute_request, - max_retries=max_execute_retries, - base_delay=0.5, - on_transport_error=on_transport_error, - ) - - if response.status_code == 200: - data = response.json() - data["decision_source"] = DecisionSource.GATEWAY - # 0.7.0 thin client: no local policy cache. The next - return data # type: ignore[no-any-return] - elif response.status_code >= 400: - # 4xx — don't retry. - # - # 2026-09-10 (NR-SDK-A015-SURFACE): before the fix, - # this branch dropped the wire envelope on the floor - # and synthesised a generic ``{"decision": "block", - # "explanation": "Gateway returned 409"}`` dict. That - # hid every wire-coded reason (`APPROVAL_REPLAY_REJECTED`, - # `APPROVAL_DENIED`, `BUDGET_HARD_BLOCKED`, etc.) behind - # a single string, so the runtime block dispatch fell - # through to ``NR-X001`` and `format_user_message` - # produced the catalogue fallback ("Something went - # wrong. Please try again.") instead of the typed - # `NR-A015` message. Cookbook callers had no way to - # branch on the precise cause. - # - # Post-fix: parse the envelope via the existing - # `_parse_v3_error_envelope` helper — it covers the - # v3 wire envelope for every /execute reject reason, - # including the six typed approval grant-consume - # outcomes (`APPROVAL_NOT_YET_APPROVED` → - # ``NullRunApprovalNotYetApprovedError`` (NR-A010), - # `APPROVAL_DENIED` → NR-A011, - # `APPROVAL_EXPIRED` → NR-A012, - # `APPROVAL_DIGEST_MISMATCH` → NR-A013, - # `APPROVAL_TOOL_DIGEST_MISMATCH` → NR-A014, - # `APPROVAL_REPLAY_REJECTED` → NR-A015 / `` - # NullRunApprovalReplayRejectedError``) — and raise - # the typed exception so the @protect / - # @sensitive / runtime.execute() exception arms - # propagate the right class up to the caller. - # - # Fall through to the synthetic block shape if the - # envelope is unrecognised (plaintext body, malformed - # JSON, unknown wire code) so behaviour stays - # backwards-compatible for legacy / non-v3 backends. - # `_parse_v3_error_envelope` always returns an - # Exception — it never silently swallows a 4xx. - try: - raise _parse_v3_error_envelope(response, "execute") - except NullRunApprovalReplayRejectedError as exc: - # The exact case the user reported: the operator - # approved, the SDK polled /execute again, and - # the backend's atomic consume_approved UPDATE - # returned zero rows (replay race — UI approve - # vs SDK re-check). Surface the typed exception - # so `format_user_message` yields the NR-A015 - # catalogue line ("Your request couldn't be - # completed because the approval has already - # been used. Please start a new request.") - # instead of the fallback. - metrics.inc_transport("execute_block_replay_rejected") - raise - except NullRunBlockedException as exc: - # All other typed blocks from the dispatch — - # budget, rate, tool, approval-deny, etc. - # Re-raise for the @protect / runtime.execute - # arms to handle. - metrics.inc_transport("execute_block_typed") - raise - except NullRunBackendError as exc: - # 5xx-classified envelope parsed as a typed - # backend error (shouldn't normally land here - # because the helper maps 5xx to GATEWAY_ERROR - # via NullRunTransportError, but stays - # defensive). Re-raise. - raise - except NullRunAuthenticationError as exc: - # 401 envelope parsed as auth error — surface - # directly so the caller can react. - raise - except NullRunTransportError as exc: - # Transport-classified (network, breaker) — not - # a real 4xx, but helper may return one if the - # envelope shape is ambiguous. Re-raise so the - # on_transport_error arm sees it. - raise - except NullRunDecision as exc: - # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): - # umbrella pass-through for typed Decision - # subclasses NOT in the NullRunBlockedException - # MRO. Specifically: - # - NullRunChainError (NR-CH001) — chain - # lifetime / cross-org / Execution Graph - # parent-lineage rejections - # - NullRunWorkflowInactiveError (NR-W004) — - # soft-deleted workflow - # - NullRunConsumeOverbudgetError (NR-O001) — - # CONSUME > RESERVE + epsilon_cents invariant - # - WorkflowPausedException (NR-W003) - # Pre-fix these fell through to `except - # Exception: pass` below and got silently - # swallowed into the synthetic block shape - # (`{"decision": "block", "decision_source": - # FALLBACK, "explanation": f"Gateway returned - # {response.status_code}"}`) — losing - # exc.chain_id / exc.parent_execution_id (Chain), - # exc.workflow_id (WorkflowInactive), - # exc.execution_id / exc.reserved_cents / - # exc.actual_cost_cents / exc.epsilon_cents - # (ConsumeOverbudget), and every typed - # `error_code`/user-action. MUST come AFTER the - # NullRunBlockedException arm above so the typed - # approval / budget / tool-block path still - # matches by MRO specificity. - metrics.inc_transport("execute_block_decision_typed") - raise - except NullRunInfrastructureError as exc: - # DEF-NR-TRANSPORT-CATCHFANIN-GAP (2026-09-10): - # umbrella pass-through for typed - # Infrastructure subclasses NOT in the - # NullRunBackendError / NullRunAuthenticationError / - # NullRunTransportError MRO branches above. - # Specifically: - # - NullRunProtocolError (NR-P001) — - # PROTOCOL_TOO_OLD / PROTOCOL_TOO_NEW / - # PROTOCOL_HEADER_INVALID / - # PROTOCOL_HEADER_REQUIRED - # - NullRunRateLimitRedisError (NR-R002) — - # RATE_LIMIT_REDIS_UNAVAILABLE - # - NullRunConfigError (NR-Cxxx) — when raised - # from a wire envelope (rare; mostly SDK-side) - # NullRunAuthError (NR-A003) IS in the - # NullRunAuthenticationError arm above (parent - # class match), but listing here for completeness - # preserves the documented recovery contract - # even if a future refactor reorders the prior - # arms. - # Pre-fix these fell through to `except Exception: - # pass` below — same synthetic-block loss as the - # Decision path. MUST come AFTER the three - # specific parent arms above (Backend, Auth, - # Transport) so the wire-classified exceptions - # still match by MRO specificity. - metrics.inc_transport("execute_block_infra_typed") - raise - except Exception: - # Unrecognised envelope (plaintext body, legacy - # slug, malformed JSON). Fall through to the - # synthetic block shape so old / non-v3 backends - # keep working and ``on_transport_error="raise"`` - # callers still see a usable dict. The retry - # helper has already given up; emitting a typed - # exception here would mask unknown wire codes - # the user hasn't yet catalogued. - pass - return { - "decision": "block", - "decision_source": DecisionSource.FALLBACK, - "explanation": f"Gateway returned {response.status_code}", - "policy_hash": None, - } - - except BreakerTransportError as exc: - # ADR-008: on_transport_error accepts callables AND strings: - if callable(on_transport_error): - return on_transport_error(exc) - if on_transport_error == "raise": - raise NullRunTransportError( - f"Gateway unreachable on /execute: {exc}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="execute", - ) from exc - if on_transport_error == "open": - return { - "decision": "allow", - "decision_source": TransportErrorSource.NETWORK_ERROR, - "explanation": f"Gateway unreachable: {exc}", - "policy_hash": None, - } - if on_transport_error == "closed": - return { - "decision": "block", - "decision_source": TransportErrorSource.NETWORK_ERROR, - "explanation": f"Gateway unreachable: {exc}", - "policy_hash": None, - } - pass # fall through to fallback mode - except NullRunTransportError: - raise # Already classified -- propagate as-is - except httpx.RequestError as exc: - if callable(on_transport_error): - return on_transport_error(exc) - if on_transport_error == "raise": - raise NullRunTransportError( - f"Network error on /execute: {exc}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="execute", - ) from exc - raise - except NullRunAuthenticationError: - raise # Don't fall back on auth errors - - # All attempts failed - apply fallback mode. - metrics.inc_transport("fallback_mode_activations") - if fallback_mode == FallbackMode.STRICT: - return { - "decision": "block", - "decision_source": DecisionSource.FALLBACK, - "explanation": "Gateway unavailable, fallback=STRICT", - "policy_version": 0, - } - else: # PERMISSIVE (opt-in) - # v3.53 audit #4 — PERMISSIVE no longer the default; it - # requires the caller to pass fallback_mode=FallbackMode. - # PERMISSIVE explicitly. Synthesizes an allow + decision_ - # source=FALLBACK so the caller / @sensitive decorator can - # still observe that the engine was unreachable. - return { - "decision": "allow", - "decision_source": DecisionSource.FALLBACK, - "explanation": "Gateway unavailable, fallback=PERMISSIVE", - "policy_version": 0, - } - - def check( - self, - check_request: dict[str, Any], - on_transport_error: Callable[[Exception], dict[str, Any]] | str | None = None, - parent_execution_id: str | None = None, - ) -> dict[str, Any]: - """ - Call /api/v1/gate endpoint for pre-execution budget checking. - - Uses the unified gate endpoint with check_type for budget validation. - Supports idempotency via operation_id field. - - Args: - check_request: Dict with: - - organization_id: Organization identifier - - execution_id: Execution identifier - - operation_id: Operation identifier (for idempotency) - - check_type: "llm" or "tool" - - model: Model name (for LLM checks) - - tool_name: Tool name (for tool checks) - - estimated_tokens: Token count (for LLM checks) - - input: Optional input data - - Returns: - Dict with: - - decision: "allow" | "block" | "throttle" - - reservation_id: Optional reservation ID - - remaining_budget_cents: Remaining budget - - projected_cost_cents: Projected cost for this operation - - explanations: List of explanation strings - - suggestions: List of suggestion strings - """ - # Convert check_request to gate_request format - gate_request = { - "organization_id": check_request.get("organization_id"), - "execution_id": check_request.get("execution_id"), - "trace_id": check_request.get("trace_id", str(uuid.uuid4())), - "tool": check_request.get("tool_name") or check_request.get("tool"), - "input": check_request.get("input"), - "mode": "auto", - "check_type": check_request.get("check_type"), - "model": check_request.get("model"), - "estimated_tokens": check_request.get("estimated_tokens"), - "operation_id": check_request.get("operation_id") or str(uuid.uuid4()), - # Forward the per-call `tools` list so the backend's - # `gate/internal.rs::check_tool_block` can match each - # tool against the workflow's effective `blocked_tools` - # aggregate. When unset (None) we omit the key entirely - # -- the backend distinguishes "no tools sent" from - # "explicit []". - **({"tools": check_request["tools"]} if "tools" in check_request else {}), - } - - # Wire-protocol v3 fields. Forwarded only when present so - if check_request.get("chain_id") is not None: - gate_request["chain_id"] = check_request["chain_id"] - if check_request.get("chain_op") is not None: - gate_request["chain_op"] = check_request["chain_op"] - if check_request.get("idempotency_key") is not None: - gate_request["idempotency_key"] = check_request["idempotency_key"] - if "stream" in check_request: - gate_request["stream"] = bool(check_request["stream"]) - # v0.16.1 (Phase-1+ wire-shape fix): runtime.check_workflow_budget - # always sets `action_digest` so the gate's - # `if req.action_digest.is_none()` version-gate passes - # (`backend/src/proxy/http/gate/gate.rs:56`, ADR-023 P1-6). - # Pre-v0.16.1 / Phase-0 callers can still omit it (forwarded - # only when truthy) without triggering a "field present - # but None" wire-shape drift. - if check_request.get("action_digest"): - gate_request["action_digest"] = check_request["action_digest"] - # Forward the `tool_arguments` bag alongside `tool` so - # the gate can hash it via `signature::compute_schema_hash` - # and write the fingerprint into `mcp_tool_signatures`. - # Legacy SDKs never set this; the backend's gate falls - # back to `tool_params` when the field is missing, so - # legacy callers do not regress. The shape is - # `Optional[dict[str, Any]]` -- the backend - # canonicalises the JSON before hashing, so field - # ordering inside the dict does not affect the - # fingerprint. - if "tool_arguments" in check_request and check_request["tool_arguments"] is not None: - gate_request["tool_arguments"] = check_request["tool_arguments"] - # Execution Graph v0 (2026-08-06, backend): additive - _parent_execution_id = check_request.get("parent_execution_id", parent_execution_id) - if _parent_execution_id is not None: - gate_request["parent_execution_id"] = _parent_execution_id - - # 2026-07-02 (v0.11.0 refactor): route through the canonical - body = _signed_request_body(gate_request) - headers = self._build_signed_headers(body=body) - - # NR-006 (audit 2026-08-24): wrap the gate POST in - # ``_retry_with_backoff`` with ``retry_on_5xx=True`` and - # ``max_retries=3`` (per audit recommendation: "less than - # 10 — /gate is critical and too many retries amplify - # load"). Pre-fix this code path returned a synthetic block - # on the FIRST 5xx — the agent caller never received a real - # gate decision, violating CLAUDE.md §4 "fail-CLOSED ≠ - # fail-NO-CHECK". A transient 503 from a rolling deploy - # would silently flip every agent to "budget blocked" even - # though the budget was fine. - def _do_gate_post() -> httpx.Response: - return self._client.post( - f"{self.api_url}/api/v1/gate", - content=body, - headers=headers, - timeout=5.0, - ) - - try: - response = _retry_with_backoff( - _do_gate_post, - max_retries=3, - base_delay=0.5, - max_delay=10.0, - backoff_factor=2.0, - jitter=0.1, - retry_on_5xx=True, - on_transport_error=on_transport_error, - ) - - if response.status_code == 200: - return response.json() # type: ignore[no-any-return] - # 4xx always -> synthetic block (real gate decision, - # never retried by ``_retry_with_backoff``). 5xx after - # retry exhaustion -> synthetic block (legacy - # fallback path preserved). - if response.status_code >= 500 and on_transport_error == "raise": - # Defence-in-depth: the helper raises 5xx-with-raise - # inside the retry loop, but if a path slips through - # (e.g. operator passes on_transport_error after - # exhaustion), we still surface the typed error - # rather than the silent synthetic block. - raise NullRunTransportError( - f"Gateway returned {response.status_code}", - source=TransportErrorSource.GATEWAY_ERROR, - endpoint="check", - status_code=response.status_code, - ) - return { - "decision": "block", - "decision_source": DecisionSource.FALLBACK, - "reservation_id": None, - "remaining_budget_cents": 0, - "projected_cost_cents": 0, - "explanations": [f"Gate endpoint returned {response.status_code}"], - "suggestions": ["Check API availability"], - } - except httpx.RequestError as e: - # NR-006: ``_retry_with_backoff`` re-raises network errors - # after retry exhaustion as ``BreakerTransportError``, but - # ``httpx.RequestError`` can still surface when the helper - # raises mid-loop on a non-retryable path (e.g. caller - # passes ``max_retries=0``). Translate to either a - # typed ``NullRunTransportError`` (opt-in) or a synthetic - # block (legacy). - if on_transport_error == "raise": - raise NullRunTransportError( - f"Network error on /check: {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="check", - ) from e - logger.warning(f"Gate request failed: {e}") - return { - "decision": "block", - "decision_source": DecisionSource.FALLBACK, - "reservation_id": None, - "remaining_budget_cents": 0, - "projected_cost_cents": 0, - "explanations": [f"Gate request failed: {e}"], - "suggestions": ["Check API availability"], - } - except BreakerTransportError as e: - # NR-006: the helper exhausted the retry budget on network - # errors and re-raised as ``BreakerTransportError``. Apply - # the same translation rule as ``httpx.RequestError`` - # above so the legacy ``on_transport_error`` opt-in - # contract is preserved — opt-in → typed error, default - # → synthetic block. - if on_transport_error == "raise": - raise NullRunTransportError( - f"Network error on /check after retry exhaustion: {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="check", - ) from e - logger.warning(f"Gate request failed after retries: {e}") - return { - "decision": "block", - "decision_source": DecisionSource.FALLBACK, - "reservation_id": None, - "remaining_budget_cents": 0, - "projected_cost_cents": 0, - "explanations": [f"Gate request failed after retries: {e}"], - "suggestions": ["Check API availability"], - } - - # ============================================================================= - # WebSocket Connection - # ============================================================================= - - async def connect_websocket( - self, - organization_id: str, - on_state_change: Callable[[dict[str, Any]], None] | None = None, - on_policy_invalidated: Callable[[str, str, int], None] | None = None, - on_key_rotated: Callable[[str, str, int], None] | None = None, - on_approval_resolved: Callable[[dict[str, Any]], None] | None = None, - ) -> "WebSocketConnection": - """ - Connect to WebSocket control plane for real-time workflow state updates. - - This replaces polling GET /status/{workflow_id} with WebSocket push. - When the workflow state changes (KILL/PAUSE), the server pushes the update. - - Args: - organization_id: Organization identifier - on_state_change: Optional callback for state change notifications - on_policy_invalidated: Optional callback for policy cache invalidation. - When called, clears local policy cache so next - gate/execute fetches fresh policy from backend. - Args: (organization_id, policy_id, new_version) - on_key_rotated: Optional callback for HMAC key rotation. - When called, should re-fetch secret_key from /auth/verify. - Args: (organization_id, key_id, new_version) - - Returns: - WebSocketConnection instance - - Raises: - ConnectionError: If WebSocket connection fails - """ - # Build the WS URL via urllib.parse instead of string - # replace. Reject unknown schemes with a clear error. - from urllib.parse import urlparse, urlunparse - - from nullrun.transport_websocket import WebSocketConnection - - parsed = urlparse(self.api_url) - if parsed.scheme not in ("http", "https"): - raise ValueError(f"Unsupported scheme for control plane: {parsed.scheme!r}") - ws_scheme = "wss" if parsed.scheme == "https" else "ws" - ws_url = urlunparse( - parsed._replace( - scheme=ws_scheme, - path=f"/ws/control/{organization_id}", - params="", - query="", - fragment="", - ) - ) - - # WS upgrade is a GET-with-no-body so the signed-headers helper (which - # adds HMAC for the body) does not fit. Use the GET helper instead — - # same Content-Type + X-API-Key + Authorization + X-NULLRUN-PROTOCOL - # + trace context shape, no HMAC. The backend's protocol middleware - # runs on the WS upgrade path too, so the header is mandatory here. - headers = self._auth_headers_for_get() - - # 0.7.0 thin client: no local policy cache; the next /gate or /execute - # call re-reads from the backend. Just forward the notification. - async def wrapped_policy_invalidated(ws_id: str, policy_id: str, new_version: int) -> None: - logger.info(f"Policy {policy_id} invalidated (v{new_version})") - if on_policy_invalidated: - on_policy_invalidated(ws_id, policy_id, new_version) - - async def wrapped_key_rotated(ws_id: str, key_id: str, new_version: int) -> None: - logger.info(f"Key {key_id} rotated (v{new_version}), re-fetching credentials") - await self._refetch_credentials() - if on_key_rotated: - on_key_rotated(ws_id, key_id, new_version) - - # Synchronous adapter: dispatch is dict-only, not awaitable. An - # async def would produce a coroutine the handler ignores. - def wrapped_approval_resolved(payload: dict[str, Any]) -> None: - if on_approval_resolved: - on_approval_resolved(payload) - - conn = WebSocketConnection( - url=ws_url, - headers=headers, - api_key=self.api_key, - secret_key=self.secret_key, - on_state_change=on_state_change, - on_policy_invalidated=wrapped_policy_invalidated, - on_key_rotated=wrapped_key_rotated, - on_approval_resolved=wrapped_approval_resolved, - ) - await conn.connect() - return conn - - async def _refetch_credentials(self) -> None: - """Re-fetch credentials from /auth/verify after key rotation. - - Routes through ``self._client`` so the same TLS configuration, - connection pool, and HMAC signing path apply. Body is serialised via - ``_signed_request_body`` so the wire bytes match the signed bytes. - """ - try: - payload = {"api_key": self.api_key} - body = _signed_request_body(payload) - headers = self._build_signed_headers(body=body) - - response = self._client.post( - # P0 #5: contract drift — other auth-verify call sites - # in this file use `/api/v1/auth/verify` (see runtime.py:599). - # Align this rotation call site to the same v1 prefix so the - # contract-drift-guard CI catches future divergence. - f"{self.api_url}/api/v1/auth/verify", - content=body, - headers=headers, - timeout=10.0, - ) - if response.status_code == 200: - data = response.json() - new_secret = data.get("secret_key") - if new_secret: - logger.info("Successfully fetched new secret_key from /auth/verify") - self.secret_key = new_secret - else: - logger.warning("/auth/verify did not return secret_key in response") - else: - logger.warning(f"Failed to refetch credentials: {response.status_code}") - except Exception as e: - logger.error(f"Error refetching credentials: {e}") - - # ============================================================================= - # Wire-protocol v3 endpoints - # ============================================================================= - # - # The v3 wire contract adds six endpoints that the legacy /gate + - # /execute + /track/batch surface does not cover. Each new method - # follows the same shape as the existing `check` method: - # - # 1. Build headers via ``_build_signed_headers`` (gets X-API-Key + - # Authorization + X-NULLRUN-PROTOCOL + HMAC + trace context). - # 2. Serialise the body via ``_signed_request_body`` so the wire - # bytes match the HMAC-signed bytes. - # 3. POST through the shared ``self._client`` (mTLS, connection - # pool, circuit breaker all apply). - # 4. Map non-2xx responses through ``_parse_v3_error_envelope`` - # so callers can ``except NullRunBudgetError`` / ``except - # NullRunConsumeOverbudgetError`` / etc. without parsing the - # raw error_code string. - - def check_v3( - self, - request: dict[str, Any], - on_transport_error: Callable[[Exception], dict[str, Any]] | str | None = None, - ) -> dict[str, Any]: - """Pre-execution gate — wire-protocol v3 (B1 fix 2026-07-04). - - Pre-fix this method POSTed to ``/api/v1/check``. That endpoint - was removed on 2026-06-27 — the handler now returns - ``410 Gone`` with a ``replacement: /api/v1/gate`` hint. The - SDK's ``check `` method already targets ``/api/v1/gate`` and - forwards every v3 wire field — ``chain_id`` - ``chain_op``, ``idempotency_key``, ``stream``. This method - is kept as a v3-named alias so existing call sites and tests - continue to work; internally it delegates to ``check `` with - the same body. - - Args: - request: Gate request body. Must include ``organization_id`` - ``execution_id`` (for backward compat — server mints its - own on /check), ``operation_id``, and ``check_type``. - on_transport_error: Mirrors the ``check `` flag. - - Returns: - Parsed JSON dict, augmented with ``decision_source = - DecisionSource.GATEWAY`` so callers distinguish it from a - fallback synthetic response. - - Raises: - NullRunAuthenticationError: 401/403 (PROTOCOL_TOO_OLD - PROTOCOL_TOO_NEW, API_KEY_REVOKED, CHAIN_CROSS_ORG). - NullRunConsumeOverbudgetError: 422 (placeholder for /track - not raised on /gate). - NullRunBudgetError: 402 BUDGET_HARD_BLOCKED / - BUDGET_SOFT_BLOCKED / BUDGET_OVERDRAFT_EXCEEDED. - NullRunChainError: 402 CHAIN_MAX_DURATION_EXCEEDED / - 403 CHAIN_ORG_MISMATCH. - NullRunWorkflowInactiveError: 403 WORKFLOW_INACTIVE. - NullRunBackendError: 5xx / BUDGET_DATA_UNAVAILABLE / - RATE_LIMIT_REDIS_UNAVAILABLE. - """ - # 2026-07-04 (B1): /api/v1/check returns 410 Gone. - return self.check(request, on_transport_error=on_transport_error) - - def track_single( - self, - request: dict[str, Any], - ) -> dict[str, Any]: - """POST /api/v1/track — wire-protocol v3 single-event consume. - - . The single-event path is the v3 - replacement for the legacy `/api/v1/track/batch` POST body. - It runs the CONSUME_SCRIPT invariant - ``actual_cost <= reserved_cents + epsilon_cents`` (§25 - ADR-005) and rejects with 422 CONSUME_OVERBUDGET on - violation. The reserved binding is the one created by the - matching ``/check`` call (same ``reservation_id``). - - The wire shape is built by ``runtime._build_v3_track_payload`` - (see ``runtime.py:2679-2776``); this method just forwards - whatever dict the caller hands it. The post-fix schema is: - - Args: - request: Consume request body. Must include: - - * ``reservation_id`` (str, server-minted uuidv7 from - the matching /check response — wired via - ``_capture_server_minted_execution_id``) - * ``workflow_id`` (str, the workflow the call belongs to) - * ``tokens`` (int, sum of input + output tokens) - * ``cost_cents`` (int, ``0`` — backend computes the - authoritative cost from tokens + the org's - pricing policy; sending a wrong number risks - double-billing, see _WIRE_STRIP_FIELDS in runtime.py) - * ``cost_source`` (str, ``"provisional"`` / - ``"authoritative"`` per — SDK always emits - ``"provisional"``) - - Optional fields: ``input_tokens``, ``output_tokens`` - ``model``, ``latency_ms``, ``metadata``, ``trace_id`` - ``span_id``, ``agent_id``, ``environment`` - ``agent_type``, ``attempt_index``, ``is_retry`` - ``idempotency_key``. - - Returns: - Parsed JSON dict from the backend's TrackResponse. - NOTE: there is NO top-level ``status`` field on the - wire — the legacy pre-v3 docstring claimed one, but - v3/v4 backends emit - ``{snapshot, actions_taken, processing_mode, - cost_source, confidence, event_id, - idempotent_replay, stored_response?}``. SDK callers - branch on the HTTP status (200 vs 4xx/5xx) and on - ``idempotent_replay`` (bool) for replay detection — - do NOT read ``data["status"]`` (KeyError on every - backend >= 3.66.2). - - Raises: - NullRunConsumeOverbudgetError: 422 CONSUME_OVERBUDGET — - ``actual_cost > reserved + epsilon_cents``. The - reservation is NOT silently re-reserved. - NullRunBackendError: 503 RESERVATION_NOT_FOUND / - EXECUTION_NOT_BOUND. - NullRunAuthenticationError: 401/403. - - 2026-07-04 (B2): pre-fix this docstring (and the - surrounding module comment) described a fictitious wire - shape ``{execution_id, actual_cost_cents, api_key_id - cost_source}``. The backend's actual ``TrackRequestRaw`` is - ``{workflow_id, tokens, cost_cents,...}``; ``execution_id`` - is replaced by ``reservation_id``, ``actual_cost_cents`` is - replaced by ``cost_cents`` (the SDK always sends 0 — see - ``_WIRE_STRIP_FIELDS``), and ``api_key_id`` is derived - server-side from the request auth, not supplied by the SDK. - The docstring now matches the real wire contract. - """ - # 2026-07-06 (bug-fix): the previous shape called - body = _signed_request_body(request) - headers = self._build_signed_headers(body=body) - - try: - response = self._client.post( - f"{self.api_url}/api/v1/track", - content=body, - headers=headers, - timeout=5.0, - ) - except httpx.RequestError as e: - raise NullRunTransportError( - f"Network error on /track: {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="track", - ) from e - - if response.status_code == 200: - return response.json() # type: ignore[no-any-return] - - raise _parse_v3_error_envelope(response, "track") - - def cancel( - self, - execution_id: str, - reason: str | None = None, - ) -> dict[str, Any]: - """POST /api/v1/cancel — cancel an in-flight execution. - - . The server uses - ``cancel:{execution_id}`` SETNX to deduplicate repeated - cancellations: a 200 OK response is idempotent. A - non-existent ``execution_id`` returns 404 — we surface it - as ``NullRunBackendError`` because retrying with the same - id is not a valid recovery path (the execution already - terminated). - - Args: - execution_id: Server-minted id from the matching /check - response. - reason: Optional human-readable reason for the - cancellation (audit trail). - - Returns: - Parsed JSON dict from the backend's CancelResponse. - NOTE: there is NO top-level ``status`` field on the - wire — the legacy pre-v3 docstring claimed one. - v3/v4 backends emit - ``{execution_id, canceled_at, reservation_released_cents, - already_canceled}``. SDK callers branch on the HTTP - status only — do NOT read ``data["status"]`` - (KeyError on every backend >= 3.66.2). - """ - request: dict[str, Any] = {"execution_id": execution_id} - if reason: - request["reason"] = reason - - # 2026-07-06 (bug-fix): same body-before-headers reorder as - body = _signed_request_body(request) - headers = self._build_signed_headers(body=body) - - try: - response = self._client.post( - f"{self.api_url}/api/v1/cancel", - content=body, - headers=headers, - timeout=5.0, - ) - except httpx.RequestError as e: - raise NullRunTransportError( - f"Network error on /cancel: {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="cancel", - ) from e - - if response.status_code == 200: - return response.json() # type: ignore[no-any-return] - - raise _parse_v3_error_envelope(response, "cancel") - - def heartbeat( - self, - chain_id: str, - ) -> dict[str, Any]: - """POST /api/v1/heartbeat — extend a chain's idle TTL. - - . The server runs - ``EXPIRE chain:{org}:{chain_id} 300`` atomically and - deduplicates repeated heartbeats via - ``heartbeat:{chain_id}:{ts_floor_30s}`` SETNX - (TTL = 35s — the 5s tail absorbs ±5s skew per). - - Recommended cadence: every 30s of wall-clock time (the - SDK's ``ping_chain`` helper wraps this method with the - time-based scheduler). Bursting heartbeats more often than - once per 30s is wasted bandwidth — the SETNX dedups them. - - Args: - chain_id: Active chain_id. - - Returns: - Parsed JSON dict (typically ``{"status": "ok" - "chain_id":..., "last_active": ts}``). - """ - request = {"chain_id": chain_id} - # 2026-07-06 (bug-fix): same body-before-headers reorder as - # track_single above. - body = _signed_request_body(request) - headers = self._build_signed_headers(body=body) - - try: - response = self._client.post( - f"{self.api_url}/api/v1/heartbeat", - content=body, - headers=headers, - timeout=5.0, - ) - except httpx.RequestError as e: - raise NullRunTransportError( - f"Network error on /heartbeat: {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="heartbeat", - ) from e - - if response.status_code == 200: - return response.json() # type: ignore[no-any-return] - - raise _parse_v3_error_envelope(response, "heartbeat") - - def chain_end( - self, - chain_id: str, - ) -> dict[str, Any]: - """Close a chain explicitly via /api/v1/gate with chain_op=end - . - - Pre-fix this method POSTed to ``/api/v1/chain/end``. That - endpoint was never registered on the backend - (``backend/src/proxy/http/routes.rs`` has zero matches for - ``chain/end`` or ``chain_end_handler``) — the only documented - way to close a chain is to POST /api/v1/gate with - ``{"chain_id": "...", "chain_op": "end"}``. The handler is - already idempotent — a no-op 200 OK for an unknown chain_id - is the documented success path. The SDK still raises through - the envelope parser on a true non-2xx so unexpected backend - regressions surface. - - Args: - chain_id: Chain to close. - - Returns: - Parsed JSON dict (typically ``{"decision": "allow" - "chain_id":...}``). - """ - # 2026-07-04 (B3): POST /api/v1/gate with - request = { - "chain_id": chain_id, - "chain_op": "end", - # execution_id is required by the backend's gate handler - # even on chain_end — the handler reads it but does not - # mint a reservation for op=end. Use a fresh uuidv7 - # call (the server ignores it on this path). - "execution_id": uuid.uuid4().hex, - } - # 2026-07-06 (bug-fix): same body-before-headers reorder as - body = _signed_request_body(request) - headers = self._build_signed_headers(body=body) - - try: - response = self._client.post( - f"{self.api_url}/api/v1/gate", - content=body, - headers=headers, - timeout=5.0, - ) - except httpx.RequestError as e: - raise NullRunTransportError( - f"Network error on /gate (chain_end): {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="chain_end", - ) from e - - if response.status_code == 200: - return response.json() # type: ignore[no-any-return] - - raise _parse_v3_error_envelope(response, "chain_end") - - def approximate_budget( - self, - organization_id: str | None = None, - ) -> dict[str, Any]: - """GET /api/v1/budget/approximate — UI-only budget estimation. - - . NEVER for enforcement — the backend stamps - ``is_approximate: true`` on every response. The endpoint - returns 503 ``BUDGET_DATA_UNAVAILABLE`` if all three sources - (Redis period counter → Postgres cost_events → last-known - cache) fail — NEVER returns 0, because a UI that displays - "≈ $0 spent" when no data is available misleads the user. - - Used by ``nullrun.cost_dashboard `` / ``examples/cost_dashboard.py`` - and the dashboard rollup panel. - - Args: - organization_id: Optional org override; defaults to the - transport's bound org via the auth/verify result. - - Returns: - Parsed JSON dict with ``current_spend_cents_estimate`` - ``is_approximate: True``, ``source`` (BudgetSource enum - string), ``confidence`` (High/Medium/Low), and - ``last_updated_at``. - - Raises: - NullRunBackendError: 503 BUDGET_DATA_UNAVAILABLE (all - sources failed) — caller should display "Data - unavailable" + retry button, NOT "$0 spent". - NullRunAuthenticationError: 401/403. - """ - # ApproximateBudget uses GET (not POST) per the wire contract - headers = self._auth_headers_for_get() - url = f"{self.api_url}/api/v1/budget/approximate" - - try: - response = self._client.get(url, headers=headers, timeout=5.0) - except httpx.RequestError as e: - raise NullRunTransportError( - f"Network error on /budget/approximate: {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="approximate_budget", - ) from e - - if response.status_code == 200: - return response.json() # type: ignore[no-any-return] - - raise _parse_v3_error_envelope(response, "approximate_budget") - - # ==================================================================== - # ADR-009 P1 — Audit log governance surface (v0.15.0) - # ==================================================================== - # Five methods exposing the /api/v1/orgs/:org_id/audit-log/* family - # of endpoints to SDK consumers. Pre-v0.15.0 SDKs had no audit - # client — operators had to curl the wire directly. Now they can - # call ``runtime.audit.list(...)`` etc. and get typed dataclasses - # back without writing JSON parsing glue. - # - # All five methods route through the same auth + protocol + - # trace-context machinery as the other Transport methods — see - # ``_auth_headers_for_get`` below. Audit reads are GET, so no - # HMAC body signing is required. - - def audit_log( - self, - organization_id: str, - query: Any | None = None, - ) -> dict[str, Any]: - """GET /api/v1/orgs/:org_id/audit-log — read governance audit log. - - Args: - organization_id: Org UUID — required because the - /audit-log endpoint is org-scoped. The runtime - proxy passes ``self.organization_id`` automatically - so direct callers rarely need to set this. - query: Optional :class:`nullrun.audit.AuditQuery` - instance describing the filter set (event_type, - decision, policy_id, execution_id, action, actor, - since, until, limit). Pass ``None`` for "all rows" - (rarely what you want — chains grow unbounded). - - Returns: - Parsed JSON dict with ``data`` (list of - AuditEntryResponse shapes) and ``meta`` (AuditLogMeta - pagination summary). Use - :func:`nullrun.audit.AuditLogPage.from_wire` to parse - into typed dataclasses. - - Raises: - NullRunBackendError: 401/403/5xx. - NullRunAuthenticationError: 401. - """ - from nullrun.audit import AuditQuery - - q: AuditQuery = query if isinstance(query, AuditQuery) else (query or AuditQuery()) - qs = q.to_query_string() - url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log" - if qs: - url = f"{url}?{qs}" - headers = self._auth_headers_for_get() - try: - response = self._client.get(url, headers=headers, timeout=10.0) - except httpx.RequestError as e: - raise NullRunTransportError( - f"Network error on /audit-log: {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="audit_log", - ) from e - if response.status_code == 200: - return response.json() # type: ignore[no-any-return] - raise _parse_v3_error_envelope(response, "audit_log") - - def audit_verify( - self, - organization_id: str, - *, - since: str | None = None, - ) -> dict[str, Any]: - """GET /api/v1/orgs/:org_id/audit-log/verify — chain integrity. - - Walks the chain forward from `since` (or from row 1 if - omitted) and re-computes content_hash + previous_hash - continuity. Returns the same payload the audit page's - "Integrity" banner reads — use - :func:`nullrun.audit.AuditVerifyResult.from_wire` to parse. - - Args: - organization_id: Org UUID — required. - since: Optional RFC3339 lower bound. With `since`, - only rows since that timestamp are walked (plus a - prior anchor row for hash continuity). Without - `since`, the full chain from row 1 is re-verified. - - Returns: - Parsed JSON dict with `verified`, `chain_valid`, - `record_count`, `first_hash`, `last_hash`, - `first_failure_reason`, `timestamp`, `hmac_checked`. - - Raises: - NullRunBackendError / NullRunAuthenticationError. - """ - params: list[tuple[str, str]] = [] - if since: - params.append(("since", since)) - qs = "&".join(f"{k}={v}" for k, v in params) - url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log/verify" - if qs: - url = f"{url}?{qs}" - headers = self._auth_headers_for_get() - try: - response = self._client.get(url, headers=headers, timeout=30.0) - except httpx.RequestError as e: - raise NullRunTransportError( - f"Network error on /audit-log/verify: {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="audit_verify", - ) from e - if response.status_code == 200: - return response.json() # type: ignore[no-any-return] - raise _parse_v3_error_envelope(response, "audit_verify") - - def audit_list_exports( - self, - organization_id: str, - ) -> list[dict[str, Any]]: - """GET /api/v1/orgs/:org_id/audit-log/export — list recent export jobs. - - Returns the raw JSON list of recent export job summaries - (last 10). Use :func:`nullrun.audit.AuditExportJob.from_wire` - to parse each entry. - - Raises: - NullRunBackendError / NullRunAuthenticationError. - """ - url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log/export" - headers = self._auth_headers_for_get() - try: - response = self._client.get(url, headers=headers, timeout=10.0) - except httpx.RequestError as e: - raise NullRunTransportError( - f"Network error on /audit-log/export (list): {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="audit_list_exports", - ) from e - if response.status_code == 200: - body = response.json() - # Wire shape is `{"exports": [...]}` per the audit export - # list handler in backend/src/proxy/http/audit.rs. - if isinstance(body, dict): - return body.get("exports", []) or [] - return body if isinstance(body, list) else [] - raise _parse_v3_error_envelope(response, "audit_list_exports") - - def audit_create_export( - self, - organization_id: str, - ) -> dict[str, Any]: - """POST /api/v1/orgs/:org_id/audit-log/export — enqueue 30-day export. - - The backend creates a job, returns ``{"job_id", "status": - "pending"}`` immediately, and processes in the background. - Poll :meth:`audit_export_status` for completion. - - The export covers the trailing 30 days; the backend hard-codes - that window today (audit.rs:692-700 — ``chrono::Utc::now() - - Duration::days(30)``). When the per-job window becomes - configurable this method will accept a `since`/`until` - override. - - Returns: - Parsed JSON dict with ``job_id`` (UUID) and ``status``. - - Raises: - NullRunBackendError / NullRunAuthenticationError. - """ - url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log/export" - headers = self._build_signed_headers(body=b"{}") - try: - response = self._client.post(url, content=b"{}", headers=headers, timeout=10.0) - except httpx.RequestError as e: - raise NullRunTransportError( - f"Network error on /audit-log/export (create): {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="audit_create_export", - ) from e - if response.status_code == 200: - return response.json() # type: ignore[no-any-return] - raise _parse_v3_error_envelope(response, "audit_create_export") - - def audit_export_status( - self, - organization_id: str, - job_id: str, - ) -> dict[str, Any]: - """GET /api/v1/orgs/:org_id/audit-log/export/:job_id/status. - - Polls a previously-enqueued export job. When ``status`` flips - to ``completed`` the ``file_url`` field carries an S3 - presigned URL (or `/tmp/...` path on dev), and an - ``error_message`` is set on the ``failed`` transition. - - Args: - organization_id: Org UUID — required. - job_id: UUID returned by :meth:`audit_create_export`. - - Returns: - Parsed JSON dict with ``job_id``, ``status``, - ``file_url``, ``record_count``, ``created_at``, - ``completed_at``, ``error_message``. - - Raises: - NullRunBackendError / NullRunAuthenticationError. - """ - url = f"{self.api_url}/api/v1/orgs/{organization_id}/audit-log/export/{job_id}/status" - headers = self._auth_headers_for_get() - try: - response = self._client.get(url, headers=headers, timeout=10.0) - except httpx.RequestError as e: - raise NullRunTransportError( - f"Network error on /audit-log/export/{job_id}/status: {e}", - source=TransportErrorSource.NETWORK_ERROR, - endpoint="audit_export_status", - ) from e - if response.status_code == 200: - return response.json() # type: ignore[no-any-return] - raise _parse_v3_error_envelope(response, "audit_export_status") - - def _auth_headers_for_get(self) -> dict[str, str]: - """Headers for an unsigned GET (no HMAC body). - - Same shape as ``_build_signed_headers`` minus the HMAC - headers. Used by ``approximate_budget`` which is a GET with - no body, so there's nothing to sign. Keeps the protocol + - CSRF-bypass + trace-context headers consistent with the - signed-POST path. - """ - headers: dict[str, str] = {"Content-Type": "application/json"} - if self.api_key: - headers["X-API-Key"] = self.api_key - headers["Authorization"] = f"Bearer {self.api_key}" - headers[HEADER_PROTOCOL] = _protocol_header_value() - self._inject_trace_context(headers) - return headers - - -# 2026-07-02 (v0.11.0): ACTIVE v3 error envelope parser. -def _extract_error_envelope( - body: Any, - raw_text: str, -) -> tuple[str, str, dict[str, Any]]: - """Pull ``(error_code, message, details)`` from any error envelope. - - Drift §3 (2026-07-06): the backend emits three distinct shapes - for non-2xx responses. This helper normalises them into the - ``(error_code, message, details)`` tuple the rest of - ``_parse_v3_error_envelope`` consumes. - - Lookup priority: - - 1. **v3 envelope** -- ``{"error_code": "BUDGET_HARD_BLOCKED", - "error_message": "...", "details": {...}, ...}``. The - canonical shape from ``gate/internal.rs`` and - ``handlers.rs::track_handler``. - - 2. **v3 mixed** -- ``{"error_code": "BUDGET_DATA_UNAVAILABLE", - "message": "...", "retry_after_ms": N}``. The 503 path - from ``budget.rs:107-112``; same v3 semantics but the - message field is called ``message`` not ``error_message``. - - 3. **Legacy slug** -- ``{"error": "chain_not_extendable", - "message": "...", "chain_state": "..."}``. From - ``heartbeat.rs:199-205`` and the ``ApiError`` path on - ``cancel.rs``. The slug is lowercased and SCREAMING_SNAKE'd - so it matches ``_V3_ERROR_CODE_MAP`` lookups. - - 4. **Plaintext** -- ``response.text`` containing a free-form - error string (heartbeat.rs:157, heartbeat.rs:166). No JSON, - so ``body`` is empty. - - Args: - body: Parsed JSON body from the response (``{}`` on parse - failure or non-JSON content). - raw_text: Raw ``response.text`` fallback for plaintext - envelopes. - - Returns: - ``(backend_code, message, details)`` where: - - * ``backend_code`` is uppercase SCREAMING_SNAKE if it - originated from the v3 envelope, or the lowercased slug - otherwise. The mapping table keys are uppercase; the - dispatcher lowercases the lookup key before consulting - the map. - * ``message`` is the human-readable string for the - exception class. Falls back to ``raw_text`` if no JSON - body. - * ``details`` is the machine-readable context payload - (``details: {...}`` on the v3 envelope, all other - JSON fields flattened on the legacy slug, ``{}`` on - plaintext). - """ - if not isinstance(body, dict) or not body: - # No JSON body -- plaintext error envelope. - # Heartbeat's 404 "chain not found" and 403 - # "chain org mismatch" land here. - return ("", raw_text or "", {}) - - # Shape 1: v3 envelope. - if "error_code" in body: - code = str(body.get("error_code", "") or "") - # The 503 budget path uses "message" instead of - # "error_message". Accept both. - message = str(body.get("error_message") or body.get("message") or raw_text or "") - details_raw = body.get("details") or {} - if not isinstance(details_raw, dict): - details_raw = {} - # Forward any extra top-level fields that look like - # context (e.g. ``chain_state`` on heartbeat 409) into - # details so downstream code can introspect them. - details: dict[str, Any] = dict(details_raw) - for key, value in body.items(): - if key in ( - "error_code", - "error_message", - "message", - "details", - "retry_after_ms", - ): - continue - details.setdefault(key, value) - return (code, message, details) - - # Shape 2: legacy slug. ``error`` is the slug, - # ``message`` is the human-readable string. - if "error" in body: - slug = str(body.get("error", "") or "") - message = str(body.get("message", "") or raw_text or "") - # Convert the legacy lowercase slug to uppercase - # SCREAMING_SNAKE so the mapping table can find it. - code = slug.upper() - # Everything except ``error`` and ``message`` goes into - # details for diagnostic context. - details = { - k: v for k, v in body.items() if k not in ("error", "message") and not k.startswith("_") - } - return (code, message, details) - - # JSON body but not a recognised envelope shape. Pass through. - return ("", raw_text or str(body), dict(body) if isinstance(body, dict) else {}) - - -def _safe_json(response: httpx.Response, endpoint: str) -> Any: - """Parse a response body as JSON, wrapping parse failures. - - DEF-ERRHDL-INVALID-JSON-01 (2026-08-11, RUN_ID 20260811-1): the SDK - previously propagated ``json.JSONDecodeError`` unchanged to user - code, which leaks internal file paths and the raw broken payload - fragment in tracebacks. This helper wraps the parse failure in - NullRunTransportError with a stable ``error_code`` so callers can - ``except`` cleanly and the user sees a short NullRun-family - message instead of a Python traceback. - - ``body_preview`` is intentionally truncated to 200 chars and the - raw ``JSONDecodeError.lineno/colno`` are NOT included in the - surfaced message -- both are info-leak surface (line numbers - hint at response shape; partial body may carry PII like - organization_id fragments). - """ - try: - return response.json() - except (json.JSONDecodeError, ValueError) as exc: - # Body preview capped at 200 chars; truncated to avoid - # flooding logs / exception chain. - try: - body_preview = (response.text or "")[:200] - except Exception: - body_preview = "" - raise NullRunTransportError( - f"Received malformed JSON from {endpoint} " - f"(status={response.status_code}): {type(exc).__name__}", - source=TransportErrorSource.GATEWAY_ERROR, - endpoint=endpoint, - error_code="NR-T001", - ) from exc - - -def _parse_v3_error_envelope( - response: httpx.Response, - endpoint: str, -) -> Exception: - """Translate a non-2xx ``httpx.Response`` into the right v3 - SDK exception. - - The backend returns errors as a JSON envelope of the shape - ``{"error_code": "BUDGET_HARD_BLOCKED", "error_message": "..." - "details": {...}, "retry_after_ms": N}``. The - parser maps the backend's ``error_code`` string to the closest - SDK exception class, attaching the structured envelope fields - as instance attributes so callers can introspect them. - - Mapping table lives at ``_V3_ERROR_CODE_MAP`` below — keep the - helper as a thin dispatcher. - """ - # Lazy imports: the exception classes import the transport - # types (TransportErrorSource), so a top-level import here - # would create a cycle. The price is one extra import - # non-2xx response — irrelevant for the failure path. - from nullrun.breaker.exceptions import ( - NullRunApprovalDeniedError, - NullRunApprovalDigestMismatchError, - NullRunApprovalExpiredError, - NullRunApprovalNotYetApprovedError, - NullRunApprovalReplayRejectedError, - NullRunApprovalToolDigestMismatchError, - NullRunAuthError, - NullRunBackendError, - NullRunBlockedException, - NullRunBudgetError, - NullRunBudgetRecheckFailedError, - NullRunChainError, - NullRunConsumeOverbudgetError, - NullRunDecision, - NullRunInfrastructureError, - NullRunProtocolError, - NullRunRateLimitRedisError, - NullRunToolBlockedError, - NullRunWorkflowInactiveError, - RateLimitError, - ) - - status = response.status_code - try: - body = response.json() - except Exception: - body = None - if not isinstance(body, dict): - body = {} - - # Drift §3 (2026-07-06): the wire envelope is NOT one shape. - backend_code, message, details = _extract_error_envelope(body, response.text) - retry_after_ms: float | None = body.get("retry_after_ms") if isinstance(body, dict) else None - # Retry-After header takes precedence over the JSON field when - # both are present (server-side convention — header is canonical - # per RFC 7231, JSON is a NullRun-specific fallback). - retry_after_header = response.headers.get("Retry-After") - if retry_after_header: - try: - retry_after_ms = float(retry_after_header) * 1000.0 - except ValueError: - # HTTP-date form is non-numeric — leave JSON value intact. - pass - - # Per-class dispatcher. Each exception has its own constructor - # signature (RateLimitError requires source+endpoint - # NullRunBackendError requires endpoint+status_code, etc.) so a - # uniform ``error_cls(**kwargs)`` does not work. The switches - # below mirror the exact field mapping from. - full_message = f"{endpoint}: {message}" - - if backend_code == "PROTOCOL_TOO_OLD" or backend_code == "PROTOCOL_TOO_NEW": - # NullRunProtocolError → NullRunInfrastructureError → - # NullRunError base. Base constructor does NOT accept - # a generic ``details=`` kwarg. Pass message only — the - # catalog value already encodes error_code + retryable. - return NullRunProtocolError(full_message) - - if backend_code == "CONSUME_OVERBUDGET": - return NullRunConsumeOverbudgetError( - full_message, - execution_id=details.get("execution_id"), - reserved_cents=details.get("reserved_cents"), - max_allowed_cents=details.get("max_allowed_cents"), - actual_cost_cents=details.get("actual_cost_cents"), - epsilon_cents=details.get("epsilon_cents"), - status_code=status, # 422 per backend mapping - ) - - if ( - backend_code == "CHAIN_MAX_DURATION_EXCEEDED" - or backend_code == "CHAIN_CROSS_ORG" - or backend_code == "CHAIN_ORG_MISMATCH" - ): - return NullRunChainError( - full_message, - chain_id=details.get("chain_id"), - backend_code=backend_code, - details=details, - status_code=status, # 402/403 per backend mapping - ) - - if backend_code == "WORKFLOW_INACTIVE": - return NullRunWorkflowInactiveError( - full_message, - workflow_id=details.get("workflow_id"), - status_code=status, # 403 per backend mapping - ) - - if backend_code == "BUDGET_RECHECK_FAILED": - # H6 / 2026-08-12 audit: dedicated typed dispatch so callers - # can branch on the post-approval recheck failure (NR-B006) - # vs a fresh /gate block (NR-B004). The dispatcher surfaces - # ``current_spend_cents`` / ``budget_cents`` from the wire - # envelope so callers can compute the remaining cap and - # decide whether to retry after re-/gate. - return NullRunBudgetRecheckFailedError( - full_message, - current_spend_cents=details.get("current_spend_cents"), - budget_cents=details.get("budget_cents"), - status_code=status, # 402 per backend mapping - ) - - if backend_code in ( - "APPROVAL_NOT_YET_APPROVED", - "APPROVAL_DENIED", - "APPROVAL_EXPIRED", - "APPROVAL_DIGEST_MISMATCH", - "APPROVAL_TOOL_DIGEST_MISMATCH", - "APPROVAL_REPLAY_REJECTED", - ): - # v3.53 / 2026-08-13 audit, A-1+A-2 bundle: dedicated typed - # dispatch so callers can branch on the precise grant-consume - # outcome. Pre-v3.53 the SDK fell through to the catalog - # fallback path which called ``catalog(full_message, **details)`` - # — NullRunBlockedException subclasses reject that signature - # (they need workflow_id as positional arg) so the catch-all - # path raised TypeError instead of the typed exception. - # Post-v3.53 each of the six codes maps to its own NR-Axxx - # subclass (NR-A010..NR-A015). Wire details carry the - # approval_id and the typed exception's NR-Axxx catalog - # value (via the class attribute) so cookbook recipes can - # ``except NullRunApprovalDeniedError:`` for terminal - # surface-to-user, ``except - # NullRunApprovalNotYetApprovedError:`` for wait/poll, - # ``except NullRunApprovalReplayRejectedError:`` for - # retry-loop detection, etc. - catalog = _V3_ERROR_CODE_MAP[backend_code] - return catalog( # type: ignore[call-arg] - workflow_id=str(details.get("workflow_id") or "unknown"), - reason=full_message, - status_code=status, # 403 per backend mapping - approval_id=details.get("approval_id"), - ) - - if backend_code == "RATE_LIMIT_REDIS_UNAVAILABLE": - # NullRunRateLimitRedisError → NullRunInfrastructureError - # → NullRunError base. Base constructor accepts only - # message + (error_code, user_action, retryable, docs_url - # cause) — NOT a generic ``details=``. The catalog value - # already encodes error_code + retryable, so we just pass - # the message. - return NullRunRateLimitRedisError(full_message) - - if backend_code == "RATE_LIMIT_EXCEEDED": - retry_after = retry_after_ms / 1000.0 if retry_after_ms else None - return RateLimitError( - full_message, - source=TransportErrorSource.GATEWAY_ERROR, - endpoint=endpoint, - retry_after=retry_after, - body=body, - ) - - # Catalog codes that map to NullRunBudgetError / NullRunBackendError - # via the fallback shape (no special signature). - catalog = _V3_ERROR_CODE_MAP.get(backend_code) - if catalog is not None: - # Special-case each constructor signature — the NullRun - # hierarchy has heterogeneous constructors (workflow_id + - # reason for NullRunBlockedException, endpoint + status_code - # for NullRunBackendError, error_code/user_action for - # NullRunError base). Universal ``catalog(message, details=)`` - # would trip one of them every time. - if catalog is NullRunBackendError: - return NullRunBackendError( - full_message, - endpoint=endpoint, - status_code=status, - ) - if catalog is NullRunExecutionNotFoundError: - # 2026-09-09 audit: dedicated dispatch so callers can - # read ``execution_id`` / ``endpoint`` / ``regate_required`` - # off the exception without indexing into ``details``. - # Mirrors the ``NullRunBackendError`` branch above (the - # parent class) but also forwards ``execution_id`` from - # the wire envelope. Without this branch the generic - # catalog fallback at line ~2615 would discard the - # ``execution_id`` field (it filters ``**details`` to - # the base NullRunError kwargs only). - return NullRunExecutionNotFoundError( - full_message, - execution_id=details.get("execution_id"), - endpoint=details.get("endpoint") or endpoint, - status_code=status, # 404 per backend mapping - ) - if catalog is NullRunBudgetError: - # NullRunBudgetError → NullRunBlockedException → requires - return NullRunBudgetError( - workflow_id=str(details.get("workflow_id") or "unknown"), - reason=full_message, - status_code=status, - ) - if catalog is NullRunRateLimitRedisError: - # NullRunError base takes (message, error_code=, user_action= - # retryable=, docs_url=, cause=). The catalog value here - # already encodes error_code + retryable, so we pass - # the message only. - return catalog(full_message) - if catalog is NullRunProtocolError: - return catalog(full_message) - # NullRunAuthError — surface the wire error_code (one of - # v3.38's API_KEY_REVOKED / API_KEY_EXPIRED / API_KEY_DISABLED - # / API_KEY_INVALID / API_KEY_MISSING / API_KEY_MALFORMED) on - # ``self.wire_code`` so callers can branch on granular - # lifecycle state without clobbering the SDK-side - # ``error_code`` taxonomy (NR-A003). Mirrors the - # ``NullRunChainError.backend_code`` pattern. - # - # Filter ``details`` to the kwargs the base NullRunError - # constructor accepts — the envelope's ``details`` dict can - # carry arbitrary keys (``expires_at``, ``ttl_seconds``, ...) - # and the base class rejects unknown kwargs with TypeError. - # Unknown fields are stored on ``self.details`` for caller - # introspection instead. - if catalog is NullRunAuthError: - allowed = {"error_code", "user_action", "retryable", "docs_url", "cause"} - forwarded = {k: v for k, v in details.items() if k in allowed} - extra = {k: v for k, v in details.items() if k not in allowed} - instance = NullRunAuthError( - full_message, - wire_code=backend_code, - **forwarded, - ) - if extra: - instance.details = extra # type: ignore[attr-defined] - return cast(Exception, instance) - # Final fallback for catalog classes with a generic - # (message, **details) signature (NullRunWorkflowInactiveError - # and any future addition). - # The details payload is forwarded as a positional kwarg - # via **details (typed as Any to satisfy mypy since - # type[BaseException] does not expose the kwargs the - # catalog subclasses actually accept). - # - # The catalog lookup produces type[BaseException] (the - # union of all class objects), but every entry in - # _V3_ERROR_CODE_MAP is a real Exception subclass. Cast - # to Exception so mypy stops flagging the return value - # as BaseException (the helper declares -> Exception). - allowed = {"error_code", "user_action", "retryable", "docs_url", "cause"} - forwarded = {k: v for k, v in details.items() if k in allowed} - if ( - catalog is NullRunToolBlockedError - or catalog is NullRunBlockedException - ): - # DEF-NR-TOOLBLOCKED-PARSER (2026-09-10): NullRunBlockedException - # subclasses require positional ``workflow_id`` + ``reason`` - # (no defaults), so the generic ``catalog(full_message, ...)`` - # fallback below raises TypeError when given a string for - # ``workflow_id``. Affects 7 catalog entries: TOOL_BLOCKED, - # LOOP_DETECTED, MODEL_REQUIRED, POLICY_UNCONFIGURED, - # TOO_MANY_PENDING_APPROVALS, BUSINESS_IMPACT_INVALID, - # VALIDATION_FAILED. Pre-fix the TypeError escaped the parser - # and got swallowed by the catch-all ``except Exception: pass`` - # in Transport.execute, surfacing the synthetic-block dict - # ``{"decision": "block", "explanation": "Gateway returned - # 403"}`` instead of the typed NR-T001 / NR-Lxxx catalog line. - # ``tool_name`` is forwarded for NullRunToolBlockedError - # (the only BlockedException subclass that surfaces it on the - # wire envelope); the parent constructor drops it for plain - # NullRunBlockedException so it's a no-op there. ``forwarded`` - # (error_code / user_action / retryable / docs_url / cause) is - # passed through so the catalog value's defaults win. - instance = catalog( # type: ignore[call-arg] - workflow_id=str(details.get("workflow_id") or "unknown"), - reason=full_message, - status_code=status, - tool_name=details.get("tool_name"), - **forwarded, - ) - return cast(Exception, instance) - instance = catalog(full_message, **forwarded) # type: ignore[call-arg] - return cast(Exception, instance) - - # Fallback — use HTTP status. The catalog may not yet cover - # every backend code, so we surface a typed backend error - # that exposes status_code + error_code for the caller. - if status in (401, 403): - return NullRunAuthenticationError( - f"Auth failed on {endpoint} (status {status}, error_code={backend_code!r}): {message}" - ) - if status == 429: - retry_after = retry_after_ms / 1000.0 if retry_after_ms else None - return RateLimitError( - f"Rate limited on {endpoint} (status 429, error_code={backend_code!r}): {message}", - source=TransportErrorSource.GATEWAY_ERROR, - endpoint=endpoint, - retry_after=retry_after, - body=body, - ) - if 500 <= status < 600: - return NullRunBackendError( - f"{endpoint}: {message} (status {status}, error_code={backend_code!r})", - endpoint=endpoint, - status_code=status, - ) - return NullRunBackendError( - f"{endpoint}: {message} (status {status}, error_code={backend_code!r})", - endpoint=endpoint, - status_code=status, - ) - - -# Lazy import to avoid a hard dependency at module import time. -# `_parse_v3_error_envelope` is a module-level helper; the exception -# classes live in `nullrun.breaker.exceptions`. Importing here -# (rather than at the top of transport.py) keeps the legacy import -# graph identical and avoids breaking the frozen -# ``_parse_error_envelope`` test contract. -def _build_v3_error_code_map() -> dict[str, type[Exception]]: - """Construct the v3 error_code → exception class mapping. - - Imported lazily because the exception classes import the - transport types (TransportErrorSource), which would create a - circular import if loaded eagerly at the top of transport.py. - """ - from nullrun.breaker.exceptions import ( - NullRunApprovalDeniedError, - NullRunApprovalDigestMismatchError, - NullRunApprovalExpiredError, - NullRunApprovalNotYetApprovedError, - NullRunApprovalReplayRejectedError, - NullRunApprovalToolDigestMismatchError, - NullRunAuthError, - NullRunBackendError, - NullRunBlockedException, - NullRunBudgetError, - NullRunBudgetRecheckFailedError, - NullRunChainError, - NullRunConsumeOverbudgetError, - NullRunExecutionNotFoundError, - NullRunProtocolError, - NullRunRateLimitRedisError, - NullRunToolBlockedError, - NullRunWorkflowInactiveError, - RateLimitError, - ) - - return { - # 400 — protocol mismatch - "PROTOCOL_TOO_OLD": NullRunProtocolError, - "PROTOCOL_TOO_NEW": NullRunProtocolError, - # 402 — budget family - "BUDGET_HARD_BLOCKED": NullRunBudgetError, - "BUDGET_SOFT_BLOCKED": NullRunBudgetError, - "BUDGET_OVERDRAFT_EXCEEDED": NullRunBudgetError, - "BUDGET_PERIOD_NOT_STARTED": NullRunBudgetError, - "REDIS_UNAVAILABLE": NullRunBudgetError, - # 402 — chain family (separate class for diagnostic clarity) - "CHAIN_MAX_DURATION_EXCEEDED": NullRunChainError, - # 403 — chain security + workflow state - "CHAIN_CROSS_ORG": NullRunChainError, - "CHAIN_ORG_MISMATCH": NullRunChainError, - # 403 — Execution Graph v0 (2026-08-06, backend). Sub-agent - "PARENT_EXECUTION_NOT_FOUND": NullRunChainError, - "PARENT_EXECUTION_ORG_MISMATCH": NullRunChainError, - "PARENT_EXECUTION_KEY_MISMATCH": NullRunChainError, - "WORKFLOW_INACTIVE": NullRunWorkflowInactiveError, - # 401/403 — auth (v3.38 distinct lifecycle states). - # The backend splits the v3.36 ``API_KEY_REVOKED`` bucket into - # five distinct wire codes so SDKs can branch on each state - # (e.g. surface "rotate this key" vs "this key was admin- - # disabled" vs "no Authorization header was sent"). All map - # to NullRunAuthError — diagnostic class is preserved; the - # granular codes live in ``details.error_code`` and are - # surfaced via NullRunAuthError.code for handler dispatch. - "API_KEY_REVOKED": NullRunAuthError, - "API_KEY_EXPIRED": NullRunAuthError, - "API_KEY_DISABLED": NullRunAuthError, - "API_KEY_INVALID": NullRunAuthError, - "API_KEY_MISSING": NullRunAuthError, - "API_KEY_MALFORMED": NullRunAuthError, - # 422 — consume invariant violation - "CONSUME_OVERBUDGET": NullRunConsumeOverbudgetError, - # 429 — rate limit - "RATE_LIMIT_EXCEEDED": RateLimitError, - # 503 — backend availability - "RATE_LIMIT_REDIS_UNAVAILABLE": NullRunRateLimitRedisError, - "BUDGET_DATA_UNAVAILABLE": NullRunBackendError, - # 402 — approval-create failure family (DEF-ARFLOW-TOOLNAME-01, - "APPROVAL_DB_UNAVAILABLE": NullRunBlockedException, - "APPROVAL_PERSISTENCE_FAILED": NullRunBlockedException, - "APPROVAL_VALIDATION_FAILED": NullRunBlockedException, - "APPROVAL_CONFLICT": NullRunBlockedException, - "APPROVAL_NOT_FOUND": NullRunBlockedException, - "APPROVAL_CREATE_FAILED": NullRunBlockedException, - # 403 — approval grant-consume outcomes (v3.53 / 2026-08-13 - # audit, A-1+A-2 bundle). Distinct from the /gate - # create-failure family above: these are the seven - # distinct outcomes that the backend's - # `gate_internal()` returns on /execute post-approval - # grant-consume (see - # `backend/src/proxy/http/gate/internal.rs:3059-3108, - # 3115-3138`). Pre-v3.53 the SDK collapsed all six - # into NullRunBlockedException — bilateral wire gap. - # Post-v3.53 each maps to a typed exception - # (NR-A010..NR-A015) so cookbook recipes can branch - # on the precise outcome (e.g. ``except - # NullRunApprovalNotYetApprovedError:`` for wait/poll, - # ``except NullRunApprovalDeniedError:`` for terminal - # surface-to-user, ``except - # NullRunApprovalReplayRejectedError:`` for retry-loop - # detection). - "APPROVAL_NOT_YET_APPROVED": NullRunApprovalNotYetApprovedError, - "APPROVAL_DENIED": NullRunApprovalDeniedError, - "APPROVAL_EXPIRED": NullRunApprovalExpiredError, - "APPROVAL_DIGEST_MISMATCH": NullRunApprovalDigestMismatchError, - "APPROVAL_TOOL_DIGEST_MISMATCH": NullRunApprovalToolDigestMismatchError, - "APPROVAL_REPLAY_REJECTED": NullRunApprovalReplayRejectedError, - # 402 — post-approval budget recheck (H6 / 2026-08-12 audit). - # Distinct from BUDGET_HARD_BLOCKED: the operator explicitly - # approved the grant at /gate, but the period-bound counter - # moved between /gate and /execute (another concurrent - # execution spent the budget). Caller should re-/gate to - # refresh the reservation envelope and retry /execute. - # Backed by GateErrorCode::BudgetRecheckFailed in the - # backend (error_codes.rs). - # 2026-09-09 audit: the per-class dispatcher in - # ``_v3_error_dispatch`` (line ~2477) already routes this to - # ``NullRunBudgetRecheckFailedError`` (NR-B006) before the - # catalog fallback — defense-in-depth, this catalog entry - # now matches the dispatcher. - "BUDGET_RECHECK_FAILED": NullRunBudgetRecheckFailedError, - # NR-007 (audit 2026-08-24): the 19 entries below were missing - # from the SDK map and caused cookbook recipes that branch on - # ``error_code`` to fall through to ``NullRunBackendError``. - # Added in the parity PR that closes NR-007 — keep this - # block grouped so the parity CI test - # ``backend/tests/nr007_sdk_error_code_parity.rs`` has a - # single regression pin surface. Family mapping rationale - # per code: - # - budget family: NullRunBudgetError - # - chain family: NullRunChainError - # - auth binding: NullRunAuthError - # - protocol / wire validation: NullRunProtocolError / - # NullRunBackendError - # - gate decision: NullRunBlockedException / - # NullRunToolBlockedError (TOOL_BLOCKED MUST use the - # dedicated class per CLAUDE.md §8 — operators expect - # ``except NullRunToolBlockedError:`` for tool-name - # branch recipes). - "BUDGET_ANTI_DOS_RESERVED_CAP": NullRunBudgetError, - "BUDGET_REDIS_UNAVAILABLE": NullRunBudgetError, - "CHAIN_ID_INVALID": NullRunChainError, - "EXECUTION_KEY_MISMATCH": NullRunAuthError, - "EXECUTION_ORG_MISMATCH": NullRunAuthError, - "ORG_MISMATCH": NullRunAuthError, - "PROTOCOL_HEADER_INVALID": NullRunProtocolError, - "PROTOCOL_HEADER_REQUIRED": NullRunProtocolError, - "TOOL_BLOCKED": NullRunToolBlockedError, - "LOOP_DETECTED": NullRunBlockedException, - "MODEL_REQUIRED": NullRunBlockedException, - "POLICY_UNCONFIGURED": NullRunBlockedException, - "TOO_MANY_PENDING_APPROVALS": NullRunBlockedException, - "BUSINESS_IMPACT_INVALID": NullRunBlockedException, - "VALIDATION_FAILED": NullRunBlockedException, - # Wire-level parsing failures (missing / malformed fields). - # Map to ``NullRunBackendError`` because the SDK treats them - # as infrastructure-side issues — the server should have - # returned a structured 4xx envelope, and a fall-through - # here indicates a wire-shape drift between client and server. - "EXECUTION_ID_MALFORMED": NullRunBackendError, - "EXECUTION_ID_REQUIRED": NullRunBackendError, - # 2026-09-09 SDK-drift audit: ``INVALID_EXECUTION_ID`` is - # emitted by the backend as a typed envelope at - # ``cancel.rs:142-149`` and ``orchestrator.rs:1327-1334`` — - # round-trips through the canonical ``v3_error_envelope`` - # helper, so the wire string is canonical. Map to - # ``NullRunBackendError`` (sibling to the EXECUTION_ID_* - # siblings above) — wire-shape drift guard. - "INVALID_EXECUTION_ID": NullRunBackendError, - # 2026-09-09 SDK-drift audit: ``EXECUTION_NOT_FOUND`` is - # emitted by the backend as a typed envelope at - # ``execute.rs:194`` and ``cancel.rs:303`` (post-DEF-SDKK-022 - # routing through ``v3_error_envelope`` + the new - # ``GateErrorCode::ExecutionNotFound`` variant). Map to the - # dedicated ``NullRunExecutionNotFoundError`` (NR-EX01) so - # cookbook code can ``except - # NullRunExecutionNotFoundError`` to distinguish a missed - # /gate (re-issue /gate then retry /execute) from generic - # wire-shape drift. - "EXECUTION_NOT_FOUND": NullRunExecutionNotFoundError, - # Rate-limit plan lookup failure (Postgres / Redis adjacent). - # Tied to ``NullRunRateLimitRedisError`` because the failure - # mode is rate-limit-specific infrastructure unavailability - # rather than generic backend error. - "RATE_LIMIT_PLAN_LOOKUP_FAILED": NullRunRateLimitRedisError, - # Idempotency layer Redis unavailability. Map to generic - # ``NullRunBackendError`` — the wire class is infrastructure - # availability, not a typed subclass (mirrors - # ``RATE_LIMIT_REDIS_UNAVAILABLE`` -> ``NullRunRateLimitRedisError`` - # family pattern at wire level). - "IDEMPOTENCY_REDIS_UNAVAILABLE": NullRunBackendError, - # Execution Graph / ADR-036 (sub-agent spawn topology). Backend - # error_codes.rs:107-382 covers six codes in this family — three - # 422 semantic rejects (cycle / depth / parent-binding) and three - # 503 infrastructure failures (depth lookup / invoke persist / - # subworkflow disabled). Map to ``NullRunChainError`` because - # the existing class already carries `parent_execution_id` per - # Execution Graph v0 docstring at `exceptions.py:388-410`. Adding - # them under a fresh ``NullRunSubworkflowError`` would force - # cookbook code to import a new exception class for the same - # lineage concept; consolidate under ChainError instead. - "WORKFLOW_CYCLE_DETECTED": NullRunChainError, - "WORKFLOW_DEPTH_EXCEEDED": NullRunChainError, - "WORKFLOW_PARENT_BINDING_EXPIRED": NullRunChainError, - "WORKFLOW_DEPTH_LOOKUP_FAILED": NullRunChainError, - "INVOKE_PERSIST_FAILED": NullRunBackendError, - "SUBWORKFLOW_INVOKE_DISABLED": NullRunChainError, - # ADR-023 (post-approval re-check race): a second operator - # already decided on the same approval row before this call's - # re-check landed. Map to ``NullRunApprovalReplayRejectedError`` - # because semantically the agent caller has the same retry-loop - # concern as a replay-rejected approval (CLAUDE.md §34c). - "APPROVAL_ALREADY_DECIDED": NullRunApprovalReplayRejectedError, - # ADR-023 (Phase-1+ wire-shape fail-CLOSED): a v3+ SDK hit /gate - # without ``action_digest`` (legacy anchor attempt). Map to - # ``NullRunBlockedException`` because the wire shape is a true - # block decision, not an infrastructure error — cookbook code - # branches on the action_digest missing path with the same - # `except NullRunBlockedException:` flow as TOOL_BLOCKED. - "LEGACY_GRANT_REJECTED": NullRunBlockedException, - } - - -_V3_ERROR_CODE_MAP: dict[str, type[Exception]] = _build_v3_error_code_map() - - -# ADR (2026-06-28, audit P2.2 close): ``_parse_error_envelope`` below -def _parse_error_envelope( - response: httpx.Response, - endpoint: str, -) -> Exception: - """Translate a non-2xx ``httpx.Response`` into the right exception - subclass per the canonical ``contracts/errors.ts`` envelope. - - 4xx/5xx/429 are mapped to distinct ``RateLimitError`` / - ``NullRunAuthenticationError`` / ``NullRunTransportError(GATEWAY_ERROR)`` - so callers branch on type instead of string-matching ``str(exc)``. - - Module-level helper (not a Transport method) so it can be called - from background threads that do not carry a Transport instance. - - **Audit F-R2-13 (2026-06-22):** no live wire path uses this. It - exists for tests only. See the comment block above. - """ - status = response.status_code - try: - body = response.json() - except Exception: - body = None - if not isinstance(body, dict): - body = {} - error_slug: str = body.get("error", "") or "" - message: str = body.get("message") or response.text or f"HTTP {status}" - - if status in (401, 403): - return NullRunAuthenticationError( - f"Auth failed on {endpoint} (status {status}, error={error_slug!r}): {message}" - ) - - if status == 429: - retry_after: float | None = None - ra_header = response.headers.get("Retry-After") - if ra_header: - try: - retry_after = float(ra_header) - except ValueError: - try: - from datetime import datetime, timezone - from email.utils import parsedate_to_datetime - - dt = parsedate_to_datetime(ra_header) - retry_after = (dt - datetime.now(timezone.utc)).total_seconds() - except Exception: - retry_after = None - upgrade_url = body.get("upgrade_url") if isinstance(body, dict) else None - return RateLimitError( - f"Rate limited on {endpoint} (status 429, error={error_slug!r}): {message}", - source=TransportErrorSource.GATEWAY_ERROR, - endpoint=endpoint, - retry_after=retry_after, - upgrade_url=upgrade_url, - body=body, - ) - - if 500 <= status < 600: - return NullRunTransportError( - f"Gateway error on {endpoint} (status {status}, error={error_slug!r}): {message}", - source=TransportErrorSource.GATEWAY_ERROR, - endpoint=endpoint, - status_code=status, - error_slug=error_slug, - ) - - return NullRunTransportError( - f"Client error on {endpoint} (status {status}, error={error_slug!r}): {message}", - source=TransportErrorSource.GATEWAY_ERROR, - endpoint=endpoint, - status_code=status, - error_slug=error_slug, - ) - - -# Public surface for `from nullrun.transport import X` consumers -# (notably runtime.py). Without this list, mypy treats every -# submodule attribute as private and rejects cross-module imports -# under `--strict`. The list mirrors the symbols runtime.py -# actually consumes plus the convenience constructors / constants -# documented in the README. -__all__ = [ - "HEADER_PROTOCOL", - "NULLRUN_PROTOCOL_VERSION", - "DecisionSource", - "FallbackMode", - "FlushConfig", - "ExecuteConfig", - "Transport", - "TransportErrorSource", - "_retry_with_backoff", - "generate_hmac_signature", - "verify_hmac_signature", - "_signed_request_body", - "RateLimitError", - "InsecureTransportError", -] From d8b32dac0a22fb743270ff58451854e0a5e9d519 Mon Sep 17 00:00:00 2001 From: Anatoly Maltsev Date: Thu, 10 Sep 2026 23:12:41 +0400 Subject: [PATCH 16/16] chore(mypy): track call-arg debt at runtime.py:3213 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit runtime.execute typed-catalog dispatch path (DEF-NR-RUNTIME-BLOCK-TYPED, introduced in 2e77902) passes kwargs into typed_cls(...) where typed_cls is selected at runtime via the catalog. mypy narrows typed_cls to Exception and reports 5 Unexpected keyword argument errors at runtime.py:3213 (workflow_id, reason, action, tool_name, details). The kwargs are catalog-aware via _TYPED_KWARGS_BY_CLASS lookup, so the call is correct at runtime. Two ways to fix: (a) a Protocol for typed_cls (b) splitting the call site per catalog class Both invasive. Track the error code in the existing runtime.py [[tool.mypy.overrides]] block per the comment block above ("Converge via per-file [[tool.mypy.overrides]] entries — each file gets explicit ignore codes so CI breaks when a NEW code appears"). Revisit when the typed-catalog surface stabilises. --- pyproject.toml | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/pyproject.toml b/pyproject.toml index fcd712c..282d4da 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -297,6 +297,18 @@ disable_error_code = [ "arg-type", "assignment", "unused-ignore", + # `call-arg` — runtime.execute typed-catalog dispatch path passes + # kwargs into `typed_cls(...)` where typed_cls is selected at runtime + # (DEF-NR-RUNTIME-BLOCK-TYPED, introduced in 2e77902). mypy narrows + # typed_cls to Exception and reports 5 Unexpected keyword argument + # errors at runtime.py:3213 (workflow_id, reason, action, tool_name, + # details). The kwargs are catalog-aware via _TYPED_KWARGS_BY_CLASS + # lookup, so the call is correct at runtime. Concrete fixes would + # require either (a) a Protocol for typed_cls or (b) splitting the + # call site per catalog class — both invasive. Tracked here per + # the comment block above; revisit when the typed-catalog surface + # stabilises. + "call-arg", ] [[tool.mypy.overrides]]