Order Exception Triage Agent

A compact reference implementation that turns scattered enterprise facts and operational policy into a typed, auditable order-exception decision.

Business problem

Traditional automation works well when exception paths are known in advance.

B2B order exceptions can combine an unstructured customer request with facts distributed across customer, order, billing, shipping and policy domains.

The model is used for context-dependent triage; deterministic systems remain the source of business facts and workflow actions.

This project demonstrates the complete path from a customer message to a controlled workflow action without pretending to be a production platform. It answers three practical questions:

• What is true about this order?
• Which operational policy applies?
• Does a person need to review the exception?

Architecture

flowchart LR
    CRM[CRM] --> Context[Deterministic context assembly]
    ERP[ERP] --> Context
    Billing[Billing] --> Context
    Shipping[Shipping] --> Context
    Context --> GetContext[get_case_context]
    Message[Customer message] --> Agent[Agent]
    GetContext --> Agent
    Policy[get_policy] <--> Agent
    Agent --> Decision[TriageDecision]
    Decision --> Host[Deterministic host]
    Host -->|requires_human=true| Escalation[create_escalation]

Loading

The model has exactly two read capabilities: get_case_context(case_id) and get_policy(topic). It has no write-side capability.

The host forces the first context lookup so every run starts from enterprise facts. Later policy selection is model-driven because the relevant rule depends on the meaning of the request and the retrieved facts.

The final response must parse into TriageDecision. Before any side effect, the host verifies the case ID, rejects duplicate citations and checks every evidence reference against facts and policies observed during that run.

Only then may deterministic code append an escalation record.

Example

CUSTOMER

"Our invoice is wrong and the shipment is late."

→ get_case_context(C-014)

  customer.tier           enterprise
  customer.priority_sla   true
  order.total_eur         €8,400
  billing.invoice_total   €9,000
  shipping.days_late      4

→ get_policy("billing_mismatch")

  Invoice discrepancies require manual review.

→ get_policy("priority_sla")

  Priority customers with an overdue shipment require manual review.

→ TriageDecision

  issue_type       billing_and_delivery

  evidence_refs
    order.total_eur
    billing.invoice_total_eur
    shipping.days_late
    customer.priority_sla
    policy.billing_mismatch
    policy.priority_sla

  requires_human   true

  next_step
    Review invoice discrepancy and delayed shipment.

→ Host creates escalation

→ Customer response ready

The escalation is not a model tool call. It is a host decision based on an already parsed and validated result.

Evaluation

The evaluation suite runs six real model cases sequentially and isolates failures so one bad case does not prevent the remaining cases from running. It disables escalation persistence while leaving decision behavior unchanged.

It reports exactly five observable metrics:

• issue classification accuracy;
• escalation decision accuracy;
• expected policy-topic micro coverage;
• required evidence-reference micro coverage;
• schema validity.

Policy coverage comes from the actual policy tool trace. Evidence coverage comes from the final validated citations. Runtime failures count as invalid and incorrect; they are recorded as short diagnostic errors.

evals/results.md is generated only by a real evaluation run. No model result is fabricated when an API key is unavailable.

Design choices

Business-shaped boundary. The model receives normalized case context rather than direct access to internal CRM, ERP, billing and logistics APIs. Internal IDs are joined by deterministic code and never exposed to the model.

Deterministic sources of truth. Enterprise systems establish business facts. The model reasons over exceptions and produces a typed triage decision. Operational rules live in policy fixtures rather than in the system prompt.

Controlled workflow action. The model has no write-side business tools. Human escalation is performed deterministically by the host application when the decision requires review and all validation has passed.

This division keeps the probabilistic work narrow: interpret the exception, choose relevant policy and produce a constrained recommendation. Fact assembly, evidence enforcement and persistence remain ordinary software.

What I would change in production

• Replace fixtures with real enterprise APIs and service identities.
• Add tracing / telemetry across model and tool calls.
• Maintain versioned prompts and evaluation datasets.
• Integrate with a real human case-management workflow.

These are deliberate next steps, not hidden abstractions in this reference implementation.

Run locally

Python 3.12 or newer and an OpenAI API key are required.

cp .env.example .env

Export the values from .env in your shell; the application intentionally does not load environment files itself. Then install dependencies:

uv sync

Run the hero case:

uv run order-triage \
  --case C-014 \
  --message "Our invoice is wrong and the shipment is late."

The command prints the typed decision and, when required, the persisted escalation record. Runtime escalations are written to .runtime/escalations.jsonl.

Run deterministic tests without any API key:

uv run pytest

Run the six live evaluations:

uv run python evals/run.py

That command overwrites evals/results.md with metrics and case diagnostics from the configured model.