Skip to content

HOLD FOR LAUNCH Document Agent classifiers; fix stale AIAM pages - #432

Open
c1-squire-dev[bot] wants to merge 17 commits into
mainfrom
sathya/add-agent-guardrails-doc
Open

HOLD FOR LAUNCH Document Agent classifiers; fix stale AIAM pages#432
c1-squire-dev[bot] wants to merge 17 commits into
mainfrom
sathya/add-agent-guardrails-doc

Conversation

@c1-squire-dev

@c1-squire-dev c1-squire-dev Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Update: renamed to Classifier, rebased on main

The product has since renamed this feature from Guardrail to Classifier (the old Guardrail* API is kept, deprecated, for compatibility — the model layer is unchanged). This revision renames all three pages (including their filenames/URLs) and the sibling corrections to match, rebases onto latest main, and keeps the OCSF audit event fields (guardrail_decision, aigov.guardrail.decision, guardrail_id) as-is with a note explaining they weren't renamed — only the admin UI and public API were. Re-ran npx mintlify broken-links: still zero broken links introduced by these pages.

What changed since the last review

The previous revision of this PR used "Agent classifiers" as an umbrella term for existing AIAM mechanisms (toolsets, hooks, kill switches, step-up auth). That name is already a shipped feature with its own object model (Classifier, ClassifierBinding, AgentClassifierRule, ToolGate) and admin page at AI > Classifiers. A reader who followed those pages and then opened that page would have found a UI sharing nothing with them but the name.

All three pages are rewritten to document the actual feature. The sibling pages the old version copied from are corrected too, since they were the source of most of its errors.

The pages

Page Content
agent-classifiers Classifiers as rule cascades over the three risk signals (private data / untrusted content / exfiltration); bindings; Agent vs Gateway surfaces; how rules relate to tool gates and hooks
agent-classifiers-configure Create a classifier → bind it → author rules → attach gates and hooks → promote Observe to Enforce → judge settings
agent-classifiers-reference Object model, rule and gate fields, CEL variables per surface, evaluation order, constraints, troubleshooting

Scope note: classifier templates/presets are intentionally not covered.

Sibling-page corrections

  • tool-call-hooks — hooks live under AI > Classifiers, not C1 Gateway (moved in an earlier PR); adds the pre-output event, the caller payload block (with the warning that it is never an authorization input), event-dependent filter variables, the six undocumented built-in patterns (11 exist, 5 were documented), the claw tool_source, corrected pattern defaults, and the Managed by classifiers toggle
  • tools-and-toolsets — classification is one five-value field, not "Action (Read/Write/Delete)" + "Risk (Sensitive/Dangerous)" axes; there is no "Delete" value; real toolset names (All approved tools / Read tools) and per-connector scope
  • enable-ai-access-management — empty allowed-client-types means all types allowed (was documented as personal-only); default classification is Sensitive, not Unclassified; real client-lifecycle labels and defaults; adds the per-server kill switch
  • ai-clients, aiam-overview — matching lifecycle and toolset fixes

Verification method

The previous revision's stated method — tracing claims to sibling docs — was the root cause of its errors: it propagated their staleness and couldn't detect a collision with an undocumented feature. This revision verified against the implementation and by exercising the feature in a dev tenant.

Caught and fixed during verification:

  • The judge does not fail closed with no score. When unreachable at run entry it seeds a cautious MEDIUM untrusted-content floor; only a mid-run rescore failure keeps the prior floor.
  • Tool gate filters expose nine CEL variables, not the two the in-product helper text names. Documented both, and noted which two the UI mentions.
  • Rule CEL failure is skipped, not denied — the opposite of hook filter behavior, where a failed filter denies the call. Both are now stated explicitly because the asymmetry is surprising.

Deliberately not documented, because they are declared but unimplemented: Classifier.default_outcome at eval time, a system baseline rule on the Classifier object, and centrally-patched template distribution. Confirmed unimplemented rather than assumed.

Test plan

  • Exercised the feature end-to-end in a dev tenant: created a classifier, inspected a rule cascade (IF ctx.private_data >= 2 && ctx.exfiltration >= 2Block / Enforce), opened the rule form, checked the judge settings tab, and deleted the classifier. All UI labels, helper text, and defaults in these pages are transcribed from that session.
  • Confirmed live that C1 Gateway > Settings contains no hook UI (only C1 Gateway, Client governance, and Default C1 Gateway Classifier policy) — the old nav path was wrong.
  • Confirmed live that the two bindings live on different pages: the agent binding on AI > Classifiers, the gateway binding on AI > C1 Gateway > Settings.
  • Confirmed live client-lifecycle defaults: Hide inactive clients after 1 day, Close credentials after 1 month, Delete closed clients after 3 months — and that the third is measured from close, not from last activity.
  • npx mintlify broken-links — no new broken links; anchors verified against real headings. The 8 reported are pre-existing in unrelated files.
  • docs.json validates.
  • Review the rendered preview.

For reviewers

Terminology. These pages call the risk axes "signals" and the cascade a "classifier," matching the product's current naming.

Tenant availability is conveyed only by the standard "Activation required" note already used across the AI access management pages. The pages name no internal gating.

🤖 Generated with Claude Code

@mintlify

mintlify Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
conductorone 🟢 Ready View Preview Jul 30, 2026, 4:18 AM

@mintlify

mintlify Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
conductorone 🟡 Building Jul 30, 2026, 4:16 AM

@c1-squire-dev

c1-squire-dev Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Review v3 addressed

2a / 2b — done (383e3ae). Documented that a guardrail with no rules (or whose every rule is Disabled) matches nothing and is indistinguishable at call time from an unbound surface, and that a call matching no rule writes no guardrail_decision event — so "traffic arrived and matched nothing" and "no traffic arrived" are identical silence in the system log. Folded the second point into the Observe troubleshooting entry, which previously gave the right advice without explaining why both cases look the same.

No "system baseline rule" documented, per the review — the proto comment claiming one is stale; baseline is only an audit label (rule_kind) applied when nothing matched, not a rule object.

1 — gate filter patterns: staying out, by doc-owner decision.

The review is factually right and I confirmed it independently: AIGOV_GUARDRAIL_PRESETS doesn't exist on main (only on the unmerged saravana.venkatesan/FN-454/guardrail-presets), it's absent from the live c1dev flag list, and ToolGateFormDrawer.tsx has no flag check of its own — the pattern selector renders for anyone who can open the gate form. The two-things-called-preset distinction in the review is accurate.

I raised the contradiction with the doc owner rather than restoring unilaterally, and the decision is to keep them out. The gate filter is documented as a CEL expression entered directly, which is correct and complete for authoring a gate; the prefill helper is just undocumented. Re-raise if that changes.

Cross-page dependency — already resolved in this PR, not a follow-up.

The review flags tool-call-hooks.mdx as stale on three counts and suggests a separate PR. All three were fixed here:

Defect Status
Hooks under AI > C1 Gateway > Settings Fixed — now AI > Guardrails → General tab → Hooks section
5 of 11 built-in patterns documented Fixed — all 11, plus a separate configuration/defaults table
"hooks cannot see the caller" Fixed — caller block in both payload examples, with the warning that it's never an authorization input

That page also gained the pre-output event, event-dependent filter variables, the claw tool_source, corrected pattern defaults, and the Managed by guardrails section. So a reader following the Step 5 link lands on corrected content. No follow-up needed for those three.

Still open for product, not docs:

  • guardrail_decision is missing from the published OCSF event catalog that /product/admin/system-log calls authoritative — a customer building a SIEM detection from that list won't find it. The docs give the literal filter values so they work regardless.
  • Guardrail.default_outcome / default_deny_reason are settable via the public API and never read by the evaluator. Not a docs bug; an admin can set "default outcome = DENIED" and get silence.
  • Observe is hard to consume as designed: no admin UI, no notification, no aggregate count, and informational severity means a warnings-and-above SIEM filter hides exactly what you're staging against.

@mindymo mindymo changed the title Document Agent guardrails; fix stale AIAM pages HOLD FOR LAUNCH Document Agent guardrails; fix stale AIAM pages Jul 30, 2026
@linear-code

linear-code Bot commented Jul 30, 2026

Copy link
Copy Markdown

FN-570

Ties together the existing controls that bound AI agents acting on
the org's behalf (toolsets/access profiles, tool call hooks, kill
switches, client lifecycle, step-up auth, the Agent policy step, and
audit logging) into a single overview page, mirroring the structure
of the Functions overview page.

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
sathyasaravanamuthu and others added 14 commits September 2, 2026 17:02
The first pass was purely conceptual. Restructure into a three-page set
mirroring the Functions docs (overview / hands-on / reference):

- agent-guardrails.mdx: tighten to an overview with a layer table
  (what each layer constrains, where it's configured, how it fails)
- agent-guardrails-configure.mdx: six-step walkthrough with real UI
  paths, a worked built-in hook config, three TypeScript guardrail
  function examples, an observe-mode rollout pattern, and a
  kill-switch scope table
- agent-guardrails-reference.mdx: hook payload/return contract, filter
  expression variables, built-in pattern options, audit statuses,
  constraints, lifecycle defaults, and troubleshooting

All claims traced back to the existing mechanism pages; no new product
behavior asserted.

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
The previous version of these pages used "Agent guardrails" as an
umbrella term for existing AIAM mechanisms (toolsets, hooks, kill
switches, step-up auth). That name is already taken by a shipped
feature with its own object model and admin page, so a reader who
followed those pages and then opened AI > Guardrails found a UI that
shared nothing with them but the name.

Rewrite all three pages to document the actual feature:

- agent-guardrails: guardrails as rule cascades over the three risk
  signals (private data / untrusted content / exfiltration), bindings,
  surfaces, and how rules relate to tool gates and hooks
- agent-guardrails-configure: instantiate from a template, bind, author
  rules, attach gates and hooks, promote Observe -> Enforce, judge
  settings
- agent-guardrails-reference: object model, rule and gate fields, CEL
  variables per surface, evaluation order, the built-in template
  catalog with the mode each ships in, constraints, troubleshooting

Also correct the sibling pages these previously copied from, which were
stale against the implementation:

- tool-call-hooks: hooks live under AI > Guardrails, not C1 Gateway;
  add the pre-output event, the caller payload block, event-dependent
  filter variables, the six undocumented built-in patterns, the claw
  tool_source, corrected pattern defaults, and Managed by guardrails
- tools-and-toolsets: classification is one five-value field, not
  Action/Risk axes; real toolset names and per-connector scope
- enable-ai-access-management: empty allowed-client-types means all
  allowed; default classification is Sensitive; real client-lifecycle
  labels and defaults; per-server kill switch
- ai-clients, aiam-overview: matching lifecycle and toolset fixes

Verified against the implementation and by exercising the feature in a
dev tenant, rather than against sibling docs.

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
Templates/presets are out of scope for this doc set, so remove them
rather than describing a surface readers may not have:

- overview: drop the template concept row and template-sourced framing
  from the risk-signals section and the use cases
- configure: replace the template-instantiation Step 1 with creating a
  guardrail directly, fold in the former from-scratch step, and
  renumber the remaining steps
- reference: remove the built-in template catalog, the template object
  row and diagram edge, the Created from template provenance field, and
  the template troubleshooting entry

Also confirmed these pages name no feature flags. Tenant availability
is conveyed only by the standard "Activation required" note used across
the AI access management pages.

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
The object-model diagram and the CEL examples were wider than the
rendered content column at common viewport widths, so they clipped
instead of scrolling.

- reference: flatten the object-model diagram (it no longer needs the
  binding arrow inline, now stated in prose) and wrap the CEL examples
- configure: convert the CEL variable list from a code block to a
  table, since tables wrap and code blocks don't; wrap the example

Widest code-block line is now 44 characters.

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
The built-in patterns table packed each pattern's full default list into
its "What it does" cell. Rendered in the docs content column that cell
wrapped to a dozen lines and broke inline code chips mid-word.

Keep the patterns table to one scannable line per pattern, and move
configuration and defaults into their own table beside it. The
replaces-not-extends warning is now stated once up front instead of
repeated per row.

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
Correctness fixes:

- reference: the Constraints table claimed gate filters see only two CEL
  variables, contradicting the nine-variable table two sections above
- reference: rule Name doesn't appear in audit records — records
  identify a matched rule by its ID
- reference: split ctx.tool_kind and ctx.caller_kind into their two
  actual vocabularies; they are not one shared list
- reference: add the 32,768-byte cap on a rule's CEL condition
- reference: Managed by can also be System (read-only, C1-curated)

New: pre-output hooks were named but never explained, and the stage
differs from the tool-call stages in four ways that each fail silently.
Added a reference section plus a hooks-page subsection covering:

- it is a separate cascade walk; a rule is only eligible if it uses
  Evaluate hooks AND selects a pre-output hook, so a Block rule never
  applies to outgoing responses and an ineligible rule falls through
- only Block output and Link filter run there; custom function hooks
  are rejected at save time
- ctx.untrusted_class (string) vs the rule's ctx.untrusted_content
  (ordered level) are the same score under two names and two types, and
  getting it wrong denies rather than errors
- a Block output hook with no surfaces selected is inert

Also drops ctx.agent_id from the documented rule-variable set per the
doc owner; it remains in the tool gate filter set, which is separate.

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
The pages said a guardrail set to "Any surface" can bind to either
surface. That's true of the API contract but false in the product: both
binding selectors filter on strict equality against their own surface,
so an unset guardrail appears in neither and can't be bound from the UI
at all.

"Applies to" was also documented as a three-way choice. The picker
offers only Agent and Gateway; "Any surface" is a display-only label for
the unset value, retained for guardrails carried over from the older
single-policy model.

- overview: a guardrail targets one surface; create one per surface to
  cover both
- configure: the agent selector matches Agent exactly
- reference: two-way choice, plus a warning about the unbindable unset
  state and how to get out of it

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
The "Start from pattern" prefill helper and its six canned patterns are
only present when the presets feature is enabled, so they don't belong
in the general documentation — same reasoning as dropping templates.

- configure: drop the prefill step; the filter is now entered directly
- reference: drop the "Start from pattern" field row and the whole
  "Gate filter patterns" section

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
Two gaps.

Observe mode was documented without saying where to look, and what it
did say was wrong: it pointed readers at AI > Conversations, which is
the chat-assistant session history and a different data store. The
guardrail evaluator never writes there.

Replace it with the real path: decisions are written to the system log
as guardrail_decision events, an Observe match carries
would_have_fired, and reading them requires a configured system log
exporter because there is no admin UI for these events. Documents the
fields to filter on, that Observe records are informational severity
(so a warnings-and-above filter hides them), and the 365-day default
retention. Says plainly that nothing notifies you.

Second: a rule can select several tool gates but only the first whose
filter matches runs, across the union with unmanaged gates, ordered
enabled-then-priority-then-age. That makes Priority load-bearing when
filters overlap — two gates with different approvers won't both apply.

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
Review v3, items 2a and 2b.

- A guardrail with no rules, or whose every rule is Disabled, matches
  nothing and is indistinguishable at call time from an unbound surface
- A call that matches no rule writes no guardrail_decision event, so
  "traffic arrived and matched nothing" and "no traffic arrived" are
  identical silence in the system log. Added to the evaluation-order
  section and folded into the Observe troubleshooting entry, which
  previously gave the right advice without saying why both cases look
  the same

Deliberately does not document a "system baseline rule": the proto
comment claiming one is stale — "baseline" is only an audit label
applied when no rule matched, not a rule object.

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Not GA — rolling out to a limited set of design partners under a
manual SKU grant. Replaces the "Activation required" Note (borrowed
from the AIAM sibling pages, which are on a different rollout track)
with the house early-access Warning, following the wording used for
other EA features (push-rules.mdx, automations.mdx's Automation
assistant, policies.mdx's Policy assistant) plus the design-partner
rollout detail from enterprise-managed-authorization/enable.mdx.

Applied to all three pages, including the reference page, which
previously carried no activation note at all.

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
Replace the customized notice (named the feature, added a
design-partner sentence, linked the support team) with the exact
wording from push-rules.mdx, the precedent for a whole feature in
early access. No paraphrasing.

Co-authored-by: c1-squire-dev[bot] <c1-squire-dev[bot]@users.noreply.github.com>
The feature is being renamed in the product from Guardrails to
Classifiers. Updates all three new pages (including their filenames and
URLs), the sidebar nav entry in docs.json, and the sibling corrections to
tool-call-hooks.mdx to match.

The audit log's OCSF event fields (activity_name: guardrail_decision,
api.operation: aigov.guardrail.decision, and the guardrail_id field) keep
their original names — only the admin UI and public API were renamed, not
the underlying event schema. Added a note calling that out explicitly so
a reader filtering their SIEM on the new name doesn't come up empty.
@mindymo

mindymo commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Screenshot verification: UI discrepancies found

I reviewed the new Classifiers/hooks/tool gate documentation against current screenshots of the actual UI. Style-wise the PR is in good shape (no docs-writing guide violations), but I found several places where the docs don't match what's in the product right now. Flagging for the project team before this ships:

  1. Hook filter variables don't match. tool-call-hooks.mdx's "Filter variables by event" table says Pre-tool/Post-tool hooks only see ctx.tool_name, and Pre-output hooks see ctx.untrusted_class/ctx.surface. The actual hook filter field's helper text reads: "Available variables: ctx.tool_name, ctx.tool_input, ctx.surface, ctx.channel_id, ctx.workspace_id, ctx.user_roles" — and this list was identical regardless of which event was selected, so the doc's per-event split doesn't reflect current UI behavior.

  2. An undocumented third Hook type exists. The Hook type dropdown offers Custom function, Built-in pattern, and Patch tool input. The docs only describe the first two.

  3. Tool gates have an undocumented "Exclusions" section, including an "Exclude task result read-backs" checkbox that's checked by default. This isn't mentioned in either the configure guide (Step 4) or the tool gate fields table in the reference page.

  4. Event isn't independently selectable for every Hook type — it's autopopulated and greyed out for two of the three. Custom function lets you choose Pre-tool use or Post-tool use; Built-in pattern autopopulates to Post-tool use (greyed out); Patch tool input autopopulates to Pre-tool use (greyed out). None of the three paths in the current UI expose Pre-output as a selectable event. The docs describe Pre-output extensively — as a hook event and as the stage for two built-in patterns (Block output, Link filter) — so the open question for the team is whether the Pre-output event has shipped in this build at all, not just a wording mismatch.

One thing that did check out correctly: the tool gate filter helper text ("Available variables include ctx.tool_name and ctx.tool_input") matches the doc's note exactly.

@mindymo mindymo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my PR comment, there are a few gaps here we need to address.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verified against pkg/api/ai_governance frontend source (HookFormDrawer.tsx,
ToolGateFormDrawer.tsx, patternTypes.ts) rather than screenshots alone, per
a review flagging several UI mismatches:

- Filter variables by event: the Pre-tool/Post-tool row was missing five of
  six variables (ctx.tool_input, ctx.surface, ctx.channel_id,
  ctx.workspace_id, ctx.user_roles), and the Pre-output row was missing
  ctx.user_roles.
- Added the third hook type, Patch tool input (RFC 7396 merge-patch, CEL or
  static source, Pre-tool-use only) - previously undocumented.
- Clarified which hook types let you choose an event and which auto-lock:
  Pre-output *is* reachable in the current UI, through Block output (locked)
  or Link filter (selectable) - not unreachable as it read before.
- Tool gates: documented the Exclusions section (two on-by-default
  checkboxes that add to the filter without touching what you typed) in
  both the configure guide and the reference page.
- Built-in patterns: added the two the table was missing (Block tool
  output, Pre-tool block) and flagged the three the table listed that
  aren't currently offered in the admin UI's pattern picker (Secrets
  masking, Encoded content guard, Prompt injection scan) rather than
  presenting all as equally available.

Copy link
Copy Markdown
Contributor

Review addressed

Verified all four points against the current frontend source (HookFormDrawer.tsx, ToolGateFormDrawer.tsx, patternTypes.ts, toolGateFilterSnippets.ts) rather than re-screenshotting, since the dev environment I'd used for the earlier live verification wasn't available. Confirmed and fixed in 929525d:

  1. Hook filter variables — the doc's Pre-tool/Post-tool row listed only ctx.tool_name; the actual helper text (and the code's TOOL_USE_CTX_HINT) exposes six: ctx.tool_name, ctx.tool_input, ctx.surface, ctx.channel_id, ctx.workspace_id, ctx.user_roles. The Pre-output row was also missing ctx.user_roles. Both fixed, split into two tables with per-variable notes. The code is event-dependent (the helper text branches on currentEvent === PRE_OUTPUT), so if what you saw live looked identical across events, that's worth a second look separately — I couldn't reproduce it from source alone.

  2. Patch tool input — confirmed real (HookFormDrawer.tsx:445, gated behind a preview flag, not by any per-hook-type UI logic). Added a new section: RFC 7396 merge-patch semantics, CEL-expression or static-fields source, locked to Pre-tool use, and a note that it still has its own top-level Filter field deciding whether it fires (separate from the patch source, which decides what to merge).

  3. Tool gate Exclusions — confirmed (ToolGateFormDrawer.tsx + toolGateFilterSnippets.ts): two checkboxes, both on by default (Exclude task result read-backs, Exclude built-in tools), each ANDed onto the saved filter without touching what you typed. Documented in both the configure guide and the reference page.

  4. Pre-output reachability — resolved with a concrete answer instead of an open question: Pre-output is selectable in the shipped UI, through two of the ten built-in patterns — Block output (locked to it) and Link filter (a real choice between Post-tool use and Pre-output; every other pattern locks to a single non-Pre-output event). Custom function hooks can't reach it; Patch tool input hooks are locked to Pre-tool use. Documented the full per-hook-type table.

One more mismatch found while verifying #4, not in your original list: the Built-in patterns table listed eleven patterns, but the proto (hooks.proto) defines thirteen and the frontend's pattern picker (patternTypes.ts) only wires up ten of those thirteen. The doc's eleven and the UI's ten don't overlap cleanly — it was missing two patterns the UI does offer (Block tool output, Pre-tool block) while including three the UI doesn't offer at all (Secrets masking, Encoded content guard, Prompt injection scan — defined in the proto, absent from patternMetaList, so there's no picker entry for them regardless of tenant/flag state as far as I can find in the frontend source). Added the two real ones with their actual config (a single deny-message field each), and flagged the three UI-unreachable ones with a note rather than silently deleting them, in case there's a flag or rollout path I'm not seeing from source alone — happy to remove them outright if you confirm they're not coming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants