Skip to content

feat: accept --analytics-intent and --analytics-task-id on every command - #295

Open
angeloashmore wants to merge 11 commits into
claude/eval-record-argvfrom
claude/cli-intent-task-id-i1wobt
Open

feat: accept --analytics-intent and --analytics-task-id on every command#295
angeloashmore wants to merge 11 commits into
claude/eval-record-argvfrom
claude/cli-intent-task-id-i1wobt

Conversation

@angeloashmore

@angeloashmore angeloashmore commented Sep 9, 2026

Copy link
Copy Markdown
Member

Resolves:

Stacked on #297. Merge that first; GitHub retargets this PR to main when it lands.

Description

Analytics see each CLI command as an isolated event. When an agent runs ten commands to serve one user request, nothing ties them together.

Now every command accepts --analytics-intent and --analytics-task-id. Agents pass the same values on every command for one request. The values reach Amplitude as userIntent and taskId, the same properties the MCP server sends, so both surfaces can be analyzed together. Sentry gets them too.

Humans never see the options. They appear in help output only when an agent is detected, along with an AGENTS section on the router help pages that explains how to use them. All commands are now tracked, including the ones that were previously excluded.

An eval checks that agents pass one UUID and one intent across every command of a task. With the skill rule alone it passed 2 of 10 trials. With the AGENTS section as well it passed 10 of 10.

Companion change in the skill: prismicio/skills#9

Checklist

  • If my changes require tests, I added them.
  • If my changes affect backward compatibility, it has been discussed.
  • If my changes require an update to the CONTRIBUTING.md guide, I updated it.

Preview

$ AI_AGENT=claude-code prismic --help
...
OPTIONS
      --analytics-intent string    The user's overall task in one short sentence. ...
      --analytics-task-id string   A globally unique ID (UUID) for the user's task. ...
  -h, --help                       Show help for command

AGENTS
  Before the first command for a user request, generate one UUID, for example
  with `node -e "console.log(crypto.randomUUID())"`. Pass that exact value as
  --analytics-task-id on every command for the request, including read-only
  commands such as list and view. Never use a placeholder and never generate a
  second UUID for the same request. Pass the request in one sentence as
  --analytics-intent on the same commands. Analytics only, no effect on behavior.

How to QA 1

  1. Run prismic --help. No AGENTS section and no analytics options.
  2. Run AI_AGENT=test prismic --help. Both appear.
  3. Run any command with --analytics-intent "..." --analytics-task-id <uuid>. It succeeds, and the Amplitude event carries userIntent and taskId.

🤖 Generated with Claude Code

https://claude.ai/code/session_015pufvNWHTyTcd38NnNPyPK


Note

Medium Risk
Expands telemetry to previously untracked commands (login, docs, etc.) and adds new global CLI flags; behavior is unchanged unless flags are passed, but analytics volume and event content change.

Overview
Adds --analytics-intent and --analytics-task-id on every CLI command so agents can tie multiple invocations to one user request. Values are analytics-only (parsed at the root in index.ts) and are sent to Segment as userIntent / taskId and to Sentry as tags/context, aligned with the MCP server.

When an agent is detected (detectAgent()), help shows those options plus an AGENTS section on router/command help explaining UUID + intent usage; humans do not see them (hidden options). All commands are now telemetry-tracked—the previous UNTRACKED_COMMANDS allowlist is removed.

Eval harness no longer infers CLI usage from shell strings: the project prismic shim logs argv to JSONL and toHaveRun matches positional args on result.calls. A new group-commands-by-task eval asserts one shared task UUID and intent across every command in a multi-step task.

Reviewed by Cursor Bugbot for commit 6e540ac. Bugbot is set up for automated code reviews on this repo. Configure here.

Footnotes

  1. Please use these labels when submitting a review:
    ❓ #ask: Ask a question.
    💡 #idea: Suggest an idea.
    ⚠️ #issue: Strongly suggest a change.
    🎉 #nice: Share a compliment.

Agents pass these two options so analytics can group the commands that
serve one user request. The options are always accepted. They appear in
help output only when an agent is detected.

The values are sent as userIntent and taskId on the "Prismic CLI Start"
and "Prismic CLI End" events, and attached to Sentry. Every command is
now tracked, including the ones that were previously excluded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pufvNWHTyTcd38NnNPyPK
Replaces the mutable global options hook. Help output decides at render
time whether to list the agent options.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pufvNWHTyTcd38NnNPyPK
Agent detection is synchronous, so help builders stay synchronous and
the agent options are plain config with a hidden flag, like the router's
hidden commands.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pufvNWHTyTcd38NnNPyPK
…mmand

The eval wraps the CLI bin to record the argv of every call, so it
checks the values the CLI receives rather than the shell text the agent
wrote. It pins the Prismic skill to the commit that adds the rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pufvNWHTyTcd38NnNPyPK
…ics-task-id

The prefix says what the options are for. Without it, the names read as
if they change behavior.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pufvNWHTyTcd38NnNPyPK
@angeloashmore angeloashmore changed the title feat: accept --intent and --task-id on every command for agent analytics feat: accept --analytics-intent and --analytics-task-id on every command Sep 9, 2026
@angeloashmore angeloashmore changed the title feat: accept --analytics-intent and --analytics-task-id on every command feat: accept --analytics-intent and --analytics-task-id on every command Sep 9, 2026
The section tells agents to generate one UUID per request and pass it,
with the request in one sentence, on every command. It appears only when
an agent is detected. Help text without it scored 0 of 3 eval trials;
with it and the skill rule, 3 of 3.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pufvNWHTyTcd38NnNPyPK
Agents generated the UUID late or more than once. The section now says
to generate it before the first command, never use a placeholder, and
never generate a second one. The option descriptions name read-only
commands too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pufvNWHTyTcd38NnNPyPK
@angeloashmore
angeloashmore marked this pull request as ready for review September 9, 2026 14:55

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e37c813. Configure here.

Comment thread src/index.ts
}

const isTracked = !help && command && !UNTRACKED_COMMANDS.includes(command);
const isTracked = !help && command;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sync command emits duplicate start events

Medium Severity

Removing UNTRACKED_COMMANDS makes index emit Prismic CLI Start for sync, but sync already calls trackCommandStart with watch: true. Amplitude now gets two starts for one watch session, and the first event lacks watch.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit e37c813. Configure here.

The pin moves to the merged skill commit in a later change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pufvNWHTyTcd38NnNPyPK
The harness now records argv for every call, so the eval no longer
needs its own bin wrapper.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pufvNWHTyTcd38NnNPyPK
@angeloashmore
angeloashmore changed the base branch from main to claude/eval-record-argv September 9, 2026 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants