docs(analytics): record assistant review cadence and script hardening - #2348
Conversation
palango
left a comment
There was a problem hiding this comment.
The crossOrigin change looks good. widget.js returns access-control-allow-origin: *, and docs-ai-assistant's next.config.ts sets it on purpose, so it won't vanish by accident. A few lines in the new ANALYTICS.md text don't match what the weekly report does, and since ANALYTICS.md is served publicly at docs.celo.org/ANALYTICS, I'd fix those before merge.
The main one is the rule against pasting raw questions into issues. The weekly report bot pastes up to 25 raw questions into a public issue every Monday (see #2343, which also includes an injection prompt the probe filter missed). The doc and the bot need to agree: either raw text in bot-filed issues is acceptable and the rule should say so, or the report should change. Details inline.
Separately, four weekly issues (#2314, #2320, #2324, #2343) are still open. The weekly issue already filters to answered: false, so a monthly Redis read mostly repeats it. It may be simpler to make triaging those issues the owned task. Also, the report doesn't skip refused: true entries, so off-topic refusals show up in the weekly gaps list. That's a fix for the assistant repo, not this PR.
|
|
||
| ### Reviewing unanswered questions | ||
|
|
||
| - **Weekly:** an automated report files a "Docs assistant: N unanswered questions" issue listing questions that returned no citation, with injection and enumeration probes counted but not listed. |
There was a problem hiding this comment.
The report does list some probes: route.ts passes probeClusters.slice(0, 3) (first 60 chars each) into a "Sampled rather than listed in full" block, and #2343 shows three. Suggest "with injection and enumeration probes counted and only a short sample shown".
|
|
||
| - **Weekly:** an automated report files a "Docs assistant: N unanswered questions" issue listing questions that returned no citation, with injection and enumeration probes counted but not listed. | ||
| - **Monthly:** `@celo-org/devrel` reads the Redis list `docs-assistant:questions`, filters to `answered: false`, and turns real gaps into doc issues. Questions that were refused as off-topic carry `refused: true` and are not gaps. | ||
| - The list holds raw question text. Do not paste entries into issues or PRs; describe the topic instead. |
There was a problem hiding this comment.
This conflicts with the weekly report two lines up, which pastes up to 25 raw questions into a public issue (#2314, #2320, #2324, #2343). Which one is the policy? If bot-filed issues are fine, scope this rule to manual copies. If not, the report needs changing in docs-ai-assistant, and this PR should point to that follow-up.
|
|
||
| ### Script hardening | ||
|
|
||
| `assistant.js` loads `https://docs-assistant.celo.org/widget.js` with `crossOrigin = 'anonymous'`; the host sends `access-control-allow-origin: *`, so the load works under CORS. There is **no Subresource Integrity hash and no Content-Security-Policy** on this site: an `integrity` hash would break the widget on every widget deploy, and the Mintlify Starter plan offers no header configuration we have confirmed. Whoever controls `docs-assistant.celo.org` can therefore run script on every docs page, so changes to that host need the same review as a change to this repo (#2302). |
There was a problem hiding this comment.
Optional: name where the header comes from (docs-ai-assistant next.config.ts, the /widget.js headers rule), so whoever edits that file knows docs.celo.org now depends on it.
| GA4's built-in "AI Assistant" channel recognizes only ChatGPT, Gemini, DeepSeek, Copilot and Grok — not Claude or Perplexity. Known limit: a large share of AI-referred sessions arrive with no referrer and land in Direct; this channel measures the floor, not the total. | ||
| - [ ] **Custom dimensions** (event-scoped): `percent_scrolled`, `link_domain`, `ai_target`, `network`, `result`, `is_automated`, plus the assistant's `answered`, `escalated`, `truncated`, `from_api`, `status` and `href`. Without these registered the assistant events still arrive, but their parameters cannot be used in any report. | ||
| - [ ] **Before the swap: the assistant events need a path that does not go through `window.gtag`.** `widget.js` calls `track()` only `if (typeof window.gtag === 'function')`. On the live site that global is a function, provided by `integrations.ga4`; on a page carrying only the GTM container it is `undefined`. Defining `gtag(){ dataLayer.push(arguments) }` by hand does *not* rescue it — no hit goes out. So all seven `assistant_*` events stop the moment the swap merges. **Two things have to be in place before it, not one**: the widget has to push to `dataLayer` directly (#2307), *and* the container needs the `^assistant_` trigger and tag from Runbook 1 to forward those pushes. Either one alone leaves the events dark. | ||
| - [ ] **After the swap, confirm on a published page** that the assistant still reports: in the browser console `typeof window.gtag === 'function'` must be `true` (or the widget must push to `dataLayer`), then ask the assistant one question and see one `assistant_question` event in GA4 Realtime with `answered` populated. |
There was a problem hiding this comment.
The bullet above says a hand-defined gtag shim sends no hit, so typeof window.gtag === 'function' passing doesn't show anything reaches GA4. Under GTM alone it's undefined, and widget.js still only tracks through window.gtag. I'd drop the gtag clause and keep the Realtime assistant_question event as the pass criterion. #2307 AC 2 still has the old gtag wording and needs the same update.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Review points addressed in edca066 (only
|
palango
left a comment
There was a problem hiding this comment.
Thanks, this covers everything from my last review. The probe wording matches what route.ts does now, the raw-question rule is scoped to manual copies and points at docs-ai-assistant#3, and making triage of the weekly issue the owned task is simpler than a monthly Redis read. I checked the /widget.js rule in next.config.ts and the live header, and the post-swap check no longer leans on window.gtag.
Two optional nits inline. One more, outside this PR: the Gap 2 paragraph in #2307 still says GTM's Google tag should define window.gtag, which contradicts what ANALYTICS.md now says. Worth fixing so the issue and the doc agree.
| - **Weekly:** an automated report files a "Docs assistant: N unanswered questions" issue listing questions that returned no citation, with injection and enumeration probes shown as a count and only a short sample. | ||
| - **Owned task:** `@celo-org/devrel` triages each weekly issue and closes it once its real gaps have been turned into doc issues. Questions that were refused as off-topic carry `refused: true` and are not gaps. | ||
| - The Redis list `docs-assistant:questions` is the raw store for digging deeper. It holds raw question text. | ||
| - Do not copy question text into other issues, PRs or commits; describe the topic instead. The weekly bot report currently pastes raw questions into a public issue. Redacting it, skipping `refused: true` entries and expiring the Redis log is tracked in the `docs-ai-assistant` repository, issue #3. |
There was a problem hiding this comment.
Nit: this page is public, and on docs.celo.org "issue #3" renders as plain text with no repo. Could you use the full URL (https://github.com/celo-org/docs-ai-assistant/issues/3)?
|
|
||
| ### Script hardening | ||
|
|
||
| `assistant.js` loads `https://docs-assistant.celo.org/widget.js` with `crossOrigin = 'anonymous'`; the host sends `access-control-allow-origin: *` from the `/widget.js` headers rule in `docs-ai-assistant`'s `next.config.ts`, so the load works under CORS. Anyone editing that rule should know docs.celo.org depends on it. There is **no Subresource Integrity hash and no Content-Security-Policy** on this site: an `integrity` hash would break the widget on every widget deploy, and the Mintlify Starter plan offers no header configuration we have confirmed. Whoever controls `docs-assistant.celo.org` can therefore run script on every docs page, so changes to that host need the same review as a change to this repo (#2302). |
There was a problem hiding this comment.
Nit: the comment in docs-ai-assistant's next.config.ts says Mintlify exposes no header configuration on any plan, and that a <meta> CSP injected by our loader would apply to nothing. This line hedges to the Starter plan. Whichever is accurate, the two should agree.
Refs #2307, Refs #2302
What changed
assistant.js:script.crossOrigin = 'anonymous'. Checked first:curl -sI -H 'Origin: https://docs.celo.org' https://docs-assistant.celo.org/widget.jsreturnsaccess-control-allow-origin: *, so the CORS load works (without that header the widget would fail to load). The header comes from the/widget.jsheaders rule indocs-ai-assistant'snext.config.ts, whichANALYTICS.mdnow names.ANALYTICS.md:@celo-org/devreltriaging the weekly "Docs assistant: N unanswered questions" issue and closing it once its real gaps are doc issues. The Redis listdocs-assistant:questionsstays as the raw store for digging deeper. Replaces the monthly Redis review.refused: trueentries and expiring the Redis log is tracked indocs-ai-assistantissue Convert the validator-guide to docusaurus. #3.assistant_questionevent in GA4 Realtime withansweredpopulated. Thetypeof window.gtagcheck is dropped, since a hand-defined shim sends no hit and passing it proves nothing under GTM. The task: make the assistant's existing telemetry reportable (GA4 custom dimensions, gtag after GTM swap, unanswered-question review) #2307 body uses the same wording.Boxes still open
Verification
🤖 Generated with Claude Code