crm: Search Console, Vercel log drain (AI crawlers, 404s, API callers), Web Vitals, time on page; site: not_found and search_no_result events - #2547
Merged
Conversation
…, opportunities), Vercel log drain receiver (AI crawlers by bot and section, search bots, 404s, API and llms.txt callers, cache ratio), Core Web Vitals p75, time on page by section, searches with no result; site: not_found and search_no_result events Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJgbZCqjR4nvCfcJSzofbw
…day out before writing and merges it back on failure; NEXT_MANUAL_SIG_HANDLE so the shutdown flush runs; content-length guard on the drain route; one status key for Search Console; page keys folded across www/http variants Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJgbZCqjR4nvCfcJSzofbw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the three gaps of the dashboard: SEO source of truth, server-side visibility, reading quality.
Search page (
lib/gsc.ts): Google Search Console through a service account (RS256 JWT, no OAuth flow,webmasters.readonly). Clicks / impressions / CTR / position for the last 7 full days (GSC lags 3 days) vs the previous 7, 28-day series, top pages and queries with w/w, and an "opportunities" list (50+ impressions, CTR < 1 %, position ≤ 15) which is the title/description work list. 5 API calls per refresh. NeedsGSC_SERVICE_ACCOUNT_JSON+GSC_SITE_URL(README has the 4 steps).Crawlers page (
lib/vercel-logs.ts,POST /api/ingest/vercel): a Vercel Log Drain receiver, authenticated by Vercel's HMAC-SHA1x-vercel-signature(never the session cookie), answering thex-vercel-verifyhandshake. Entries are folded per UTC day in memory (request-id dedupe, bounded) and flushed to the volume every minute and on SIGTERM; counts only, no IPs or raw user agents. Shows AI crawler hits by bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, …) with a 28-day series and the sections they read, search bots, human requests by section, who calls/api/stat,/api/citable,llms.txt(human / AI / search / other bots), top 404 paths, cache hit ratio. NeedsVERCEL_LOG_DRAIN_SECRET(set) +VERCEL_LOG_DRAIN_VERIFYand the drain added in the Vercel team settings.Audience: Core Web Vitals p75 by device (LCP/INP/CLS/FCP with Google's thresholds), time on page by section (leave-weighted median) and the pages read longest. Actions: searches with no result. Crawlers also lists the 404s visitors hit, with the referrer (browser event).
Site:
not_found{path, referrer host} on the 404 page,search_no_result{query} after 800 ms on an empty search (3+ chars).PostHog budget: 19 queries per refresh, 76/hour at the 15-minute cadence (3.2 % of the org's 2400/hour). Tests: 36 (classification of user agents, folding and merging of drain entries, body parsing, engaged-time aggregation). Drain route tested locally with a signed body (bad signature 401, duplicate request id folded once), Crawlers and Search pages render. Scheduler moved to
instrumentation.node.ts(Node APIs cannot sit in the dual-runtimeinstrumentation.ts).🤖 Generated with Claude Code
https://claude.ai/code/session_01HJgbZCqjR4nvCfcJSzofbw