crm: Search Console, Vercel log drain (AI crawlers, 404s, API callers), Web Vitals, time on page; site: not_found and search_no_result events

crm/.env.example

@@ -16,3 +16,12 @@ REFRESH_MINUTES=15
 SNAPSHOT_DIR=/data
 # Optional: Dune API key, to show credits left on the plan.
 DUNE_API_KEY=
+# Optional: Vercel Log Drain (Settings → Log Drains, JSON, request logs) pointed at
+# https://<crm>/api/ingest/vercel. Secret = the drain's custom secret (16+ chars),
+# verify = the token Vercel shows for endpoint verification.
+VERCEL_LOG_DRAIN_SECRET=
+VERCEL_LOG_DRAIN_VERIFY=
+# Optional: Google Search Console. A service account JSON (one line) added as a
+# user of the property, and the property URL (sc-domain:openchainbench.com or https://openchainbench.com/).
+GSC_SERVICE_ACCOUNT_JSON=
+GSC_SITE_URL=sc-domain:openchainbench.com

crm/Dockerfile

@@ -16,6 +16,10 @@ FROM node:22-alpine AS run
 WORKDIR /app
 ENV NODE_ENV=production
 ENV HOSTNAME=0.0.0.0
+# Next's standalone server exits on SIGTERM on its own; with this set it leaves
+# the signal to instrumentation.node.ts, which flushes the log-drain
+# accumulator to the volume before exiting.
+ENV NEXT_MANUAL_SIG_HANDLE=1
 COPY --from=build /app/.next/standalone ./
 COPY --from=build /app/.next/static ./.next/static
 COPY --from=build /app/public ./public

crm/README.md

@@ -10,8 +10,10 @@ One shared password, per-login sessions, one Railway service, no database.
 |---|---|---|
 | Overview | PostHog, blob, Prometheus | visitors, pageviews, sessions (7 d vs previous 7 d), **AI-referred visitors** (ChatGPT, Perplexity, Claude, Gemini, Copilot, …), search-referred visitors, 28 d daily series, 12 w weekly series with the AI share, channels, AI domains, sections |
 | Pages | PostHog | sections week over week, biggest gains and losses, top 100 pages (filter by section), entry pages |
-| Audience | PostHog | new vs returning, bounce, countries, devices, UTM sources, referring domains with their channel |
-| Actions | PostHog custom events | outbound clicks by destination host (visitors sent to providers), copies (endpoint, API URL, MCP, embed) per bench, search queries with the result picked |
+| Audience | PostHog | new vs returning, bounce, Core Web Vitals p75 by device, time on page by section, countries, devices, UTM sources, referring domains with their channel |
+| Actions | PostHog custom events | outbound clicks by destination host (visitors sent to providers), copies (endpoint, API URL, MCP, embed) per bench, searches with no result (content gaps), search queries with the result picked |
+| Search | Google Search Console (service account) | clicks, impressions, CTR, position 7 d vs previous 7 d, 28 d series, opportunities (many impressions, CTR < 1 %, position ≤ 15), top pages and queries |
+| Crawlers | Vercel Log Drain | AI crawler hits by bot (GPTBot, ClaudeBot, PerplexityBot, …) and the sections they read, search bots, human requests by section, who calls /api/stat, /api/citable, llms.txt, top 404 paths, cache hit ratio |
 | Data health | index blob, Prometheus, Dune | live / stale (> 24 h) / expired (> 7 d) benches per category, benches needing attention, scrape targets down, Dune credits and period end, the daily history kept on the volume |
 
 The site captures `$pageview`, `$pageleave` and three custom events
@@ -29,10 +31,10 @@ from the sitemap before publishing, which is exactly what this page must show.
 PostHog allows **2400 query requests per hour per organisation**, shared by
 every key and every team member. This app never queries in the request path:
 
-- a refresh runs a **fixed list of 15 HogQL queries**, one at a time
+- a refresh runs a **fixed list of 19 HogQL queries**, one at a time
   (`lib/traffic.ts`), and writes a snapshot; pages read the snapshot;
 - the scheduler (`instrumentation.ts`) refreshes every `REFRESH_MINUTES`
-  (default 15): **60 queries per hour, 2.5 % of the organisation's budget**;
+  (default 15): **76 queries per hour, 3.2 % of the organisation's budget**;
 - the Refresh button is refused for 5 minutes after any refresh;
 - a local budget (`POSTHOG_HOURLY_BUDGET`, default 300 per rolling hour) is a
   second guard; a 429 or an exhausted budget stops the batch, the sections that
@@ -93,4 +95,28 @@ and each layer gets its own module instance otherwise.
 3. `pnpm test`: the query test checks every query stays scoped to
    `$pageview` on the production host.
 
-Non-PostHog sources go in `lib/ocb.ts` and get a `step()` in `lib/snapshot.ts`.
+Non-PostHog sources go in `lib/ocb.ts` (or their own module: `lib/gsc.ts`,
+`lib/vercel-logs.ts`) and get a `step()` in `lib/snapshot.ts`.
+
+## Connecting Search Console
+
+1. Google Cloud console → a project → APIs → enable **Google Search Console API**.
+2. IAM → Service accounts → create one, add a JSON key, download it.
+3. Search Console → property `openchainbench.com` → Settings → Users and
+   permissions → add the service account e-mail (Full or Restricted, read is enough).
+4. Railway service variables: `GSC_SERVICE_ACCOUNT_JSON` = the key file's
+   content on one line, `GSC_SITE_URL` = `sc-domain:openchainbench.com` (or the
+   URL-prefix property as listed in Search Console). Five API calls per refresh.
+
+## Connecting the Vercel Log Drain
+
+1. Railway variables: `VERCEL_LOG_DRAIN_SECRET` (random, 16+ chars),
+   `VERCEL_LOG_DRAIN_VERIFY` (the verification token Vercel displays when you add
+   the drain; set it, redeploy, then click Verify).
+2. Vercel → team settings → **Log Drains** → Add: project `openchainbench-mobula`,
+   sources **Request logs** (proxy) only, format **JSON**, endpoint
+   `https://<crm host>/api/ingest/vercel`, custom secret = the value above.
+3. Requests are folded per UTC day into `/data/vercel/YYYY-MM-DD.json`
+   (counts only: bot names, sections, status codes, path families; no IPs, no
+   raw user agents). One log line per request; a busy day is a few thousand
+   POSTs of batched entries, negligible for the service.

crm/app/actions/page.tsx

@@ -114,6 +114,24 @@ export default async function ActionsPage({ searchParams }: { searchParams: Prom
         </div>
       </section>
 
+      <section className="panel mt-6 p-4">
+        <p className="label">Searches with no result, 7 d (content gaps)</p>
+        {(t.noResults ?? []).length > 0 ? (
+          <table className="data mt-2">
+            <tbody>
+              {(t.noResults ?? []).map((r) => (
+                <tr key={r.query}>
+                  <td className="mono">{r.query}</td>
+                  <td className="num mono">{fmtInt(r.count)}</td>
+                </tr>
+              ))}
+            </tbody>
+          </table>
+        ) : (
+          <Empty text="No empty search recorded." />
+        )}
+      </section>
+
       <section className="panel mt-6 p-4">
         <p className="label">What people search for, 7 d (a result was picked)</p>
         {searches.length > 0 ? (

crm/app/api/ingest/vercel/route.ts

@@ -0,0 +1,34 @@
+import { drainConfigured, ingestEntries, parseBody, verifySignature } from "@/lib/vercel-logs";
+
+// Public route (proxy.ts lets it through): authenticated by Vercel's HMAC
+// signature over the raw body, never by the session cookie. The
+// verification handshake answers with the token Vercel expects.
+export const dynamic = "force-dynamic";
+const MAX_BODY = 8 * 1024 * 1024;
+
+function verifyHeaders(): HeadersInit {
+  const v = process.env.VERCEL_LOG_DRAIN_VERIFY;
+  return v ? { "x-vercel-verify": v } : {};
+}
+
+export async function GET() {
+  return new Response("ok", { status: 200, headers: verifyHeaders() });
+}
+
+export async function HEAD() {
+  return new Response(null, { status: 200, headers: verifyHeaders() });
+}
+
+export async function POST(request: Request) {
+  if (!drainConfigured()) return new Response("drain not configured", { status: 503, headers: verifyHeaders() });
+  const declared = Number.parseInt(request.headers.get("content-length") ?? "", 10);
+  if (Number.isFinite(declared) && declared > MAX_BODY) return new Response("too large", { status: 413 });
+  const raw = await request.text();
+  if (raw.length > MAX_BODY) return new Response("too large", { status: 413 });
+  if (!verifySignature(raw, request.headers.get("x-vercel-signature"))) {
+    return new Response("bad signature", { status: 401, headers: verifyHeaders() });
+  }
+  const entries = parseBody(raw);
+  const folded = ingestEntries(entries);
+  return Response.json({ received: entries.length, folded }, { headers: verifyHeaders() });
+}

crm/app/audience/page.tsx

@@ -1,7 +1,12 @@
 import { Shell } from "@/components/shell";
 import { Bars, Delta, Empty, fmtInt, fmtPct, Kpi } from "@/components/ui";
+import { SECTION_LABEL } from "@/lib/channels";
 import { readSnapshot } from "@/lib/snapshot";
 
+const fmtMs = (v: number) => (v >= 1000 ? `${(v / 1000).toFixed(2)} s` : `${Math.round(v)} ms`);
+const fmtSec = (v: number) => (v >= 60 ? `${Math.floor(v / 60)} min ${Math.round(v % 60)} s` : `${Math.round(v)} s`);
+const vitalTone = (v: number, good: number, poor: number) => (v <= good ? "var(--good)" : v <= poor ? "var(--warn)" : "var(--bad)");
+
 export const dynamic = "force-dynamic";
 
 const CHANNEL_LABEL = { ai: "AI", search: "Search", social: "Social", direct: "Direct", referral: "Referral", internal: "Internal" } as const;
@@ -57,6 +62,91 @@ export default async function AudiencePage({ searchParams }: { searchParams: Pro
         </div>
       </section>
 
+      <section className="mt-6 grid gap-3 md:grid-cols-2">
+        <div className="panel p-4">
+          <p className="label">Core Web Vitals, 7 d (p75, by device)</p>
+          {t.vitals && t.vitals.length > 0 ? (
+            <table className="data mt-2">
+              <thead>
+                <tr>
+                  <th>Device</th>
+                  <th className="num">Samples</th>
+                  <th className="num">LCP</th>
+                  <th className="num">INP</th>
+                  <th className="num">CLS</th>
+                  <th className="num">FCP</th>
+                </tr>
+              </thead>
+              <tbody>
+                {t.vitals.map((v) => (
+                  <tr key={v.device}>
+                    <td>{v.device}</td>
+                    <td className="num mono">{fmtInt(v.samples)}</td>
+                    <td className="num mono" style={{ color: vitalTone(v.lcpP75, 2500, 4000) }}>{fmtMs(v.lcpP75)}</td>
+                    <td className="num mono" style={{ color: vitalTone(v.inpP75, 200, 500) }}>{fmtMs(v.inpP75)}</td>
+                    <td className="num mono" style={{ color: vitalTone(v.clsP75, 0.1, 0.25) }}>{v.clsP75.toFixed(3)}</td>
+                    <td className="num mono" style={{ color: vitalTone(v.fcpP75, 1800, 3000) }}>{fmtMs(v.fcpP75)}</td>
+                  </tr>
+                ))}
+              </tbody>
+            </table>
+          ) : (
+            <Empty text="No web vitals sample yet." />
+          )}
+          <p className="mt-3 text-[11px]" style={{ color: "var(--faint)" }}>
+            Google&apos;s thresholds: LCP 2.5 s, INP 200 ms, CLS 0.1 (good), 4 s / 500 ms / 0.25 (poor). Ranking signal on mobile.
+          </p>
+        </div>
+        <div className="panel p-4">
+          <p className="label">Time on page by section, 7 d (median of leaves)</p>
+          {t.engaged && t.engaged.length > 0 ? (
+            <table className="data mt-2">
+              <thead>
+                <tr>
+                  <th>Section</th>
+                  <th className="num">Leaves</th>
+                  <th className="num">Median</th>
+                  <th className="num">p75</th>
+                </tr>
+              </thead>
+              <tbody>
+                {t.engaged.map((e) => (
+                  <tr key={e.section}>
+                    <td>{SECTION_LABEL[e.section]}</td>
+                    <td className="num mono">{fmtInt(e.leaves)}</td>
+                    <td className="num mono">{fmtSec(e.medianSec)}</td>
+                    <td className="num mono">{fmtSec(e.p75Sec)}</td>
+                  </tr>
+                ))}
+              </tbody>
+            </table>
+          ) : (
+            <Empty text="No pageleave data yet." />
+          )}
+          <p className="label mt-4">Pages read longest (3+ leaves)</p>
+          {t.engagedPages && t.engagedPages.length > 0 ? (
+            <table className="data mt-2">
+              <tbody>
+                {[...t.engagedPages]
+                  .sort((a, b) => b.medianSec - a.medianSec)
+                  .slice(0, 10)
+                  .map((p) => (
+                    <tr key={p.path}>
+                      <td className="mono truncate" style={{ maxWidth: 300 }} title={p.path}>
+                        {p.path}
+                      </td>
+                      <td className="num mono">{fmtInt(p.leaves)}</td>
+                      <td className="num mono">{fmtSec(p.medianSec)}</td>
+                    </tr>
+                  ))}
+              </tbody>
+            </table>
+          ) : (
+            <Empty text="Nothing yet." />
+          )}
+        </div>
+      </section>
+
       <section className="panel mt-6 p-4">
         <p className="label">Referring domains, 7 d (direct and internal excluded)</p>
         {referrers.length > 0 ? (