Charge training crawlers a dollar a day with @profullstack/x402-gateway

[ralyodio]

What changes, for whom

Visitor	Before	After
A person, Googlebot, Bingbot, Applebot	the site	the site, untouched
OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Google-Extended	the site, named as welcome in robots.txt	same
GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider, Applebot-Extended, FacebookBot	the site, named as welcome in robots.txt	402 Payment Required with an x402 offer ($1 buys a day, USDC settled by CoinPay), or the HTML sales page at /crawl; robots.txt says Disallow: / + Allow: /crawl
A training crawler with a paid pass in x-crawl-pass	n/a	the site, for the day
Any crawler on /llms.txt, /skill.md, /api/feeds, /opml, /mcp (and /api/mcp), /robots.txt	open	still open, via openPaths

Context: a Railway log sample had ClaudeBot at 54% of upstream compute and meta-externalagent at another 11%, none of it sending a reader back. Matching is on the self-declared crawler token, so a browser user agent never sees any of this.

How

• apps/web/src/lib/crawl-gateway.js builds the gateway once, from @profullstack/x402-gateway@0.1.0, so the proxy and robots.txt share one set of lists. It reads SITE_URL itself rather than importing siteUrl() from lib/db.js, because that module pulls in the libSQL client and the gateway must stay edge-clean. Every variable is read through a non-literal key: Next inlines process.env.NAME at build time and the Docker image is built without the secrets.
• apps/web/src/proxy.js now asks the gate first (const answer = await gate(request); if (answer) return answer;) and then runs the throttle, the hint repair and the counter as before. A 402 is still recorded in the traffic ledger, as a refusal, so the price can be judged against what it turns away. The matcher is unchanged.
• apps/web/src/app/robots.txt/route.js is generated by gateway.robotsTxt({ disallow: ['/login', '/api/'], allow: ['/api/feeds', '/api/mcp'], comments }). The doc comment records the policy change; the entry-point advertisement survives as the comment block at the top of the file. Note the new Disallow: /login and Disallow: /api/ for the welcomed groups (both used to be allowed by the blanket Allow: /).
• apps/web/test/crawl-gateway.test.js: robots.txt refuses GPTBot and meta-externalagent with Disallow: /, names OAI-SearchBot with Allow: /, keeps the sitemap and the entry-point comments; the gate answers a meta-externalagent request to /feeds/anything with 402 (empty offer, no key set), lets Chrome, Googlebot and OAI-SearchBot through, leaves the entry points open, and answers /crawl for everyone. proxy.js cannot be imported under plain Node (next/server has no ESM export map), so its ordering is asserted from source the way test/proxy.test.js reads the matcher.
• .env.example documents the two variables.

Railway: two variables on the web service

• COINPAY_X402_KEY = a scoped CoinPay key (cp_live_…, from the business's API Keys tab, with payments:create; the legacy business key is refused by CoinPay's x402 routes). Also the pass-signing secret.
• CRAWL_PAY_TO = the EVM address that receives the USDC (Base, Polygon and Ethereum alike).

Until both are set, training crawlers get 402 with an empty offer and a sales page that says payments are off: nothing is sold, but nothing is given away either.

Verified

• pnpm -r test: 1,462 tests, 0 failures, 12 workspaces (9 new).
• pnpm --filter @rssamplifier/web build: compiles, Proxy (Middleware) listed, 31 s.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC