Refuse the OVH fleet by address and charge browsers that cannot prove it

[ralyodio]

Why

Two callers the crawler lists in #173 never see, both from the edge logs:

• The OVH VPS fleet (measured 2026-08-28): vps-*.vps.ovh.net, every request wearing a spoofed Chrome/148. Zero hits today, but the owner wants the ranges shut.
• A residential-proxy rotation (today, 2026-09-05): 499 distinct IPs in 500 requests, never more than 2 hits per IP, cycling three generic Chrome strings (Chrome/142 Win, Chrome/144 Win, Chrome/145 Mac), walking /topics/*, /api/topics/* and /authors/* at 25 to 250 req/s, with 78% of a burst shed by server.mjs's in-flight cap. No address list can touch it.

The two behaviours

1. denyCidrs — the eight OVH /16s (51.38, 54.38, 141.94, 145.239, 149.202, 151.80, 57.129, 213.32) are answered a tiny 403 Not available from this network. before any other check, including /crawl. Not 402: there is nothing on sale to a hosting range that will not say who it is. The address is X-Real-IP or the last X-Forwarded-For hop (the one the edge appended); a client-seeded first entry is ignored.
2. chargeSpoofedBrowsers: true — a request that claims Chrome/… and sends no Sec-Fetch-Mode (which every Chromium since 76 sends on every request, and which is a forbidden header no script or extension can remove) is charged like GPTBot: 402 plus the x402 offer, a hash instead of a render. Firefox and Safari are not judged. A real browser, headless Chrome included, passes untouched.

Exempt, never charged (exempt option, apps/web/src/lib/crawl-gateway.js):

• a reader with a session: the signed_in hint cookie the masthead reads, or the rsa_session cookie it describes;
• a caller with something shaped like one of our API keys (rsa_<8 hex>_<secret>) as Authorization: Bearer or X-API-Key. Shape only: verifying it is a database lookup that lib/tiers.js does a moment later, and this module has to stay edge-clean (@rssamplifier/auth imports node:crypto). A made-up key gets past the gate and lands in the anonymous tier, metered like a curl user agent already is;
• crawlers that name themselves on the gateway's retrieval list or as a search engine (Googlebot, Bingbot, Applebot, DuckDuckBot, YandexBot) — Googlebot's evergreen string claims Chrome and sends no browser headers, so without this the spoof check would charge Google. The training half of a pair (Applebot-Extended contains "Applebot") stays charged.

Files

• apps/web/src/lib/crawl-gateway.js — DENY_CIDRS, exempt, chargeSpoofedBrowsers: true, with the measurements in the comments.
• apps/web/test/crawl-gateway.test.js — 7 new tests: each OVH range refused even with a full browser header set (last hop, single hop, x-real-ip; a seeded first entry 54.38.1.2, 203.0.113.9 is not refused; 203.0.113.9, 54.38.1.2 is); the three rotation strings get 402 on /topics/x, /api/topics/x, /authors/*; the same strings with sec-fetch-mode: navigate (and cors, no-cors) pass; hint cookie / session cookie pass, emptied session and look-alike cookie name do not; bearer and x-api-key keys pass, Bearer nope does not; evergreen Googlebot, Bingbot, OAI-SearchBot pass while Applebot-Extended and ClaudeBot are charged; /crawl still answers and the deny list still wins there.
• apps/web/package.json, pnpm-lock.yaml, pnpm-workspace.yaml — @profullstack/x402-gateway 0.1.0 → 0.2.1 (pinned), minimumReleaseAgeExclude entry updated.

No new environment variables.

Verified

• pnpm -r test: 1,469 tests, 0 failures, 12 workspaces (apps/web 382, 7 new).
• pnpm --filter @rssamplifier/web build: compiled in 15 s, ƒ Proxy (Middleware) present.

One thing worth knowing: lib/crawlThrottle.js's callerAddress still keys on the first X-Forwarded-For entry (its test says so explicitly), while the gateway now reads the last. On Railway the header carries one entry so both agree; behind a second hop they would not, and the throttle is the one that would be reading a client-seeded value.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WJaXiqE9BDoNfoJBhfXroC

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​profullstack/​x402-gateway@​0.1.0 ⏵ 0.2.1	+1		+1

View full report