[Bug]: Pooled browsers get ~5 s/navigation slower with use, and the janitor never recycles a busy pool

[talelboussetta]

crawl4ai version

0.9.2 (Docker server, unclecode/crawl4ai:0.9.2)

Expected Behavior

A pooled browser serves the thousandth navigation about as fast as the first, or the pool replaces it when it stops doing so.

Current Behavior

A pooled browser degrades with sustained use until it is roughly 5 s per navigation slower than a browser created fresh in the same container, at the same instant, against the same host. Because janitor() only closes browsers that have been idle past a TTL, a server under continuous load never has an idle browser and therefore never recycles one. The degradation persists until the container restarts.

GET /health stays green throughout, so nothing an orchestrator watches reflects it.

Same workload, same code, same config, on one container 30 minutes apart:

Sidecar state	wall clock, same job	pages
just restarted	23.0 s	18/18
~1 h of crawling	183.3 s	18/18

The same 20-URL batch, isolated:

degraded pool      128.2 s   (and 150.2 s -> HTTP 504 on the attempt before)
after restart       10.6 s / 10.4 s / 11.2 s
cold container      16.0 s   (first request after boot - cold start is not the problem)

The penalty is per-navigation and origin-independent. Measured in the same minute on the degraded container, comparing the pooled browser against a browser created fresh inside that same container:

                  degraded pooled browser     fresh browser
site A (Shopify)            6.59 s                2.12 s
example.com                 5.79 s                0.72 s
iana.org                    5.92 s                0.72 s

What it is not - each ruled out by measurement

• The target site. curl TTFB 0.58 s; raw Playwright goto inside the container 0.56 s, while the pooled browser needed 6.59 s.
• Target-side throttling of our IP. A fresh browser was fast at the same instant, from the same container, to the same host.
• DNS / egress. getaddrinfo 0.02 s in the container.
• Concurrency. In-process arun_many of 12 URLs: 5.2 s. Three concurrent rounds back to back left single-page time at 1.0 s.
• context.route handlers accumulating from the block_resources hook (the factory calls context.route("**/*") per page on a shared context). A real suspect, but 30 pages with the hook attached stayed flat at 1.1 s.
• Browser age or page count alone. 60 sequential real pages on one crawler: flat 1.05-1.15 s. 160 inline raw: page creations: flat.

I could not reproduce it in-process. Four attempts failed to trigger it, so something about sustained real page crawling through the server's pooled crawler is responsible and I cannot name it. The threshold is somewhere between 60 pages (clean) and several hundred (badly degraded); I have not bracketed it further. I would rather report the effect with the evidence than guess at the mechanism.

Why it never recovers on its own

crawler_pool.janitor() closes a browser only after it has been idle past cold_ttl / hot_ttl, and skips any crawler with active_requests > 0. Under steady load neither condition is ever met, so the workload that causes the degradation is exactly the workload that prevents recycling. release_crawler is called in a finally, so the count does return to zero - idleness, not leakage, is the blocker.

Two suggestions

1. Recycle on use, not only on idleness. USAGE_COUNT and LAST_USED already exist in crawler_pool.py, so a pool.max_uses / pool.max_age_s sweep in janitor() is a small change: close a browser past either threshold when active_requests == 0, and let get_crawler() re-create it on demand. That is a mitigation, not a fix - it bounds the damage without explaining the cause. Happy to open a PR if you want it.

We run this today from outside the process, on a cron, against /monitor/browsers + POST /monitor/actions/kill_browser, and it works: after a kill the next /crawl completed in 569 ms. Doing it in the janitor would remove the need for anyone to build that.

Two things that do not work as recycling mechanisms, both measured:

• POST /monitor/actions/restart_browser {"sig": "permanent"} wedges the worker - it deadlocks the pool lock (fix(docker): restart_browser on the permanent browser deadlocks the pool #2230).
• gunicorn --max-requests. It fires correctly and then hangs in Waiting for application shutdown. because the ASGI shutdown blocks closing the browser pool; with the image's --timeout 1800 the arbiter will not kill the stuck worker for 30 minutes. Every worker wedged within 7 minutes and the service returned 502 throughout.

2. Make the pool's state visible to /health. Every failure mode here - this one, a dead pooled browser after a Page.goto timeout (#842, PR #1946), and the restart deadlock - is invisible to GET /health, because health never touches the pool. An optional deep/readiness check would let an orchestrator restart a container that is answering nothing, instead of a human noticing a latency graph an hour later.

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

1. Run the 0.9.2 Docker server with a pool config that keeps one browser hot.
2. Drive real (not `raw:`) page crawls through `/crawl` for ~30-60 minutes, several hundred pages, never letting the pool go idle past the TTL.
3. Time a single navigation through the pool, then create a browser directly inside the same container (`playwright` via `docker exec`) and time the same URL. The pooled one is ~5 s slower.
4. Restart the container and repeat step 3 - the gap is gone.

Code snippets

OS

Linux (Docker)

Python version

3.12 (image)

Browser

Chromium (bundled)

Browser version

No response

Error logs & Screenshots (if applicable)

The gunicorn --max-requests attempt at recycling, which is how the shutdown-blocks-on-the-pool behaviour surfaced:

18:16:59 Maximum request limit of 63 exceeded. Terminating process.
18:16:59 Shutting down
18:16:59 Waiting for application shutdown.   <- stops here

/health latency went 1.2 ms -> 8 s and the proxy answered 502 for everything until the containers were restarted.

[ntohidi]

Root cause: the browser context, not the browser process. The pool keeps one context per config for the life of the browser and never clears the state real sites leave in it (cookies, localStorage, service workers). After 1000 pages, a fresh context in the same Chromium was as fast as a brand-new browser (7.1 ms vs 7.0 ms) while the worn one took 32 ms.

Opened #2232 with a fix. It would be great if you could try it against your workload and let us know.