fix(docker): recycle pooled browser contexts by pages served (#2231)

[ntohidi]

Fixes #2231.

The problem

A pooled browser in the Docker server gets slower with sustained use, and janitor() never recycles it — it only closes browsers that have been idle past a TTL, and a server under continuous load never has an idle one. The workload that causes the degradation is exactly the workload that prevents recycling.

Where the slowdown actually lives

Measured, because the reporter could not reproduce it in-process and neither could I at first. After 1000 real page loads on one pooled browser, timing the same navigation three ways:

	median navigation
the worn pooled context	32.0 ms
a new context, same Chromium process	7.1 ms
a brand-new Chromium process	7.0 ms
the worn context after clearing its state in place	16.3 ms

A fresh context inside the same process is as fast as a fresh process, so the browser process is not the problem — the context is. A control run with pages that leave no state behind stayed flat over 2000 pages; the same pages with cookies + localStorage + a service worker degraded 4-5x from ~500 pages on.

That matches the code: browser_manager.py keeps one context per config for the life of the pooled browser, makes a new page per crawl, and never clears that context's cookies or storage. It also explains why the effect needs real pages — raw: pages leave no state.

The change

browser_manager already has version-based recycling that replaces the context under load without waiting for a quiet moment (no idleness required, old contexts drain naturally). It was simply never enabled for the Docker server. This turns it on:

• config.yml: crawler.pool.max_pages_before_recycle: 200, next to the other pool policy (max_pages, idle_ttl_sec). 0 disables.
• crawler_pool.py: applied in get_crawler() / init_permanent().

It is applied in the pool rather than in crawler.browser.kwargs because /crawl and /crawl/stream build their BrowserConfig from the request body — the server's browser kwargs never reach them. Every endpoint goes through get_crawler(), and pooling is what makes a browser long-lived, so the pool is the right owner of the policy.

Applied before _sig(), so all requests still share one signature and pooling is unchanged. An explicit per-request value wins.

No new lifetime logic in janitor() — that would put the same rule in two places.

Effect

pages	penalty off	penalty with recycling at 200
500	31 ms	10 ms
1500	29 ms	14 ms

It bounds the rot to one recycle window rather than removing it. The threshold is the knob.

Caveats, stated plainly

• My reproduction shows a 25 ms penalty, not the ~5 s the reporter measured. Same shape, much smaller — local pages are cheap, and I wore out one origin's state, not hundreds.
• My penalty was same-origin only; the reporter's was origin-independent. That gap is not closed, so this is "the right lever", not "proven sufficient at their scale". @talelboussetta, if you can run this against your workload, that is the confirmation this needs.
• The /health suggestion in the issue is not addressed here; it is a separate concern.

Tests

tests/docker/test_pool_recycle_config.py — covers the config value, the pool applying it to a request-supplied config, an explicit caller value winning, signature stability, and the get_crawler() call site. Removing the config line fails 2 tests; removing the call site fails 1.

🤖 Generated with Claude Code

https://claude.ai/code/session_019EQGr8ey8riKFr4inp9B8f

[talelboussetta]

Thanks for digging into this — the context-vs-process split is a much better answer than anything I got to, and the fresh-context-in-the-same-process measurement is the part that settles it.

On the gap you flagged between your ~25 ms and my ~5 s: I think that is scale in the same mechanism rather than two different effects. My workload crawls several hundred distinct origins through one pooled context; you wore out one origin. Cookie jar entries, localStorage, service worker registrations and CacheStorage all accumulate per origin inside that single context, so the number of origins is a multiplier on exactly the state you identified.

It also fits the origin-independence I saw. My control was a fresh browser in the same container hitting example.com and iana.org while the worn pooled one was slow to both. If the cost lived in the visited origin's own state, the worn browser would have been fast on origins it had never touched, and it was not. A context-wide cost that every navigation pays is consistent with both sets of numbers. Offered as an explanation that fits the data, not something I have proven.

Two review notes.

The new test file errors at collection when the deps are absent, rather than skipping. from crawl4ai import BrowserConfig is a top-level import that runs before pytest.importorskip("crawler_pool"), so the guard never gets its chance — and your skip reason suggests that case was meant to be handled. On a checkout without the server deps installed, collection dies in crawl4ai/ssl_certificate.py on import OpenSSL.crypto with ModuleNotFoundError: No module named 'OpenSSL', giving 1 error during collection instead of a skip. Pulling BrowserConfig in through an importorskip as well fixes it.

max_pages_before_recycle is on UNTRUSTED_FIELD_ALLOWLIST["BrowserConfig"], so "an explicit per-request value wins" is reachable from the wire and not only from config.yml. Two consequences that might deserve a sentence in the config comment: an operator can A/B the knob per request against a stock image with no redeploy, and a caller sending a value that differs from the pool default forks _sig() into a second pooled browser.

On confirmation at my scale: yes, I will run it and post numbers back here. The allowlist point above makes that cheaper than I expected, since I can set the field per request against the pinned image I already run and keep the recycling behaviour as the only thing under test. I will report whatever I get, including if it fails to reproduce your improvement.

[talelboussetta]

Ran the knob against a local rig. Three results, one of them a null.

1. The field applies from an untrusted request body on stock 0.9.2. Sending max_pages_before_recycle: 200 inside browser_config.params to an unmodified unclecode/crawl4ai:0.9.2 produced, in the server log:

[BROWSER] ℹ Bumping browser version 1 -> 2 after 200 pages (1 active, 0 idle sigs)

and 15 more across a 3000-page run. That is worth a line in the PR: operators can adopt this today without waiting for a release, and it makes the knob A/B-able against an image they already run. I checked the log rather than the status code, since an unrecognised key is dropped silently with a 200.

2. Recycling is free. 3000 pages per arm, fresh container per arm, browser_config identical apart from the field. The probe is a single navigation to a dedicated origin, timed every 250 pages.

arm	median probe	spread	drift, first half to second half
control, no recycling	337.5 ms	63.7 ms	+1.9 ms
recycle at 200, 15 recycles fired	319.1 ms	50.4 ms	-2.4 ms

15 context replacements cost nothing measurable — the recycling arm was, if anything, marginally faster. That seems like the result you would want in hand before defaulting it on, and it supports doing so.

3. I did not reproduce the decay, and my rig cannot refute your 25 ms. 3000 page loads across 99 distinct origins, each setting 5 cookies, ~20 KB of localStorage and an IndexedDB store, left the probe flat: +1.9 ms of drift.

Two reasons not to read that as a negative result:

• Resolution. My probe times a whole /crawl round trip, ~320 ms with a 50-60 ms spread, so a 25 ms navigation delta sits inside my noise. The rig can see a multi-second effect; it cannot see yours.
• A missing ingredient. No service workers. The fixture is plain HTTP, which is not a secure context, so serviceWorker.register() never runs. Service workers are one of the three state types you named, and cookies plus localStorage plus IndexedDB alone did not do it at this scale.

So the honest reading is that my synthetic rig failed to reproduce the mechanism, not that the mechanism is absent.

The real-workload confirmation is still what I owe you, and I have not run it: the deployment that shows the effect is in production and I am not going to experiment on it. I will get it onto a staging copy and post numbers. If it is useful sooner, I can rebuild the local rig over HTTPS so service workers actually register, which would test your third ingredient directly — say the word and I will run that next.