Adaptive concurrency through llmgate's limiter; parallel documents and page batches (HAL-1372)

[hallelx2]

Closes HAL-1372 (engine half; llmgate v0.5.0 carries the limiter).

limit.Client / limit.Judge wrap the chat client and the Judge in both binaries, inside retry, every change logged. Config llm.concurrency {initial, max}. judgewalk sends its page-ranking batches at once; tocdump -parallel N and navbench -parallel N run documents and questions concurrently through one shared limiter.

Measured, no rate limits hit in either run, the limiter widening itself on success:

	sequential	parallel
21 filings, whole TOC stage	199 s	122 s at 8 in flight; limiter 8 → 11; identical trees, 47/47 gold pages
40 questions, navigation	824 s	see PR comment; limiter 4 → 12

Per-request latency rises under load (9.5 → 21 s per filing), so the gain is 1.6×, not 8× — the provider's capacity, observed rather than guessed, is now the limit.

Summary by Sourcery

Enable adaptive, provider-aware concurrency across document ingestion and navigation to improve throughput without exceeding provider capacity.

New Features:

• Add configurable adaptive per-provider concurrency limits for LLM and Judge requests.
• Enable parallel document processing in tocdump and parallel question processing in navbench.
• Send independent page-ranking batches concurrently during navigation.

Enhancements:

• Apply adaptive limiting across retries and log limiter widening, narrowing, and throttling events.
• Expose parallel execution controls and report wall-clock runtime and final limiter capacity.
• Update navigation tests for concurrent request execution.

Build:

• Upgrade the llmgate dependency to v0.5.0.

Documentation:

• Document adaptive concurrency settings in the example application and server configurations.

Tests:

• Make navigation mock request counters safe for concurrent access.

Summary by CodeRabbit

• New Features

  • Added adaptive concurrency controls for LLM and Judge requests, automatically adjusting limits based on success and failures.
  • Added configurable LLM concurrency settings with initial and maximum limits.
  • Added parallel processing options for navigation benchmarks and document processing.
  • Added progress summaries including elapsed time, processed counts, and final concurrency limits.

• Performance

  • Page ranking and document workflows can now execute concurrently while preserving results and error reporting.

• Reliability

  • Improved retry behavior and coordination during transient request failures.

[sourcery-ai]

Sorry @hallelx2, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 6 days by commenting @sourcery-ai review. Upgrade to get a review now.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[hallelx2]

Navigation at parallel=4: 258 s wall for 40 questions (sequential 824 s, 3.2×), limiter 4 → 12, zero 429s; 33/40 every-gold-page (34/40 sequential — one question flipped, within run-to-run noise), 37/40 right section, 4.0 requests and $0.0029 per question.

[sourcery-ai]

Reviewer's Guide

This PR upgrades the engine to llmgate’s adaptive limiter, applies separate logged limits to chat and Judge traffic inside retry, and uses bounded parallelism for document processing, navigation questions, and independent page-ranking batches so provider capacity—not a fixed concurrency guess—controls throughput.

Sequence diagram for bounded parallel document processing

sequenceDiagram
    participant User
    participant TOCDump
    participant Workers as Document workers
    participant Limiter as Adaptive limiter
    participant Provider as LLM provider
    User->>TOCDump: run with -parallel N
    TOCDump->>Workers: start documents with bounded concurrency
    Workers->>Limiter: Build document TOC requests
    Limiter->>Provider: admit requests up to current limit
    Provider-->>Limiter: success or 429/transport failure
    Limiter-->>Workers: widen or narrow capacity
    Workers-->>TOCDump: write document results
    TOCDump-->>User: report wall time and limiter limit

Loading

Sequence diagram for parallel navigation and page-ranking batches

sequenceDiagram
    participant NavBench
    participant Navigator as JudgeNavigator
    participant Limiter as Shared Judge limiter
    participant Judge
    participant Provider as LLM provider
    NavBench->>Navigator: Navigate questions concurrently
    Navigator->>Navigator: Build independent page-ranking batches
    par Each ranking batch
        Navigator->>Limiter: Judge batch request
        Limiter->>Judge: admit request
        Judge->>Provider: send batch
        Provider-->>Judge: ranking response
        Judge-->>Limiter: success or failure
    end
    Limiter-->>Navigator: adapt concurrency from outcomes
    Navigator-->>NavBench: return selected pages and evidence

Loading

File-Level Changes

Change	Details	Files
Add adaptive per-provider concurrency control around chat and Judge requests.	Upgrade llmgate to v0.5.0 and wrap clients/Judges with adaptive limiters. Configure initial and maximum concurrency through YAML. Place Judge limiting inside retry so each attempt consumes a slot and failures adjust capacity. Log limiter widening and narrowing events with provider and failure details.	cmd/engine/main.go cmd/server/main.go config.example.yaml config.server.example.yaml pkg/config/config.go go.mod go.sum
Parallelize document and question workloads while sharing limiter-controlled provider capacity.	Add parallel worker execution for tocdump documents and navbench questions. Keep result ordering deterministic and synchronize reporting/output writes. Expose parallelism flags and print wall-clock and final limiter metrics. Share one Judge limiter across concurrent navigation or document operations.	cmd/tocdump/main.go cmd/navbench/main.go
Run independent navigation page-ranking batches concurrently.	Build all request-sized batches before dispatch. Execute batches concurrently with cancellation on the first error. Synchronize request counts, usage aggregation, and score updates. Make mock request-count tests safe under concurrent execution.	pkg/retrieval/judgewalk.go pkg/retrieval/judgewalk_test.go

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 3de71e8b-c17c-44fb-93bd-4a37cc924c9b

📥 Commits

Reviewing files that changed from the base of the PR and between b07a642 and 0172c13.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (10)

• cmd/engine/main.go
• cmd/navbench/main.go
• cmd/server/main.go
• cmd/tocdump/main.go
• config.example.yaml
• config.server.example.yaml
• go.mod
• pkg/config/config.go
• pkg/retrieval/judgewalk.go
• pkg/retrieval/judgewalk_test.go

 _________________________________________
< Make it work, make it right, make it 🥕. >
 -----------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}