Repository navigation
CI: wire the Librarian eval harness in as a merge gate (explicit 100% + optional recall@k) #956
Description
Follow-up from @northdpole's reviews on #925 and #937 (Module C - The Librarian).
The gap: the eval harness (scripts/evaluate_librarian.py) already fails loudly on an explicit-slice regression - it exits non-zero if the "explicit" slice drops below 100% on the golden dataset. The problem is nothing in CI actually runs it. make test only runs the unit tests, so today that 100% gate exists on paper but isn't enforced. A change could quietly break the explicit slice and still merge green.
What we'd like to do: wire the harness into CI so it guards merges automatically.
- Add a small subprocess test (or a
Makefiletarget) that runsevaluate_librarian.pyagainst the committed golden dataset (application/tests/librarian/fixtures/golden_dataset.json). If the explicit slice isn't at 100%, the check fails and blocks the merge. This runs fully offline - no API key needed. - Later, extend the same setup to cover the C.1 retriever's recall@k. That part needs live embeddings (a populated cache DB + an embedding API key), so it can either be an optional CI job or a documented manual pre-merge check.
The offline explicit gate is the priority since it's deterministic and needs no credentials.
Scope: Module C. Not blocking W1–W3 - this just makes sure the harness is doing its job before the heavier ML slices (W4+) land.
Activity
tanishbhandari11t commentedon Aug 30, 2026
Hello @PRAteek-singHWY, I’d be interested in tackling this task if it is still up for grabs.
I have looked at the current
evaluate_librarian.pytest framework and the idea of the golden dataset mentioned here. The strategy would be to maintain full determinism and offline operation of the current implementation by wiring up the existing explicit slice 100% check as part of CI with no API credentials or dependencies introduced.Then there would be coverage for regression in failure/exit code logic such that going below 100% would automatically block the CI process.
Regarding the recall@k, it would be done separately from the main implementation as was discussed in the issue due to the dependency on the embeddings framework.
Would you be willing to assign it to me if it is still up for grabs?