MyOpenCRE mapping: reuse Module C Librarian retriever/reranker (RFC TODO 3) #965

Activity

1. Glitch-On commented on Aug 29, 2026

   

   Glitch-On

   on Aug 29, 2026

   Last edited by Glitch-On

   More actions

   @northdpole https://github.com/northdpole, https://github.com/northdpole I'd like to take this — the merge-order blocker is now clear #937 #937 #937 and #957 #957 #957 are both merged to main , so this is unblocked and a natural next step.

   My read on the approach:

   Implement run_mapping in application/utils/mapping_job.py by composing the existing Librarian primitives directly rather than re-implementing them: build_retriever(...) from candidate_retriever.py for the top-20 shortlist (RetrievalAudit.candidates[]), then CrossEncoderReranker.rerank(...) from cross_encoder.py to fill reranked[] and produce the top-N — the same RetrievalAudit contract Module C already emits, so no parallel data shape gets invented.

   For the retriever backend, default to in_memory unless CRE_LIBRARIAN_RETRIEVER_BACKEND=pgvector is set and the DB is actually Postgres with embedding_vec populated (per the W3 loud-warning behavior) — reuses the existing config gate instead of adding a second one for MyOpenCRE.

   Reuse get_embedding_contents_by_doc_type from db.py (added in #957) rather than writing a new embeddings-lookup query, and reuse the UUID→external_id translation pattern from scripts/evaluate_librarian.py/cre_main.py's resolve() fast path, since that's a real gotcha already hit twice in the Librarian PRs (recall read 0% until fixed).

   Map RetrievalAudit.reranked[] scores to the user_standard_sections status enum with explicit thresholds documented in the RFC (proposing score_rerank >= 0.80 → AUTO_MAPPED, 0.50–0.80 → NEEDS_REVIEW, < 0.50 → UNMAPPED, informed by C.2's measured 75% top-1 accuracy at the 0.80 target) — happy to adjust these once I have a fixture set to validate against.

   Keep the rq job wrapper in mapping_job.py unchanged; only its internal call swaps to the two Librarian calls above.
   Tests: a fixture upload asserting deterministic status counts (relying on the stable-sort fix already landed in candidate_retriever.py/cross_encoder.py for tie-order determinism), plus a regression test asserting USER_CONFIRMED rows are excluded from the run_mapping write path on rerun.

   One decision to confirm before I start: should the confidence thresholds live as constants in mapping_job.py itself, or in librarian/config_loader.py alongside CRE_LIBRARIAN_* so both MyOpenCRE and future Module C consumers share one source of truth? I'd lean toward the latter for consistency, but happy to scope it to mapping_job.py only if you'd rather keep MyOpenCRE's policy decisions separate from Module C's config surface.

   Will keep the PR scoped to run_mapping + threshold mapping + tests per the acceptance criteria, no changes to C.3–C.5 or the pgvector migration itself.
2. tanishbhandari11t commented on Aug 30, 2026

   

   tanishbhandari11t

   on Aug 30, 2026

   More actions

   @northdpole Hello, I would be interested in contributing towards this issue too. I have gone through the requirement for the same and the Module C retrieval/reranking flow as of now and find the idea of reuse to be especially fascinating.
   From what I understand, @Glitch-On has already mentioned that he would like to contribute to this topic. However, if there are no further contributors and if the problem statement is still available, I would definitely like to contribute.

   Thanks!