perf(insights): rank investigation candidates with Jev - #849
izadoesdev wants to merge 3 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
The latest updates on your projects. Learn more about Unkey Deploy
|
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
1 participant
Insights candidate selection uses one bounded Jev evaluation to rank or exclude optional work, then hands ordered signal keys to the native planner. Investigation questions and measurement constraints pass through unchanged. Due rechecks, critical reliability, correlation grouping and run limits remain code-owned. Strict response validation, probability-rounding bounds, request limits, deadline handling and usage accounting preserve the existing fallback.
This remains a research draft. The acceptance bar is better useful findings and faster completed investigations. Lower selector cost alone does not meet it.
Fresh quality and latency comparison
Three matched synthetic cases ran the original Luna selector plus investigator against Jev plus the investigator with the original question unchanged. Both selectors retained the same target in these cases. All six investigations and both failures are retained; tools used static synthetic data. Times below include selection and the target investigation, but exclude other selected targets and real tool/network I/O.
These results do not demonstrate a better-and-faster investigation system. The earlier fixed-objective comparison is historical: the generic appendix is now removed, along with its effect on subject-context recall. The original native agent and its evidence/publication checks are unchanged.
Earlier frozen selection holdout checks passed 12/12 for both models, with median selection latency 549 ms versus 3,481.5 ms. Those checks were too coarse to prove equivalent business findings: one calibration case counted both portfolios as useful while Luna selected delivery and Jev selected generic traffic alongside the protected error. Jev also retained one extra explained investigation. Do not use selector pass counts or token savings as a product-quality claim.
The real-data native-agent comparison remains unrun pending explicit approval for the external model payload. Nothing has been deployed or published to customer Insights.
Validation
bun run lintbun run check-types— all 33 tasks passedbun run test --filter=@databuddy/insights— 687 passed, 6 optional live cases skippedbun run test— all 29 tasks passedScope: candidate selection, its existing evaluation harness, and usage pricing. No schema, UI, credentials or second agent loop. No unmerged dependencies. Known overlaps: #842 also pins Gateway 4.0.85 and touches bun.lock; #751 touches generation/business-context documentation. AI-assisted, maintainer-directed implementation.