perf(slack): classify thread replies with Jev - #842
izadoesdev wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
ef905b2 to
0170391
Compare
|
The latest updates on your projects. Learn more about Unkey Deploy
|
1 participant
Unmentioned follow-ups in engaged Slack threads now use Jev's Boolean evaluation instead of generating a JSON routing decision with Gemini. The classifier uses structured speaker/history context, a compact prompt, a 0.5 decision threshold and a two-second deadline, retaining the existing silent fallback. Returned question IDs, answer types and probability bounds are validated before use.
The evaluator uses pinned
@ai-sdk/gateway4.0.85 with the existing Gateway credential. The application's chat models remain on AI SDK 6. The reply gate is a directly injected function; it reuses the shared Slack message type and the classifier's timeout default. NativeAbortSignal.timeoutreplaces manual timer management. Existing mention, stop-command, Slack Connect and action-permission handling stays in place.A fresh synthetic holdout of 36 conversations, each repeated twice, produced 68/72 correct Jev decisions versus 67/72 with Gemini. Median classifier latency was 475 ms versus 748 ms; mean cost per request with reported cost was 72% lower. Jev had two false replies, two missed replies and one timeout; Gemini had five false replies and no misses/timeouts. The timeout's cost is unknown. These are classifier measurements on a small synthetic set, not end-to-end Slack performance claims. Prompt and threshold were frozen using separate calibration cases before the holdout. The cleanup preserves that prompt, threshold and provider payload.
Validation:
bun run lintbun run check-types— all 33 tasks passedbun run test— all 29 tasks passed; AI suite 1034 passed, 33 skippedScope: reply classification only. Skill/tool/parameter routing is not integrated. No schema or Slack installation changes. No dependent PRs; the package manifest overlaps dependency-update PRs #707 and #807.