Applied AI: evaluation, inference, agents. I build and measure LLM systems, then turn what I learn into tools and write-ups. Product leader at NVIDIA by day; a two-GPU applied-AI lab by night.
Current work
- scribe-bench — open-data evaluation harness for ambient clinical documentation: seven ASR systems, note generation, LoRA fine-tuning, a claim-level verifier, and clinician-defined scorers on PriMock57 and ACI-Bench. Paper draft included.
- coding-agent-bench — a coding-task benchmark for local models and agent harnesses, with the four ways the harness lied to me and how the scoring was fixed.
- Serving frontier open models on RTX PRO 6000 Blackwell — NVFP4 and FP8 recipes, kernel workarounds, expert offload, measured tokens/s.
- Refusal removal with measurement — method and evaluation scripts for abliteration that keeps benchmark scores intact, plus evidence that a popular community upload was a no-op. No weights.
- AutoTalent — an agentic recruiting product with a 32-tool MCP server (private).
Everything here has a run behind it. Field notes with numbers: sethpratt.com/notes.