AI labs training agents on real-world tasks need environments where an agent can attempt a task and receive an honest, automatic grade — not "did it say something plausible" but "did it actually do the job, provably, by code."
TaskForge is a scaled-down version of that infrastructure: two realistic tool-use environments (a spreadsheet system and a support ticket system), a deterministic verifier that grades agent task completion by state-diff, a benchmark harness that runs multiple models in parallel, and a failure-mode graph that classifies why models fail across environments.
Built to demonstrate understanding of what AI labs are actually building — not a chatbot wrapper.
Six panels total — the benchmark matrix and failure graph each span two screenshots because they did not fit in one frame.
Pass rate matrix: GPT-4o-mini vs Gemini-2.5-Flash across both environments and all difficulty levels
Episode trajectory viewer: every tool call logged with input, result, and token cost
Failure graph: REPORT_INCOMPLETE appears in 11 episodes across both models
Live benchmark progress: SSE-streamed episode completion log
Primary run: phase9-final, 60 scored episodes, ~$0.048 (judge not included). Pass = final_score >= 0.5. n=3 episodes per (model × task) cell (easy rows n=6 — two easy templates × 3).
| Difficulty | GPT-4o-mini (Spreadsheet) | GPT-4o-mini (Ticket) | Gemini-2.5-Flash (Spreadsheet) | Gemini-2.5-Flash (Ticket) |
|---|---|---|---|---|
| Easy | 100% | 100% | 100% | 100% |
| Medium | 100% | 100% | 100% | 100% |
| Hard | 100% | 67% | 100% | 0% |
| Expert | 100% | 33% | 100% | 100% |
| Finding | Detail |
|---|---|
| Dominant failure | REPORT_INCOMPLETE: 11/27 failure edges, both models |
| Cross-env difference | Gemini matches GPT on spreadsheet, diverges sharply on ticket tasks |
| Zero-step failure | Gemini ticket-hard: 3/3 episodes with 0 tool calls |
| Aborted episodes | 3 token-budget aborts still scored ≥ 0.5 (close-then-verify confirmed correct) |
Full write-up: findings.md.
Task Template + Seed
│
▼
┌──────────────────┐ tool calls ┌────────────────┐
│ AI Agent │ ──────────────▶ │ Environment │
│ (any LLM via │ ◀────────────── │ (Spreadsheet / │
│ OpenRouter) │ tool results │ Ticket) │
└──────────────────┘ └───────┬────────┘
│
trajectory │ final state
▼ ▼
┌──────────────────┐ ┌────────────────┐
│ Episode Log │ │ Verifier │
│ (append-only, PG)│ │ (deterministic │
└──────────────────┘ │ + LLM judge) │
└───────┬────────┘
│ score + criteria
▼
┌────────────────┐
│ Failure Graph │
│ (Kuzu) │
└────────────────┘
FastAPI + PostgreSQL (ACID) + Redis (hot state + queues + SSE) + Kuzu (failure graph) + Next.js.
Prerequisites: Docker Desktop, an OpenRouter API key ($0.05 gets you a full benchmark run)
git clone https://github.com/AnassNadeem/taskforge.git
cd taskforge
cp .env.example .env
# Open .env and set OPENROUTER_API_KEY=your_key
docker compose up -dThat's it. Four containers start (Postgres, Redis, API, Dashboard).
- API: http://localhost:8000/docs
- Dashboard: http://localhost:3000
First-time graph population (run once after first startup):
# In .env, set RUN_GRAPH_BACKFILL_ON_STARTUP=true
docker compose restart api
docker compose logs -f api | Select-String "backfill.complete"
# Then set RUN_GRAPH_BACKFILL_ON_STARTUP=false and restart apiRun your first benchmark:
# The default GPT-4o-mini agent is pre-seeded
# Open http://localhost:8000/docs → POST /benchmarks
# Or use PowerShell:
$body = '{"name":"my-first-run","models":["bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"],"environments":["spreadsheet"],"task_count_per_env":1,"max_workers":3}'
Invoke-RestMethod -Method Post -Uri http://localhost:8000/benchmarks -ContentType "application/json" -Body $bodyThen open the dashboard and watch it run live.
Spreadsheet — 26×100 in-memory grid, financial year layout, one seeded defect per instance. Six tools: read_cell, write_cell, read_range, set_formula, get_column_summary, list_named_ranges. Five tasks from "read B7" through "build a summary row."
Ticket System — 5 customers, 8–12 tickets, explicit status machine. Eight tools: list_tickets, get_ticket, get_customer, update_ticket_status, set_priority, assign_ticket, add_resolution_note, close_ticket. Five tasks from "fetch TK-003" through triage-and-assign.
Both implement EnvironmentBase. The runner never reads environment state except through tools.
Deterministic checkers compare final state and trajectory to the rubric (cell values, ticket status, who was assigned). LLM-as-judge is used only when the criterion is about the natural-language report. Close-then-verify: the episode close transaction writes final_state and a terminal status, commits, then verification runs as a separate call — the LLM judge never sits inside a database transaction.
pip install -e ".[dev]"
pytest tests/ -v # 238 tests
cd dashboard && npm run devCI runs the same suite on every push (Postgres 16 + Redis 7 service containers).
- Append-only steps — each tool call is written through to Postgres; no step row is ever updated.
- Close-then-verify — LLM-as-judge stays outside the ACID close transaction; aborted episodes still get a score.
- Deterministic seed derivation — the runner always
env.reset(seed); Redis is working memory only. - Kuzu for the failure graph — criterion failures become queries like "which patterns appear in 2+ models?"
- Per-worker isolated engine — each benchmark worker builds its own
pool_size=1engine withexpire_on_commit=False.
Windows PowerShell: use Invoke-RestMethod instead of curl.
curl in PowerShell is an alias for Invoke-WebRequest which behaves differently.
Port 5432 in use: the compose file publishes Postgres on host port 5433 (5432 may be taken by another local Postgres). Connection strings use 5433 on the host.
Kuzu graph is empty: run the backfill (see First-time graph population above). New episodes auto-ingest after every benchmark — backfill is only needed for historical data.
Uvicorn --reload hangs on Windows: run without reload flag.
uvicorn taskforge.api.main:app (no --reload).
One API process at a time: Kuzu uses a single-writer file lock. If a previous process is still running, the new one will fail on Kuzu init.
Personal research / portfolio project. See the repository for contact.