Skip to content

Leaderboard v1: OpenCode / Gemini 4 Argon - #113

Open
c-suvamsa wants to merge 1 commit into
mainfrom
leaderboard/opencode-gemini-4-argon-2026-10-19
Open

c-suvamsa wants to merge 1 commit into
mainfrom
leaderboard/opencode-gemini-4-argon-2026-10-19

Conversation

@c-suvamsa

@c-suvamsa c-suvamsa commented Oct 19, 2026 •

Copy link
Copy Markdown

Summary

  • Agent / Model: Opencode / Gemini 4 Argon (Google).
  • Source Harbor job URL: https://hub.harborframework.com/jobs/e361aad1-8f0b-42ad-8f7a-44f88e998770.
  • Run: 2026-10-19; tasks/v1 via Daytona; 14 tasks × 10 attempts; concurrency 4; Harbor 0.24.0.
  • Route: OpenRouter; recorded model openrouter/google/gemini-4-argon.
  • Result: 103/140 reward-1.0 trials (73.6%); 190,454,394 recorded input + output tokens.
  • Cost: $95.98.

PR checklist – leaderboard row entry

Provenance and access

  • Source job is public and owned by the submitter; entry header and job URL match.
  • Required metadata and metrics present; 140 distinct trial IDs match n_trials.
  • Reward-derived accuracy, pass@k, and recorded token totals reconcile with the source job.
  • All 14 task content digests match enterprise-bench main; local tasks/v1 path used instead of a registry-pinned version.
  • Default scored timeout multipliers and no CPU/memory/storage overrides; no scored trial omitted.
  • All trials have ATIF trajectories and usage records.
  • Reviewer to verify logged-out trajectory replay, benchmark-answer access boundaries, and applied upstream model route.

Validation

  • Main-branch leaderboard validator passes locally.
  • Repository validate and validate-leaderboard CI checks pass.

Notes for reviewers

The run recorded 10 AgentTimeoutError trial(s) at the default limit; all are retained with their recorded rewards. 1 timed-out trial(s) received reward 1.0 because the verifier still had a submitted answer.

@nimit2801

Copy link
Copy Markdown
Collaborator

Thanks @c-suvamsa. Nimit's intake workflow has picked up this OpenCode / Gemini 4 Argon leaderboard submission. Both CI checks pass; technical review is still required, not an approval. Review will cover source-trace metric reconciliation, logged-out replay and benchmark-answer access boundaries, and the applied model route plus timeout handling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants