Skip to content

docs: add intro video and make the README agent-agnostic - #157

Open
uipreliga wants to merge 2 commits into
mainfrom
docs/readme-generic-agents-video
Open

docs: add intro video and make the README agent-agnostic#157
uipreliga wants to merge 2 commits into
mainfrom
docs/readme-generic-agents-video

Conversation

@uipreliga

Copy link
Copy Markdown
Collaborator

Summary

Two things, both docs/copy only — no source or behavior changes.

1. Add the intro video. New ## Watch the intro section in the README with a clickable YouTube thumbnail and the linked title: Coder Eval: UiPath open-source framework to test AI Coding Agents. Placed after the feature list so it does not orphan those bullets under the heading.

2. Make the framing agent-agnostic. The README and the Pages stub both read as a Claude Code tool, even though the framework has driven four harnesses for a while. OpenCode was missing from several surfaces entirely. New tagline throughout: "Playwright for coding agents".

README

  • Title: "…AI coding agents and their skills" (was "Claude Code skills").

  • Tagline under the badges: Playwright for coding agents — one declarative test file, any agent runtime, a real sandbox, and a pass/fail gate in CI.

  • Intro states the switch cost explicitly: changing harness is one field (agent.type); tasks, criteria, scoring, telemetry, and reports stay the same.

  • Quick Start prerequisites were "the Claude CLI" only. Now a table of all four agents with agent.type, install command, and a link to each guide:

    Agent agent.type Runtime
    Claude Code (default) claude-code brew install claude
    OpenAI Codex codex pip install 'coder-eval[codex]'
    Antigravity (Gemini) antigravity pip install 'coder-eval[antigravity]'
    OpenCode opencode npm install -g opencode-ai
  • Task Definition notes swapping agent.type or -D agent.type=opencode, and links Run-Limit Parity.

  • Known limits: "bring your own agent runtime and model credentials", including an OpenRouter key for open-weight models via OpenCode.

  • Acknowledgments credit all four harnesses, not just the Claude Agent SDK.

Left Claude Code specific on purpose: the "Use inside Claude Code" plugin marketplace section — it genuinely is — but it now says so, and notes the suites authored there run on every harness.

Pages stub + package metadata

  • .github/pages-stub/index.html: <title>, <meta name="description">, and the visible lead paragraph. OpenCode was absent from all three. The old lead also read "agents and their Claude Code skills", which parsed wrong even under the Claude-only framing.
  • pyproject.toml: the stub's own comment says its description mirrors the package metadata, so description and keywords gain OpenCode — otherwise that comment stops being true the moment the stub changes. This ships to PyPI on the next release; happy to drop it if you would rather keep metadata out of a docs PR.

Verification

make lint (388 passed) and make check green. No doc-surface rule tripped — CE026's Action snippet, its prerequisite steps, and the CE028-generated docs index tables are all untouched. Both YouTube thumbnail URLs verified 200.

Not covered

The coder-eval.com website source is not in this repo, so its copy still carries the old Claude-Code-first framing and needs a separate change.

🤖 Generated with Claude Code

https://claude.ai/code/session_016v9L8gXbq4MzH6V9JZPqcb

uipreliga and others added 2 commits September 4, 2026 15:26
Add the YouTube intro video and rework the README so it reads as a
harness-agnostic framework rather than a Claude Code tool:

- New tagline: "Playwright for coding agents".
- Title drops "Claude Code skills" for "their skills".
- Intro names all four harnesses and states that changing harness is one
  field (agent.type) with tasks/criteria/scoring unchanged.
- Quick Start prerequisites become a per-agent runtime table (claude-code,
  codex, antigravity, opencode) instead of the Claude CLI only.
- Task Definition section explains swapping agent.type and links Run-Limit
  Parity.
- Known limits: bring your own agent runtime as well as model credentials;
  mention OpenRouter for open-weight models via OpenCode.
- Acknowledgments credit every harness the framework drives.

The plugin section stays Claude Code specific — it is, factually — but says
so and notes the suites it authors run on every harness.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016v9L8gXbq4MzH6V9JZPqcb
Bring the GitHub Pages stub in line with the reworked README: drop
"Claude Code skills" from the title, name all four harnesses (OpenCode
was missing entirely), and lead with the "Playwright for coding agents"
tagline in both the meta description and the visible lead paragraph.
The old lead also read "agents and their Claude Code skills", which
parsed wrong even for a Claude-only framing.

The stub's own comment says its description mirrors the package
metadata, so pyproject's description and keywords gain OpenCode too —
otherwise that comment stops being true the moment the stub changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016v9L8gXbq4MzH6V9JZPqcb
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

Claude finished @uipreliga's task in 1m 43s —— View job


Code Review in Progress

Todo List:

  • Read .github/code_review.md for review standards
  • Read CLAUDE.md for project conventions
  • Review the full diff (git diff origin/main...HEAD)
  • Read full context of changed files
  • Check cross-file consistency
  • Analyze "what's missing"
  • Provide structured review feedback

Starting review now...

@bai-uipath bai-uipath left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix what you agree with, otherwise lgtm. The agent-agnostic rework is the right call and the video placement is clean. The notes below are all about the framing that's left.

  • The skills-only read survives, and that's the one costing us. A prospect benchmarking agents on their own domain task bounced off this page believing the framework was skills-only. The title still leads with skills, the audience line still says "built for CLI and skill builders", and the "not an agentic coding benchmark" disclaimer reads as "we don't do domain benchmarks". Fix: keep the SWE-bench contrast but recast it as "not a fixed leaderboard: you bring the tasks and the scoring", and widen the audience line to include benchmark authors.

  • The docs site still says "Claude Code skills". docs/index.md (title, description, H1), docs/llms.txt, docs/comparison.md, and mkdocs.yml's site_description live in this repo, sync to coder-eval.com/docs, and outrank both surfaces this PR touches. Cheap to fold in here.

  • The Quick Start asks for the contributor toolchain. --extra dev pulls pytest, ruff, pyright, and pre-commit onto someone who just wants one run, and the pip install path is parked under a heading that reads CI-only. uv itself is the right call given the 3.13 floor, so this is about which extras and which ordering, not about uv.

  • Minor: the prerequisites table hands users pip install three lines above uv sync (should be --extra codex / --extra antigravity); "installs the harness adapters, never the agents themselves" isn't true for codex or antigravity, whose extras ship the runtime; and the agent roster is now restated in five places with nothing holding it to AgentType, which is how OpenCode went missing in the first place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants