feat: add native E2B backend for cloud-based agent evaluation - #11
Merged
Merged
Conversation
…and published by maintainers
…names survive the submit_paths match
…LI docs with code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds native E2B support to SForge, allowing EdgeBench tasks to run in managed cloud Sandboxes without requiring a local Docker daemon or Kubernetes cluster.
Work, Judge, and Game environments are created from pre-built official E2B Templates, while SForge preserves the existing task definitions, iterative evaluation workflow,
scoring semantics, and result layout used by the Docker and Kubernetes backends.
What changed
E2B backend
E2BBackendimplementation for Sandbox creation, command execution, file transfer, service exposure, and cleanup.Long-running evaluation reliability
on_timeout=kill, explicit deletion, and bounded cleanup fallback.Agent and CLI support
--backend e2band the related configuration options.--effortoption translated to each supported agent.Documentation
Architecture
The evaluation flow remains consistent with the existing SForge model:
sforge runcreates an E2B Work Sandbox from the task's published Work Template.No managed Controller Sandbox is required.
Compatibility
e2bdependency extra.Validation
The E2B path was validated with the complete 51-task open-source EdgeBench suite using the official leaderboard resource specifications.
Additional checks:
git diff --checkUser-facing example