Run a scenario across every SDK: sdk-matrix workflow, script and toolchain image - #496
Draft
pcarleton wants to merge 10 commits into
Draft
Run a scenario across every SDK: sdk-matrix workflow, script and toolchain image#496pcarleton wants to merge 10 commits into
pcarleton wants to merge 10 commits into
Conversation
uv sync --all-packages fails on a case-sensitive filesystem for the v1.x branch: examples/clients/simple-chatbot declares README.md but ships README.MD, so hatchling refuses to build it. Sync the two conformance packages one at a time, which is what the SDK's own v1.x conformance CI does and what the python-sdk entry already does for main.
Fans one scenario list, suite or requirement set out over KNOWN_SDKS by invoking the existing 'conformance sdk' command per SDK, so cloning, building and running stay in one place. One SDK failing to build never stops the others: each invocation's log is classified (checkout, build, server start, ran) and the toolchain error is carried into the report. Writes matrix.json (sdk x scenario x check, plus errors verbatim and per-SDK toolchain versions) and matrix.md (SDK overview, scenario summary, per-check tables, failure messages), and prints the markdown. --merge re-renders from saved matrix.json files, which is how CI combines per-SDK legs; --ref tests another conformance ref or PR; --harness-dir drives an already-built checkout. All SDK-produced text is escaped before it reaches the markdown. npm run sdk-matrix is the entry point; results default to sdk-matrix-results/ (ignored).
One job per SDK on ubuntu-latest with that SDK's toolchain (setup-node and corepack, setup-uv, setup-go, rust-toolchain, setup-dotnet at the band csharp-sdk's global.json pins, setup-ruby, setup-java), running scripts/sdk-matrix.mjs for that leg against the requested conformance ref or PR, then a report job that merges the legs into the run summary and an optional job that upserts one sticky comment on the PR. Same trust split as traceability.yml: the jobs that build and run SDK code (and code from the ref under test) are read-only, keep no git credentials, hold no token in the SDK step's environment and use no actions caches; the report job only checks out the dispatching branch; the comment job holds the single write permission, checks nothing out and posts the table as data.
…ocally docker/sdk-matrix installs Node + corepack, uv, Go, rustup, the .NET SDK, Ruby + Bundler and a JDK from their public sources on ubuntu 24.04. scripts/sdk-matrix-docker.sh builds it on first use and runs scripts/sdk-matrix.mjs against the bind-mounted checkout with SDK clones, builds and package caches on a named volume, so reruns are fast. Registries default to each ecosystem's public one; mirror configuration can be forwarded per run with explicit flags (--npmrc, --cargo-config, --pip-conf, --uv-config, --nuget-config, --gemrc, --env NAME). The wrapper never mounts SSH keys, git credentials or tokens, publishes no ports, and handles rootless Docker by running as container uid 0 (the invoking host user).
README section covering the workflow trigger, the docker wrapper and the plain script; AGENTS.md and CONTRIBUTING.md ask authors to attach the matrix when a change adds a check to an existing scenario or changes a check's severity.
…this branch workflow_dispatch is only accepted once the workflow exists on the default branch. Revert before merge.
Scenarios in the same area emit the same check ids (every auth/* flow shares ~19), so per-scenario tables repeated themselves. Union the ids across scenarios into a single table, show the worst status per cell, and break out only the checks whose outcome differs between scenarios.
… table doesn't count
…aseline The question the matrix exists to answer is whether a harness change turns an existing SDK's CI red. Read the baseline file 'conformance sdk' already passes as --expected-failures, and lead the report with a Regressions section: failing checks that baseline does not excuse, baseline entries that now pass (stale, which also fails that SDK's CI), and SDKs that could not be built or run. Baselined failures stay visible but are marked separately in the summary cells, the SDK x check table and the message list. --strict now means 'would turn some SDK's CI red'.
…ow from this branch" This reverts commit fab1e88.
commit: |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+216
to
+220
| - uses: ruby/setup-ruby@v1 | ||
| if: contains(matrix.spec, 'ruby-sdk') | ||
| with: | ||
| ruby-version: '4.0' # what ruby-sdk's own conformance CI runs | ||
| - uses: actions/setup-java@v5 |
Comment on lines
+226
to
+230
| - name: Build the harness under test | ||
| working-directory: under-test | ||
| run: npm ci && npm run build | ||
|
|
||
| - name: Run the matrix leg for this SDK |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A way to run one conformance selection (a scenario list, a suite, or a requirement set) across every SDK in
KNOWN_SDKSand get back a single SDK x check table, from any of three places:.github/workflows/sdk-matrix.yml(primary):workflow_dispatchwithsdks,mode,scenario|suite|requirements,ref,pr,pr_comment, plus a weekly schedule. One job per SDK onubuntu-latestwith that SDK's toolchain (setup-node + corepack, setup-uv, setup-go, rust-toolchain, setup-dotnet at the version csharp-sdk'sglobal.jsonpins, setup-ruby, setup-java), then areportjob that merges the legs into the run summary and an optionalcommentjob that upserts one sticky comment on the conformance PR.scripts/sdk-matrix-docker.sh+docker/sdk-matrix/Dockerfile: the same run locally with every toolchain in one image and a cache volume, for people who do not want six toolchains on their laptop.scripts/sdk-matrix.mjs(npm run sdk-matrix): the plain script both of the above call. It is thin orchestration over the existingconformance sdkcommand (no new way to clone, build or run an SDK), never lets one SDK's failure stop the others, and writesmatrix.json(sdk x scenario x check, plus build/exec errors verbatim) andmatrix.md. The report reads each SDK's own expected-failures baseline (the fileconformance sdkalready passes) and leads with a Regressions section: failing checks that baseline does not excuse, stale baseline entries, and SDKs that could not run. That is the "would this change turn an SDK's CI red" answer; baselined failures are shown separately.Why
Reviewing a check change means answering "what does this do to each SDK". Today that takes whichever toolchains the reviewer happens to have installed, and the rest get predicted by reading SDK code. For #488 three of eight SDKs were checked that way. The harness already knows how to clone, build and drive every SDK (
conformance sdk), so the missing piece was fan-out, aggregation and a place to run it that has all the toolchains. Hosted runners are that place; the container is the offline equivalent.Security model of the workflow
Same split as
traceability.yml.planandrunexecute code from the ref under test and third-party SDK code withcontents: read,persist-credentials: false, no secrets, no token in the environment of the step that builds and runs SDKs, and no actions caches.reportchecks out only the dispatching branch and runs no SDK code.commentis the only job withpull-requests: write; it checks nothing out and posts the rendered table read from the artifact, with all SDK-produced text escaped as data. The workflow is dispatch/schedule only, so it always runs as defined on a branch of this repository; a fork PR's harness code only ever runs in the unprivileged jobs, andplanflags fork PRs.Testing
scripts/sdk-matrix.test.ts: aggregation and rendering against canned result directories, log classification (build failure, missing toolchain, server never ready, checkout failure), merge of artifact-style directories, markdown escaping, and a cross-check that theKNOWN_SDKSsource parse matches the module's keys.--mode client --scenario auth/metadata-var2,auth/metadata-defaultand--mode server --scenario tools-listacross all SDKs.workflow_dispatch --refworks): a targeted run for the feat(client/auth): check the resource parameter matches the PRM-published identifier #488 scenarios, apr=488dispatch (resolves to the merge commit), and a fullsdks=all mode=bothrun, which covers 44 client and 31 server scenarios per SDK in about 3 minutes wall time. The full report from that last run is in its run summary: https://github.com/modelcontextprotocol/conformance/actions/runs/34057591441plancan only dispatch by name once the workflow has run at least once in the repo, which it now has; after merge the documented command works as-is.Docs
README section "Running a Scenario Across All SDKs" (workflow trigger, docker wrapper, plain script), and a line in AGENTS.md / CONTRIBUTING.md asking authors to attach the matrix when a change adds a check to an existing scenario or changes a check's severity.