Skip to content

Modules have clients, not dependencies - #22

Draft
eunomie wants to merge 9 commits into
dagger:mainfrom
eunomie:python-unified-clients-lead-afe256c1
Draft

Modules have clients, not dependencies#22
eunomie wants to merge 9 commits into
dagger:mainfrom
eunomie:python-unified-clients-lead-afe256c1

Conversation

@eunomie

@eunomie eunomie commented Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Modules have clients, not dependencies

A module's dependency and a standalone generated client are the same thing:
generated bindings for one module plus a serve preamble. This series makes the
Python SDK generate exactly that, once, for both uses — the port of
dagger/java-sdk#17's design, re-verified against v1.0.0-beta.11 and
redesigned where Java's constraints do not apply to Python. Design doc:
future/done/modules-have-clients.md.

What changes

  • A dagger-module.toml module vendors sdk/src/dagger/client/gen.py with the
    core API only, plus sdk/src/dagger/clients/<dep>.py for every declared
    dependency (under its final, possibly aliased, name) and
    sdk/src/dagger/clients/<self>.py: a client for itself, which is how a
    module calls itself through the engine and gets function-level caching on
    its own calls. dag.dep() is gone; module code does
    from dagger.clients.dep import dep and await dep().fn().
  • Standalone clients: generateClient, initClient and a client rollup
    in the @generate hook, mirroring the Go SDK's shape on beta.11's API.
    dagger api client init python <path> <module> registers, seeds a
    pyproject.toml and generates <path>/sdk/ — a Python project that opens
    its own engine session. The client file is byte-identical to the one a
    module vendors for the same module (checked in e2e).
  • Serve preamble (dagger.client._binding): the binding rides on the
    query context; outside a module runtime it serves the bound module
    unconditionally on first use per session (no __type probe), refetches the
    schema, and remembers the tuple on the ClientSession. Inside a module
    runtime it is a no-op: the engine builds that session with the module's
    dependencies and the module itself already served, and a baked local path
    would resolve against the caller's workspace for a module consumed from
    git. The signal is a flag the runtime entrypoint sets, not DAGGER_MODULE
    (a CLI selector users export).
  • Code generator: --mode core|client partitioned on the engine's
    @sourceMap(module:) directive; core references in a client go through one
    alias of the core package; entry point and shims for module-owned fields on
    core types; naming and collision rules; a bound module declared below
    1.0.0 is refused.
  • Local dependency staging by ownership: dependencies registered to this
    SDK are generated recursively and overlaid (the engine's
    generateLocalDependencies fails in a value workspace); dependencies
    registered to another SDK are handed to the engine; unregistered ones are
    assumed committed. The self client is produced through a staged-workspace
    bootstrap (pass 1 carries the committed self client or a stub).
  • runtime/: requireGeneratedFiles also requires the clients package
    marker; the runtime fixture's vendored sdk/ is regenerated.

Verified against a live beta.11 engine

  • clientsGenerateCheck: a module generates from nothing with dep, an
    aliased greeter and its own client; second generate empty; an added
    function appears in the regenerated self client; a dropped alias is
    reported as a removal.
  • clientsRuntimeCheck: greet-via-dep, greet-via-alias and greet-self
    answer through the generated clients under a real CLI — including the self
    call the Java series could not test.
  • standaloneClientCheck / standaloneRuntimeCheck: byte-identity with the
    vendored client, core with Host, stamped CLI_VERSION, a git-bound client
    baking ref and pin, initClient seeds only pyproject.toml, the rollup
    materializes a registered client and is empty once applied; a plain Python
    process serves github.com/dagger/sdk-sdk by ref and pin into its session
    and calls it. The local-binding standalone flow (own session, workspace
    path) was verified by hand on the host; under nesting a process can only
    attach to the check's session.
  • foreignDependencyCheck: a dependency another SDK manages is handed to the
    engine rather than skipped.
  • Unit: 261 tests (tests/codegen, tests/mod, tests/client).

Found on the way

  • Workspace.withoutDirectory(<sub>) followed by withDirectory(<ancestor>)
    replaces the ancestor instead of merging (reproduced with a raw query).
    Every overlay composes the full directory first (Codegen.overlay);
    worth a dagger/dagger issue.
  • A Go module cannot be loaded through Workspace.moduleSource
    (Directory.asModuleSource fails on the relative-path runtime SDK), so the
    clients fixtures run on the engine's python runtime; the Go runtime keeps
    its own check.
  • For a local binding the engine cannot detect a same-name conflict on
    beta.11 (isSameModuleReference treats an empty AsString as "same");
    git bindings are conflict-checked. Recorded as a follow-up.

Pre-existing, left alone

  • initModule renders src/probe_2/ for a module named probe2
    (strcase.ToSnake) while pyproject.toml names probe2, so a module name
    with a digit cannot build. Unrelated to this series.
  • sdk-sdk:contract:* fail locally on main itself in this environment
    (.dagger/lock is a version-1 lock the local beta.11 engine refuses to
    parse); bisected with stg pop -a. Green in CI on main.

Breaking

Every dagger-module.toml module breaks on its next dagger generate:
dag.<dep>() → from dagger.clients.<dep> import <dep>. Legacy dagger.json
modules are untouched. Migration recipe in the README.

Review follow-ups

Module-level functions for module-owned fields on core types are now always named <parent>_<field> (only the entry point is bare), so an unrelated field addition never renames an export. generateClient, initClient and initModule reject paths that leave the workspace (.. segments, with \ treated as a separator as the engine does). The client project name is the snake-cased directory name, validated as a PEP 508 name. Generated docstrings and deprecation messages escape backslashes, quotes, CR and NUL, and long words are no longer split inside a literal. Aliased local dependencies are staged once.

eunomie added 3 commits August 27, 2026 19:27
@eunomie
future: modules have clients, not dependencies
43be2a1 
Port the java-sdk design (dagger/java-sdk#17) to the Python SDK: a module's
dependency and a standalone generated client become one artifact, a generated
client, made of bindings plus an idempotent serve preamble. Core moves to a
per-consumer dagger.client.gen, each bound module gets dagger/clients/<m>.py,
and a module generates a client for itself through a staged-workspace
bootstrap.

Every engine claim is re-verified on v1.0.0-beta.11 by source and by probing a
live engine from inside a Python module: serve-by-workspace-path deduplicates,
currentWorkspace resolves, and the client schema carries exactly one owned type
plus Query.<module>.

Signed-off-by: Yves Brissaud <yves@dagger.io>
@eunomie
codegen: attribute schema symbols to their module
7cd52ca 
The engine marks every type and field it installs on behalf of a module with
@sourcemap(module: "<name>"); core symbols carry no module. That directive is
the partition a client needs: the core API on one side, each module's own
symbols on the other.

partition.py reads it off the raw introspection result: which types a module
owns, which fields it contributes to core types (Query.<name> is its entry
point), and a narrowing of the result to core alone, with possibleTypes pruned
of dropped types so build_client_schema accepts what is left. A module's root
type is the return type of its Query field, read from the schema rather than
derived from its name: module e2e has root type E2E. Scalars get the same
directive stubs as the other kinds, so a module-owned scalar attributes
correctly instead of falling into core; unions, which the engine never emits,
are rejected rather than guessed at.

Nothing uses it yet.

Signed-off-by: Yves Brissaud <yves@dagger.io>
@eunomie
sdk: serve a bound module before its first query
d40a8a6 
A generated client is bindings plus a serve preamble. The bindings come later;
this is the preamble, in hand-written runtime so the generated files carry
data and no logic.

ModuleBinding names the one module a client is bound to: its final name, its
kind, a workspace-root-relative path or a git ref, and a pin. It rides on the
query context, and Context.execute serves every binding it carries before
building the request, so a lazily built chain serves on its first await and a
second session serves again. The serve is a raw GraphQL document sent
unconditionally the first time per session: the engine deduplicates a repeat
of the same source and reports a different one under the same name, which a
__type probe would silently paper over. The schema the session cached on
connect predates the serve, so it is fetched again before the binding is
marked served; the served set and the lock that keeps serve, refetch and mark
one step live on the ClientSession, and go away with it.

Inside a module runtime the preamble does nothing: the engine builds that
session with the module's dependencies and the module itself already served,
and a baked local path would resolve against the caller's workspace, not the
module's, once the module is consumed from git. The entrypoint marks the
process rather than the preamble reading DAGGER_MODULE, which is a CLI
selector users export.

Two guards come with it. A module's API can only use its own types and the
core API, so the converter rejects any class from dagger.clients before
classifying it; and a missing dagger.clients import is reported as the same
run-generate-and-commit error the runtime gives for any missing generated
file, instead of a bare traceback.

Signed-off-by: Yves Brissaud <yves@dagger.io>
@eunomie
eunomie force-pushed the python-unified-clients-lead-afe256c1 branch 2 times, most recently from 9559c30 to 2dc64c7 Compare August 27, 2026 19:45
eunomie added 6 commits August 28, 2026 11:37
@eunomie
codegen: render core and per-module client modes
fc0fd89 
The generator learns what it is rendering. In core mode — the default, and what
mod.dang still drives — the introspection is narrowed to the symbols no module
owns before the schema is built, and the output is today's gen.py: every core
type, Client and dag. On a schema with no owned symbols the bytes are
unchanged; on the module-facing schema of a real beta.11 module they are
identical to the previous generator's.

In client mode only what one module owns is rendered: its types, with all their
fields; a module-level entry point for its Query field, taking the module's
constructor arguments plus a keyword-only client that defaults to the global
one, and starting the chain with the module's binding attached; and a function
for every other field it contributes to a core type, always named by its parent
and the field, so adding a field to another parent never renames an exported
one. Every reference to a type the file does not define goes through one alias
of the core package, including the concrete client class of an interface return
and enum defaults, so a client's bytes depend on the bound module alone. The
entry point is named after the module the way its file is, e2e for module e2e,
and its root type is read off the schema rather than derived from the name.

The CLI grows the flags for it, a client-name subcommand so Dang and Python
agree on the file name, and a cli-version subcommand that turns
Query.version's v1.0.0-beta.11+a4e1e4ff into the bare 1.0.0-beta.11 the
provisioning downloader builds release URLs from. A bound module declaring an
engine version whose numeric core is below 1.0.0 is refused: it would render
a legacy core view a modern core does not have.

Signed-off-by: Yves Brissaud <yves@dagger.io>
@eunomie
python-sdk: generate a client for every dependency and for the module…
c705db8 
… itself

A dagger-module.toml module no longer gets one flat gen.py holding core and
every dependency merged. It gets the core API from its own module-facing
schema, one dagger/clients/<dep>.py per declared dependency under the
dependency's final name, and dagger/clients/<self>.py: a client for itself,
which is how it calls itself through the engine and gets function-level
caching on its own calls.

The self client needs the module's own client-facing schema, and reading that
builds the module, which needs the vendored sdk/ this generation produces. The
circle is broken by staging: pass one vendors the library, the core and the
dependency clients plus the committed self client — or a stub whose every name
is a placeholder that says the client is not generated yet — onto a scratch
workspace, the engine builds the module from that, and the self client is
generated from the schema it reports.

Local dependencies are staged first, by ownership. Those registered to this
SDK are generated here, recursively, and overlaid onto the staging workspace,
because the engine's own generateLocalDependencies runs each dependency's
generator through Workspace.generators, which is empty in a value workspace
and fails there; the recursion carries the active path and fails on a cycle.
A dependency registered to another SDK is handed to the engine, the only thing
that knows its generator. Anything else local is assumed committed, as the
engine assumes too. The staging is an input to resolving the module source,
never the diff baseline, so a dependency's generated files do not ride along.

The generated-clients subtree is exclusively generated and swept before the
merge, so a dropped dependency or a renamed alias is reported as a removal;
the rest of sdk/ keeps merging onto what is there. Legacy dagger.json modules
still go through the engine's generated context, which works around
dagger/dagger#13947.

Verified against a live beta.11 engine: a module depending on a local module
twice, once aliased, generates from nothing; its three clients bind the right
names and the same path; and greet-via-dep, greet-via-alias and greet-self all
answer through the generated clients at runtime.

Signed-off-by: Yves Brissaud <yves@dagger.io>
@eunomie
python-sdk: standalone clients
1828244 
A client is the same artifact a module vendors for a dependency, generated
for a Python project that is not a module: generateClient(ws, module, path)
writes it, initClient seeds the project file the engine's client init
dispatches for, and generateAll — the one @generate hook — generates the
registered clients after the managed modules, on the modules' generated
state, so a client bound to a local module is never read off an ungenerated
one. One hook rather than two, because the engine runs a workspace's
generators concurrently; and only modules at or below the cwd, because a
client initialized inside a module runs the hook scoped to the client's own
directory, where the enclosing module's changes could not be applied anyway.

The client vendors the library with engine provisioning and with
_engine/_version.py stamped from the engine it was generated against, as the
bare release tag the downloader builds URLs from; the core API in the client
view, which hides nothing; and dagger/clients/<module>.py, byte-identical to
what a module depending on that module gets. Its bound module's identity is
read off the resolved source: a workspace module binds by workspace-root path,
a git module by ref and pin.

The code generator and the vendored library move to a Codegen type both
module and client generation call, along with the one way generated files are
written into a workspace: the whole directory composed and written at once.
Workspace.withDirectory merges onto an untouched path but replaces it once a
withoutDirectory below it has run, so writing only the generated tree after
sweeping the clients subtree would drop everything else at the path — the
module's own sources, on the second generate of any module with a committed
client.

Verified against a live beta.11 engine: dagger api client init registers,
seeds and generates a client whose script opens its own session, serves the
module by workspace path and calls it; the standalone client file is
byte-identical to the module's own; a second generate is empty.

Signed-off-by: Yves Brissaud <yves@dagger.io>
@eunomie
e2e: dependency clients, self clients and standalone clients
cb220e3 
Two modern fixtures, dep and app: app declares dep twice, once aliased greeter,
and imports its own client from the start, so the bootstrap stub is what
carries its first build. They run on the engine's python runtime — a Go module
cannot be loaded through Workspace.moduleSource, which is also why the runtime
fixture was never registered for generation — and what they exercise is the
vendored library, the same under either runtime. A third fixture depends on
this repository's Go runtime module to pin that a dependency another SDK
manages is handed to the engine rather than skipped.

Five checks. Generation from nothing: the core without Host, one client per
dependency binding the right name and the same path, the self client, nothing
of the dependency's riding along; a second generate empty; a function added
afterwards in the regenerated self client; a dropped alias reported as a
removal. The generated clients at runtime under a real CLI: greet-via-dep,
greet-via-alias and greet-self answer — including the self call the Java
series could not test. A standalone client byte-identical to the vendored one,
with Host, provisioning, the engine's version stamped, a git binding with its
pin, initClient seeding only the project file, and the rollup empty once
applied. A plain Python process serving a git-bound module into its session
and calling it: under nesting the only session a process can reach is the
check's own, whose workspace has no generated fixture, so the git binding is
what runs end to end here; the local one was verified by hand on the host.

The runtime's generated-files check now also requires the clients package
marker, with the same actionable error, and the runtime fixture's vendored
sdk/ is regenerated to carry it. tests/client joins the library's test check.

Signed-off-by: Yves Brissaud <yves@dagger.io>
@eunomie
docs: clients, entry points and the migration recipe
c4c7ad7 
The README says what generation now produces, how module code calls a
dependency and itself through a client, how a standalone client is
registered, generated and used, and the two-line migration every
dagger-module.toml module needs on its next generate. The design doc records
what implementation found: the fixtures' runtime, the withoutDirectory
overlay quirk, and the shape of the standalone runtime check.

Signed-off-by: Yves Brissaud <yves@dagger.io>
@eunomie
future: archive the clients design
7b2efb9 
CI is green on the draft pull request, so the design moves to future/done
with its ledger closed: what shipped, what was verified where, and what is
left for follow-ups.

Signed-off-by: Yves Brissaud <yves@dagger.io>
@eunomie
eunomie force-pushed the python-unified-clients-lead-afe256c1 branch from 3fb084f to 7b2efb9 Compare August 28, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@eunomie