feat(workflow-compiling-service): export a workflow as a standalone Python script by kz930 · Pull Request #8327 · apache/texera

kz930 · 2026-09-01T23:37:46Z

What changes were proposed in this PR?

A workflow can be built and read in the editor, but there is no form of it that runs anywhere else. This adds one: given a plan, the compiling service returns a single Python file that reads the same sources, applies the same operators in the same order, and prints its results.

An operator says how it reads outside the engine by implementing StandaloneCodeGenerator, returning a block of pandas that names its inputs and outputs by position — in1df, in2df, out1df. The translator walks the plan in topological order, gives every output port a variable, substitutes those placeholders for the variables its upstreams were given, and prints the leaves. Union takes the whole list of upstreams rather than a fixed count, since any count an operator states would be wrong for some workflow.

An operator that has no generator yet leaves a commented placeholder rather than a line that looks like it works, so the export is useful before every operator implements the trait.

Five operators implement it here to show the shape and to give the translator something real to walk: Distinct, Filter, Limit, Projection and Union. The rest of the operator set follows a family at a time.

Any related issues, documentation, discussions?

Part of #8325, 1 of 20; that issue lists the set in order.

How was this PR tested?

WorkflowToPythonTranslatorSpec covers what the translator does with a plan: the topological order, the variable each port is given, the placeholder substitution, the variadic port, and the operator that has no generator. Each of the five operators asserts the block it emits in its own spec.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

…ython script A workflow can be read in the editor but not taken away: there is no form of it that runs anywhere else, so a user who wants to keep a pipeline, hand it to someone without Texera, or step through it in a notebook has nothing to take. This adds the seam for one and the first few operators through it. An operator says how it reads outside the engine by implementing `StandaloneCodeGenerator`, returning a block of pandas that names its inputs and outputs as `in1df` / `out1df`. The translator walks the plan in topological order, gives every port a variable, substitutes those placeholders, and prints the leaves; `inAlldf` stands for the whole list of upstreams, which is what a variadic port like Union's needs, since any fixed count the code stated would be wrong for some workflow. An operator with no generator yet leaves a commented TODO rather than a line that looks like it works. `GET /workflow-to-python` on the compiling service returns the script for a plan it is given. Five operators implement it here — Distinct, Limit, Projection, Filter and Union — chosen to cover the shapes the translator has to handle: a single input, a config-driven one, one that renames columns, one that builds a predicate, and the variadic port. The rest of the operator set follows in later changes. `pyStringLiteral` renders a value as a Python literal with the escaping that keeps a quote or a newline in a column name from ending the literal early. The generators cannot use the runtime's decode expression, which needs an operator instance to decode through. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…re the files The standalone export claims that a generated script does what the operator does. Nothing checks it. This adds the two runners that make the claim checkable, and the file format they meet in. `OpExecHarness` runs a LogicalOp the way the engine does — compiling it to a physical plan and driving the executor — but outside a workflow, against JSONL files rather than a live upstream. `PyOpExecHarness` does the same for a Python operator, through the worker the engine uses. `StandaloneRunner` takes the other path: it asks the operator for its standalone code, wraps it in a script that binds `in1df` from the same files, and runs it. `TupleIO` is what the two meet in. A JSONL row carries values and no types, so the schema travels beside it in a sidecar; without one, a column written as INTEGER reads back as a number and the two paths disagree over a difference neither operator made. Both runners produce files, not assertions, so what to make of a difference is left to a later change. What is here is enough to run one operator both ways and see that the answers match, which is what the spec does with Distinct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-01T23:38:13Z

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

Contributors with relevant context: @aglinxinyuan, @Yicong-Huang, @kunwp1
You can notify them by mentioning @aglinxinyuan, @Yicong-Huang, @kunwp1 in a comment.

codecov-commenter · 2026-09-01T23:40:41Z

Codecov Report

❌ Patch coverage is 72.04301% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.01%. Comparing base (cd4fd5a) to head (6c49e46).

Files with missing lines	Patch %	Lines
...ra/service/resource/WorkflowToPythonResource.scala	0.00%	9 Missing ⚠️
.../amber/translator/WorkflowToPythonTranslator.scala	85.71%	1 Missing and 5 partials ⚠️
...exera/amber/operator/StandaloneCodeGenerator.scala	20.00%	4 Missing ⚠️
...a/amber/operator/projection/ProjectionOpDesc.scala	71.42%	2 Missing and 2 partials ⚠️
...mber/operator/filter/SpecializedFilterOpDesc.scala	71.42%	1 Missing and 1 partial ⚠️
...a/org/apache/texera/amber/operator/LogicalOp.scala	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8327      +/-   ##
============================================
- Coverage     94.03%   94.01%   -0.03%     
- Complexity     4771     4783      +12     
============================================
  Files          1196     1199       +3     
  Lines         48757    48845      +88     
  Branches       5916     5927      +11     
============================================
+ Hits          45850    45920      +70     
- Misses         1458     1468      +10     
- Partials       1449     1457       +8

Flag	Coverage Δ		*Carryforward flag
access-control-service	`81.00% <ø> (ø)`
agent-service	`99.32% <ø> (ø)`		Carriedforward from 16da22b
amber	`89.84% <73.17%> (-0.06%)`	⬇️
computing-unit-managing-service	`73.67% <ø> (ø)`
config-service	`87.12% <ø> (ø)`
file-service	`87.91% <ø> (ø)`
frontend	`96.85% <ø> (+0.04%)`	⬆️	Carriedforward from 16da22b
notebook-migration-service	`79.31% <ø> (ø)`
pyamber	`98.47% <ø> (ø)`		Carriedforward from 16da22b
workflow-compiling-service	`74.31% <71.15%> (-2.89%)`	⬇️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-09-01T23:44:08Z

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 5 worse · ⚪ 10 noise (<±5%) · 0 without baseline

Compared against main cd4fd5a benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	397	0.243	25,127/36,859/36,859 us	🔴 +19.1% / 🔴 +130.7%
🔴	bs=100 sw=10 sl=64	812	0.496	123,311/140,010/140,010 us	🔴 +5.7% / 🔴 +28.1%
⚪	bs=1000 sw=10 sl=64	912	0.557	1,089,812/1,192,168/1,192,168 us	⚪ within ±5% / 🔴 +14.3%

Baseline details

Latest main cd4fd5a from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	397 tuples/sec	439 tuples/sec	755.36 tuples/sec	-9.6%	-47.4%
bs=10 sw=10 sl=64	MB/s	0.243 MB/s	0.268 MB/s	0.461 MB/s	-9.3%	-47.3%
bs=10 sw=10 sl=64	p50	25,127 us	21,101 us	12,938 us	+19.1%	+94.2%
bs=10 sw=10 sl=64	p95	36,859 us	36,932 us	15,980 us	-0.2%	+130.7%
bs=10 sw=10 sl=64	p99	36,859 us	36,932 us	19,233 us	-0.2%	+91.6%
bs=100 sw=10 sl=64	throughput	812 tuples/sec	846 tuples/sec	976.3 tuples/sec	-4.0%	-16.8%
bs=100 sw=10 sl=64	MB/s	0.496 MB/s	0.516 MB/s	0.596 MB/s	-3.9%	-16.8%
bs=100 sw=10 sl=64	p50	123,311 us	118,228 us	102,340 us	+4.3%	+20.5%
bs=100 sw=10 sl=64	p95	140,010 us	132,489 us	109,262 us	+5.7%	+28.1%
bs=100 sw=10 sl=64	p99	140,010 us	132,489 us	118,827 us	+5.7%	+17.8%
bs=1000 sw=10 sl=64	throughput	912 tuples/sec	919 tuples/sec	1,006 tuples/sec	-0.8%	-9.4%
bs=1000 sw=10 sl=64	MB/s	0.557 MB/s	0.561 MB/s	0.614 MB/s	-0.7%	-9.3%
bs=1000 sw=10 sl=64	p50	1,089,812 us	1,085,494 us	999,855 us	+0.4%	+9.0%
bs=1000 sw=10 sl=64	p95	1,192,168 us	1,137,243 us	1,042,833 us	+4.8%	+14.3%
bs=1000 sw=10 sl=64	p99	1,192,168 us	1,137,243 us	1,070,722 us	+4.8%	+11.3%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,503.24,200,128000,397,0.243,25126.63,36859.03,36859.03
1,100,10,64,20,2463.35,2000,1280000,812,0.496,123310.77,140010.42,140010.42
2,1000,10,64,20,21928.97,20000,12800000,912,0.557,1089811.84,1192168.20,1192168.20

…he operator it came from The standalone export claims a generated script does what the operator does. This is what checks it, for every operator, on every configuration the operator offers. An operator is run twice. `OpExecHarness` drives it the way the engine does, compiled to a physical plan but outside a workflow, reading JSONL files rather than a live upstream; `PyOpExecHarness` does the same for a Python operator through the worker the engine uses. `StandaloneRunner` takes the other path, wrapping the operator's standalone code in a script that binds the same files. Both write files, and `Comparator` reads them back: order-insensitive by default, since the engine interleaves across workers and only the sort family promises an order. A visualization is compared as a figure rather than as a frame. What to run an operator ON is decided rather than written by hand for each. `ConfigGenerator` reads the operator's own schema — its enums, defaults, declared ranges and column pickers — and produces a base configuration plus one variant per branch the operator offers, so a switch nobody thought to try is still tried. `CanonicalFixture` is the table they run against, one column per shape an operator might ask for. `CuratedHandlers` is the escape hatch for an operator whose input cannot be derived, and `TransformVerificationRunner` decides which of the three tiers each operator takes and reports what it could not run and why. `LogicalOp.orderSensitive` and `@SampleColumn` are the two things the operators had to say for this to read them: whether row order is part of the contract, and which column a field should be pointed at when the first unused one would be a poor choice. Most of the operator set does not implement the generator yet — it arrives a family at a time — and the runner reports each of those rather than passing over it. The tier assertions for a family land with the change that gives that family its generator. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…f the one without The split this change relies on was declared but never wired. The specs carry `@IntegrationTest` and `build.sbt` reads `WCS_TEST_FILTER` to act on it, but nothing set that variable, so the filter was a no-op and the specs that fork Python ran in the job that provisions none — failing on `No module named 'pandas'` rather than on anything they were testing. The platform job now sets `skip-integration`, which excludes them. The platform-integration job sets `integration-only` and provisions what they need: Python 3.12, amber's requirements, protoc, and the generated proto bindings, which are gitignored and so have to be regenerated before a forked driver can import pyamber. Every step is guarded on the service, so the other entries in that matrix are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The comment claimed a knob carrying a `defaultValue` is left alone. It is not: a knob holding what a fresh instance holds counts as untouched, because a config where the user typed the default is indistinguishable from one where they typed nothing. Moving it off that value is the only way the branch for a knob that IS set gets exercised. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… have one The example was an operator another batch gives a generator to, so the assertion held only until that batch landed. A Python UDF holds whatever order these land in: its body is written by whoever drops the operator, so there is nothing for a generator to emit. The word cloud assertion goes for the same reason. It says the operator is withheld, which a later batch stops being true once its placement is seeded, and the prediction op alone already covers what the test is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

One row is withheld because the fixture cannot carry what the operator consumes, the other because the operator has no physical execution to run at all. Only the first was asserted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

sbt lifts a `.value` written inside a lambda to the top of the task, so the options were already read once rather than per suite. Written where it was, it read as the other thing, and sbt warned on every run. Hoisting it says what actually happens and quiets the warning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The licence check compares the file against what pip resolved, and the two had drifted apart, which blocks every run on Python 3.12: the file claimed 4.14.1 while 4.16.0 was installed. The pin in requirements.txt still says 4.14.1 and does not hold, because operator-requirements.txt is installed in a second pass whose resolution the first file's constraint does not reach. That is worth fixing on its own; this only stops the check reporting a version nothing installs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…talled" This reverts commit f7a0243.

Everything here that is not a source operator belongs to apache#8327 and was carried only so this branch could compile and run its own tests before that one landed. Reviewing it twice costs more than the red build does: what is left is the thirteen files this change is actually about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Everything here that is not this batch's own operators belongs to apache#8327 and was carried only so the branch could compile and run its own tests before that one landed. Reviewing it twice costs more than the red build does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The rows these operators add to the runner and its config tests sit in files apache#8327 introduces, so they land once that does rather than as whole new files here. Each operator's own spec stays. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Everything here that is not this batch's own operators belongs to apache#8327 and was carried only so the branch could compile and run its own tests before that one landed. Reviewing it twice costs more than the red build does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The rows these operators add to the runner and its config tests sit in files apache#8327 introduces, so they land once that does rather than as whole new files here. Each operator's own spec stays. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Everything here that is not this batch's own operators belongs to apache#8327 and was carried only so the branch could compile and run its own tests before that one landed. Reviewing it twice costs more than the red build does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The rows these operators add to the runner and its config tests sit in files apache#8327 introduces, so they land once that does rather than as whole new files here. Each operator's own spec stays. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Ten thousand lines is more than anyone reads in one sitting. What stays is the export itself: the trait an operator implements, the translator that stitches the blocks into a script, the endpoint that serves it, and five operators to show the shape. The harness that runs an operator both ways and compares the answers follows in changes of its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A type says a column is a number; it does not say which number is the opening price. Operators carry the annotation, so it belongs with the trait they also carry rather than with the generator that reads it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kz930 · 2026-09-03T00:31:40Z

/request-review @mengw15

kz930 · 2026-09-03T00:31:58Z

@carloea2 May you take a look at this?

kz930 and others added 2 commits September 1, 2026 16:23

github-actions Bot assigned kz930 Sep 1, 2026

github-actions Bot added feature dependencies Pull requests that update a dependency file common platform Non-amber Scala service paths labels Sep 1, 2026

kz930 changed the title ~~test(workflow-compiling-service): run an operator both ways and compare the files~~ test(workflow-compiling-service): verify a generated script against the operator it came from Sep 1, 2026

kz930 changed the title ~~test(workflow-compiling-service): verify a generated script against the operator it came from~~ feat(workflow-compiling-service): export a workflow as a standalone Python script, and verify the export Sep 1, 2026

This was referenced Sep 1, 2026

feat(workflow-compiling-service): export a workflow as a standalone Python script #8326

Closed

feat(workflow-operator): export the base transform operators as Python #8340

Open

feat(workflow-operator): export the source operators as Python #8341

Open

github-actions Bot added the ci changes related to CI label Sep 2, 2026

kz930 mentioned this pull request Sep 2, 2026

feat(visualization): export the hierarchy and graph charts as Python #8344

Open

This was referenced Sep 2, 2026

feat(visualization): export the coordinate-system and domain plots as Python #8346

Open

feat(machine-learning): export the four advanced trainers #8348

Open

feat(gui): export a workflow as Python, and read the schema's constraints in the panel #8350

Open

kz930 and others added 4 commits September 2, 2026 14:05

kz930 mentioned this pull request Sep 2, 2026

ci: record the typing-extensions version that is actually installed #8355

Closed

Revert "ci: record the typing-extensions version that is actually ins…

b91fe73

…talled" This reverts commit f7a0243.

kz930 and others added 2 commits September 2, 2026 16:28

Merge upstream/main

e5fa931

github-actions Bot removed the ci changes related to CI label Sep 2, 2026

kz930 changed the title ~~feat(workflow-compiling-service): export a workflow as a standalone Python script, and verify the export~~ feat(workflow-compiling-service): export a workflow as a standalone Python script Sep 2, 2026

github-actions Bot requested a review from mengw15 September 3, 2026 00:31

Merge branch 'main' into feat/standalone-verify-harness

6c49e46

Navigation Menu

Uh oh!

feat(workflow-compiling-service): export a workflow as a standalone Python script - #8327

feat(workflow-compiling-service): export a workflow as a standalone Python script#8327
kz930 wants to merge 14 commits into
apache:mainfrom
kz930:feat/standalone-verify-harness

kz930 commented Sep 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 1, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Sep 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 1, 2026 •

edited

Loading

Uh oh!

kz930 commented Sep 3, 2026

Uh oh!

kz930 commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

kz930 commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this PR?

Any related issues, documentation, discussions?

How was this PR tested?

Was this PR authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Reviewer Suggestions

Uh oh!

codecov-commenter commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Benchmark changes need a look

Uh oh!

kz930 commented Sep 3, 2026

Uh oh!

kz930 commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kz930 commented Sep 1, 2026 •

edited

Loading

github-actions Bot commented Sep 1, 2026 •

edited

Loading

codecov-commenter commented Sep 1, 2026 •

edited

Loading

github-actions Bot commented Sep 1, 2026 •

edited

Loading