Repository navigation
feat: share the native engine across language bindings - #259
Open
Ayman Bagabas (aymanbagabas) wants to merge 12 commits into
Open
Ayman Bagabas (aymanbagabas) wants to merge 12 commits into
Ayman Bagabas (aymanbagabas) wants to merge 12 commits into
Conversation
Promote the Go C adapter to tui-test-ffi and generate native declarations from its Rust API. Replace PyO3 with ctypes and load the shared library dynamically from N-API. Update package assembly, release builds, examples, and ownership coverage. Preserve the supported runtimes and typed binary screen buffers. Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
Signed-off-by: Ayman Bagabas <ayman.bagabas@gmail.com>
Ayman Bagabas (aymanbagabas)
marked this pull request as ready for review
October 13, 2026 16:28
Ayman Bagabas (aymanbagabas)
requested review from
cpendery (cpendery) and
Dan Fiedler (danfiedler-msft)
and
a balanced review from Copilot
October 13, 2026 16:28
Copilot started reviewing on behalf of
Ayman Bagabas (aymanbagabas)
October 13, 2026 16:28
View session
There was a problem hiding this comment.
🔵 Needs a closer look
The broad cross-language FFI, ownership, concurrency, and release-packaging changes warrant final human review despite successful automated validation.
0 open findings
What changed in this PR
Unifies Go, Python, and JavaScript bindings around the shared tui-test-ffi C ABI while preserving their public APIs.
Changes:
- Adds generated ABI layouts, conversions, declarations, and asynchronous task entrypoints.
- Replaces PyO3 and N-API adapters with ctypes and Koffi.
- Updates cross-platform builds, packaging, documentation, and regression coverage.
| File | Description |
|---|---|
.github/scripts/release/package-node.mjs |
Packages shared libraries by platform. |
.github/scripts/release/package-node.test.mjs |
Tests Node package assembly. |
.github/scripts/release/prepare-python-linux.sh |
Prepares Rust/Zig wheel builds. |
.github/workflows/ci.yml |
Validates all shared bindings. |
.github/workflows/release.yml |
Builds shared-library packages. |
.gitignore |
Ignores generated native artifacts. |
Cargo.lock |
Updates Rust dependencies. |
Cargo.toml |
Adds FFI and bindgen workspace members. |
README.md |
Documents the shared adapter. |
bindings/go/CONTRIBUTING.md |
Updates Go development instructions. |
bindings/go/README.md |
Documents the shared library setup. |
bindings/go/integration_test.go |
Expands screenshot regression coverage. |
bindings/go/internal/native/abi.go |
Generates Go ABI layouts. |
bindings/go/internal/native/functions.go |
Generates Go function registrations. |
bindings/go/internal/native/loader.go |
Loads the shared FFI library. |
bindings/go/internal/native/loader_test.go |
Updates loader tests. |
bindings/go/internal/native/session.go |
Uses ABI version and screenshot pointer API. |
bindings/go/native/src/input.rs |
Removes the Go-specific Rust adapter. |
bindings/js/.gitignore |
Ignores shared-library builds. |
bindings/js/Cargo.toml |
Removes the N-API crate. |
bindings/js/README.md |
Documents Koffi and packaging. |
bindings/js/build.rs |
Removes N-API build setup. |
bindings/js/native/abi.js |
Generates Koffi ABI declarations. |
bindings/js/native/adapter.js |
Implements Koffi transport and ownership. |
bindings/js/native/index.d.ts |
Generates native TypeScript declarations. |
bindings/js/native/index.js |
Loads and validates the shared library. |
bindings/js/native/lib.rs |
Removes the N-API implementation. |
bindings/js/native/models.js |
Generates model metadata. |
bindings/js/package-lock.json |
Updates JavaScript dependencies. |
bindings/js/package.json |
Replaces N-API tooling with Koffi. |
bindings/js/scripts/build-native.mjs |
Builds and copies shared libraries. |
bindings/js/src/native.ts |
Updates native-library error handling. |
bindings/js/test/integration.test.mjs |
Expands runtime compatibility tests. |
bindings/js/test/native.test.mjs |
Tests ownership, concurrency, and loading. |
bindings/js/test/options.test.mjs |
Updates native conversion terminology. |
bindings/python/.gitignore |
Ignores packaged shared libraries. |
bindings/python/MANIFEST.in |
Defines source-distribution contents. |
bindings/python/README.md |
Documents ctypes builds and behavior. |
bindings/python/examples/shared_library.py |
Adds a shared-library example. |
bindings/python/native/Cargo.toml |
Removes the PyO3 crate manifest. |
bindings/python/native/src/lib.rs |
Removes the PyO3 implementation. |
bindings/python/pyproject.toml |
Switches to setuptools-rust. |
bindings/python/scripts/generate_stubs.py |
Removes the old stub generator. |
bindings/python/setup.py |
Builds and packages the shared library. |
bindings/python/src/tui_test/_abi.py |
Generates ctypes ABI declarations. |
bindings/python/src/tui_test/_native.py |
Implements the ctypes adapter. |
bindings/python/src/tui_test/_native.pyi |
Updates private adapter types. |
bindings/python/src/tui_test/client.py |
Removes the obsolete panic probe. |
bindings/python/src/tui_test/py.typed |
Marks the package as typed. |
bindings/python/stub-gen/.gitignore |
Removes obsolete stub-generator config. |
bindings/python/stub-gen/Cargo.lock |
Removes obsolete stub dependencies. |
bindings/python/stub-gen/Cargo.toml |
Removes the stub-generator crate. |
bindings/python/stub-gen/src/main.rs |
Removes PyO3 stub generation. |
bindings/python/tests/test_ctypes_adapter.py |
Tests ctypes ownership and workers. |
bindings/python/tests/test_integration.py |
Updates native error regression testing. |
crates/tui-test-ffi/Cargo.toml |
Renames the shared FFI crate. |
crates/tui-test-ffi/README.md |
Documents ABI generation and ownership. |
crates/tui-test-ffi/include/tui_test.h |
Generates the public C declarations. |
crates/tui-test-ffi/src/input.rs |
Implements shared input conversion. |
crates/tui-test-ffi/src/input_generated.rs |
Generates model input conversions. |
crates/tui-test-ffi/src/lib.rs |
Exposes ABI version 4 and shared entrypoints. |
crates/tui-test-ffi/src/output.rs |
Uses generated output conversions. |
crates/tui-test-ffi/src/output_generated.rs |
Generates output model conversion. |
crates/tui-test-ffi/src/session.rs |
Centralizes session behavior. |
crates/tui-test-ffi/src/session_generated.rs |
Generates session context validation. |
crates/tui-test-ffi/src/tasks.rs |
Implements asynchronous native tasks. |
crates/tui-test-ffi/src/tasks_generated.rs |
Generates typed task entrypoints. |
crates/tui-test-ffi/src/tests.rs |
Adds FFI ownership and validation tests. |
crates/tui-test-ffi/src/type_helpers.rs |
Defines foundational ABI types. |
crates/tui-test-ffi/src/types.rs |
Generates complete ABI structures. |
tools/bindgen/Cargo.toml |
Adds generator dependencies. |
tools/bindgen/src/boundaries.rs |
Generates model boundaries and conversions. |
tools/bindgen/src/javascript.rs |
Generates JavaScript models and types. |
tools/bindgen/src/koffi.rs |
Generates Koffi layouts and task wrappers. |
tools/bindgen/src/main.rs |
Orchestrates cross-language generation. |
Files not reviewed (1)
- bindings/js/package-lock.json: Generated file
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tui-test-ffilibrary and typed C ABI for Go, Python, and JavaScript.All three bindings accept
TUI_TEST_NATIVE_LIBRARY.Python wheels and JavaScript platform packages include the matching shared library.
Koffi supplies its own native addon; the package is not native-free.
Related work
Related to #255 (
test: go binding automation), which adds bundled Go engines, extraction into a local cache, release automation, and C-compiler ABI layout checks.This PR shares the native engine across languages and generates their ABI declarations.
This work may supersede the Go-only header generation and screenshot ABI fix in #255.
It does not yet replace the bundled Go distribution, cache extraction, or C-compiler layout checks.
Those parts still need integration or a follow-up before this PR can fully supersede #255.
The changes overlap in the Go loader, ABI declarations, and CI/release workflows.
Integration must use
tui-test-ffi, its shared-library artifact names,TUI_TEST_NATIVE_LIBRARY, and the generated C header.The layout checks from #255 should use that header rather than a separate Go-only header generator.
This PR advances the shared contract from ABI version 3 to 4.
The branches' binaries are not interchangeable: #255 changes
tui_screenshotin place, while this PR preserves it and addstui_screenshot_ptr.Integration must select one final ABI; matching version numbers alone do not make these branch builds compatible.
Authoritative definitions
tools/bindgenreads the existing Rust API, locator, diagnostics, profile, shell, backend, and terminal models.The C signatures remain authoritative for C entrypoints.
The generator produces the C header, Go declarations, Python declarations, and JavaScript ABI layouts, model metadata, and TypeScript declarations.
It also generates typed asynchronous wrappers from the C signatures.
There is only one generated Rust boundary
types.rs.JavaScript uses generated
native/abi.js,native/models.js, andnative/index.d.ts.Handwritten JavaScript handles library selection, promises, validation, transport mappings, and native ownership.
Public Go, Python, and TypeScript convenience APIs remain handwritten.
The check covers generated output.
Regression tests cover source-field, enum, operation, and result propagation, transport mappings, and stale output.
Code size
The full PR at
25254f1bf189has 8,567 additions and 9,601 deletions: a net reduction of 1,034 lines.The Koffi step (
1ed511273814to8b837033dcb5) removes 2,431 lines overall, including generated code and lockfiles.The counts below cover the shared FFI, generator, and native JavaScript sources.
They include Rust tests but exclude manifests, lockfiles, documentation, and JavaScript tests.
The generator counts as handwritten code.
Concurrency and ownership
Direct blocking waits exhausted Koffi's worker pool in a local reproduction.
Thirty-two pending waits delayed an input call by over four seconds.
Typed native start functions copy their inputs before return and execute owned operations independently.
Pending waits cannot prevent input or close calls.
The synchronous C entrypoints remain available to Go and Python.
Immediate getters use Koffi's asynchronous calls without a polling timer.
A bounded submission queue keeps large read batches within Koffi's limits.
Reads remain responsive while long waits are pending.
The adapter copies output strings, arrays, and bytes before
tui_result_free.It releases results and tasks after errors as well as successful calls.
Explicit input retention keeps nested buffers alive through native marshaling.
The function table retains the Koffi library handle, which also avoids a reproduced Bun finalizer crash.
Windows pins the DLL until process exit so Rust thread-local cleanup cannot call unloaded code after a JavaScript environment exits.
Freeing a pending native task abandons its result; it does not cancel the operation.
Packaging and compatibility
The seven existing release targets remain supported.
The build uses Cargo or cargo-zigbuild, not the N-API CLI.
Release assembly generates platform manifests from the shared target map.
It verifies each shared-library artifact and sets matching package versions.
Optional dependencies remain enabled because Koffi also uses them for its platform binaries.
The library and language package must come from the same source revision.
Go users must replace
TUI_TEST_GO_NATIVE_LIBRARYwithTUI_TEST_NATIVE_LIBRARYand rebuild the renamed library; the old variable has no fallback.The shared contract uses ABI version 4 for the pointer and asynchronous entrypoints.
Existing synchronous C signatures and layouts remain compatible with their version 3 forms.
Python wheels use
py3-none-<platform>tags instead ofabi3tags.JavaScript remains ESM with Node 20+ support; Bun and Deno remain best effort.
Earlier CI repairs
Go screenshot options now use a generated pointer entrypoint.
This fixes the reproduced AMD64 PureGo by-value argument corruption.
Go generation selects pointer entrypoints when both forms exist.
The raster fixture uses bundled-font characters after its Unicode text and SVG checks.
Validation
Local validation covers Rust formatting, strict Clippy, generator drift, 12 generator tests, 14 FFI tests, all Go packages, and 164 Python tests.
The final JavaScript suite passes 102 tests under Node 20.
Bun and Deno each pass 101 tests with one existing skip.
Linux AMD64 also passes its suite; the short-read optimization has separate concurrency and worker-exit regressions.
Linux musl passes a typed-call and Unicode smoke check.
The packed Node package works without the CLI.
The documented JavaScript example prints
Terminal size: 80x30andReady 界🙂.The short-call optimization removes the original one-millisecond getter delay.
A local alternating 200-call sample measured
getSize()at a 23-microsecond median, versus 38 microseconds through the previous addon.These local artifact measurements are a smoke comparison, not a portable performance guarantee.
The first Koffi CI run passed Linux, macOS, and all runtime compatibility jobs.
Windows exposed an input fixture that sent LF instead of terminal Enter.
CodeQL also reported nullable-pointer flows through synchronous tests and the pending-result match.
Commit
9a4addcbf3c0uses the Enter-aware submit operation, asserts synchronous result pointers, and separates pending and completed result branches.That run fixed the input fixture but exposed a Windows process-teardown access violation.
Commit
27abbff4b279replaces raw result dereferences with checked pointer conversions.Commit
2e222b271520retains the Windows DLL for process lifetime and adds a worker-only teardown regression.Local native and integration tests pass all 54 cases; the FFI tests and strict Clippy also pass.
All checks passed at
2e222b271520, including Windows and CodeQL: https://github.com/microsoft/tui-test/actions/runs/38425111081Commit
25254f1bf189advances the ABI version to 4 and regenerates all language checks and the C header.Local validation passes the generator drift check, 14 FFI tests, 22 JavaScript native tests, 10 Python adapter tests, and Go native tests and smoke example.
All checks pass at
25254f1bf189, including Windows, Linux, macOS, runtime compatibility, and CodeQL.The successful ABI version 4 CI run is https://github.com/microsoft/tui-test/actions/runs/38428880038.