Skip to content

[Python] Align Native event writes and historical row updates with Java - #10668

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-io-parity-audit
Oct 20, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-io-parity-audit

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Depends on apache/paimon-rust#1180.

What changes

Align existing PyPaimon write/update behavior with current Java and route the supported cases through Native core.

  • Apply Java append row-kind validation and ignore filters before routing or opening Blob inputs. Preserve generated event and routing fields for Row/Pandas partial write types without converting unwritten payloads.
  • Enable Native historical row-ID, predicate, shard, DELETE and matched-only MERGE operations. Reuse resolved-schema/captured-snapshot transport, retain empty views and authorization sessions, and normalize the effective authorization boolean.
  • Fix Python MERGE to use the selected snapshot instead of latest, normalize captured selectors using the existing snapshot-copy helper, and keep the same conflict baseline for predicate and partition deletes.
  • Resolve commit-time indexed update columns through each prepared file's schema/field IDs, including renamed/readded columns and complete-row metadata.
  • Add end-to-end write, historical schema, conflict, deletion-vector, empty-view, authorization, changelog and index-policy regressions for both Python and Native modes.

No new public API, dependency, version probing or CI workaround. Historical upserts and MERGE with INSERT clauses retain their existing fallback; ordinary Native historical write/commit protection remains in place.

Verification

  • Focused write/schema/index/commit regression group: 273 passed, 139 skipped, plus 16 subtests.
  • Historical/update/MERGE/chunked regression group: 211 passed, 36 skipped, plus 30 subtests.
  • Authorization regression file: 112 passed, 110 skipped.
  • Flake8, Python 3.6 syntax for all changed Python files, diff checks.

Native Python integration cases were not run against the stale installed extension. No extension was rebuilt or installed; the companion Rust Core/binding test executables passed. Native CI continues to build Rust main, so this PR stays draft until the Rust dependency lands.

Review follow-up

  • Keep distinct MERGE source/target read views separate. A historical target with a newer same-table source now uses the source values, and an empty captured source cannot invoke callbacks or stage files. Preserve same-view self-MERGE optimization.
  • Convert selected Pandas inputs before composing routing/event transport, preserving missing-column errors and named-index values without converting unwritten Blob objects.
  • Share schema-aware batch projection with the dedicated writer, retaining field/schema metadata and zero-column row counts using APIs available in Arrow 6.
  • Remove duplicate schema replacement for ordinary REST authorization tables and retain historical resolution.

Follow-up verification: historical/update/MERGE/chunked group 217 passed, 42 skipped, 30 subtests passed; focused writer group 75 passed, 53 skipped; legacy data-evolution/schema/write group 93 passed, 1 skipped, 6 subtests passed. Flake8, Python 3.6 syntax for all 20 changed Python files and diff checks passed. Independent review of the fixes found no remaining actionable regression.

Native Python cases remain deferred to CI after Rust #1180 lands; no extension was rebuilt or installed locally.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi marked this pull request as ready for review October 20, 2026 03:06
@JingsongLi
JingsongLi force-pushed the codex/native-io-parity-audit branch from 9323e5a to 0ff4c84 Compare October 20, 2026 03:09
@JingsongLi
JingsongLi merged commit fbca5dc into apache:master Oct 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants