Skip to content

Support append row kinds and historical row updates - #1180

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-io-parity-audit
Oct 20, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-io-parity-audit

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

What changes

Close verified gaps in the existing PyPaimon Native write/update path, using current Java behavior.

  • Support rowkind.field on append and data-evolution tables, including CHAR/VARCHAR fields. Check required values before filtering; reject surviving DELETE/UPDATE_BEFORE events; honor ignore-delete and ignore-update-before before routing or materializing Blob inputs. Keep append files free of KV system fields.
  • Accept rowkind.field with INPUT changelogs for primary-key tables and preserve the exact generated events. Resolve the event column once per batch and preserve its identity through reordered write types.
  • Allow row-ID, predicate, shard, DELETE and matched-only MERGE operations over historical read views. Partial files retain their historical schema/field IDs and messages retain the captured snapshot as the conflict baseline. Captured empty views stay empty.
  • Match updated columns to global-index metadata by field identity from each file's schema. Handle renamed columns, dropped/readded names, extra indexed fields and complete-row updates with omitted write_cols.

Ordinary write/commit entry points, upserts and MERGE clauses which can INSERT continue to reject historical handles. Publish historical update messages through a current table. No new public API or dependency.

Java references

  • AppendOnlyFileStoreTable.newWrite, TableWriteImpl.writeAndReturn, RowKindGenerator, RowKindFilter, and AppendOnlyWriter.write.
  • Spark MergeIntoPaimonDataEvolutionTable.withMatchedUpdateScanOptions, partial update write type, captured read snapshot and global-index update policy.

Verification

  • Reproduced append acceptance/schema guards and renamed index-policy failures before their fixes.
  • Rust 1.98.0 Core library: 4320 passed, 7 ignored.
  • Append, historical update, filtered input and retained-delete integrations: 17 passed.
  • Python binding library (embedded Python; no extension installation): 104 passed.
  • Rust 1.98.0 Core and Python binding Clippy, all targets, -D warnings.
  • cargo fmt --all -- --check and diff checks.

Companion Python PR: apache/paimon#10668 (draft). Its Native CI continues building Rust main and depends on this PR.

Review follow-up

  • Include read metadata when validating MERGE table-source columns, so independent source snapshots can join on _ROW_ID without borrowing the target view. Added committed-result and empty-source regressions.
  • Replace obsolete historical-updater rejection assertions with real partial-update/delete commits, while retaining ordinary writer/commit guards.

Follow-up verification: 64 related Rust integration tests passed; Core/Python binding all-target Clippy on Rust 1.98.0, fmt and diff checks passed. Independent review found no remaining actionable regression in the reviewed changes.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi force-pushed the codex/native-io-parity-audit branch from e44ef2e to ba5bf75 Compare October 20, 2026 03:01
@JingsongLi
JingsongLi merged commit b23bde4 into apache:main Oct 20, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants