Skip to content

[core] Align delete and update nullability with Java - #1179

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-io-path-completion
Oct 19, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/native-io-path-completion

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Align existing Python Native write/update behavior with current Java, preserving logical NOT NULL constraints while allowing valid deduplicate DELETE records and unused update inputs.

Companion Python PR: apache/paimon#10666.

Changes

  • Derive a nullable physical KV value schema for deduplicate tables without cross-partition updates, matching PrimaryKeyFileStoreTable.store(). Preserve primary-key/sequence fields and the logical table schema.
  • Preserve physical root nullability in the Paimon fields supplied to the Parquet writer, including data and INPUT changelog files. Preserve field IDs, descriptions, defaults and nested nullability.
  • Resolve row kinds and enforce logical top-level NOT NULL checks before the optional row-kind filter, including when no ignore option is enabled. Other merge engines and cross-partition writes retain strict checks.
  • Reuse the existing write-input Arrow importer for update/upsert/MERGE sources. Preserve actual NULLs, sliced buffers and metadata until Core selects target payloads; selected target constraints remain enforced in Core.
  • Keep one common transport schema across the chunks of an Arrow Table by unioning relaxed root nullable flags without copying payload buffers. Core MERGE retains its strict schema check; heterogeneous iterable upsert inputs retain their supplied field sets.
  • Update the existing invalid-row-kind test to assert rejection at write time and no staged output.

Verification

  • Review follow-up: real PyArrow-to-Core MERGE reproduces the old mixed-chunk schema failure, then passes with NULL chunks in either order, NULL ON keys, unmatched NULL payloads, and unused dictionary values that decode to NULL. Source schema/field metadata remains intact. Rust 1.98.0 binding all-target Clippy, rustfmt and diff checks pass.

  • New retained-delete integration suite: 3 passed. Covers explicit/configured row kinds, STRING/LargeString, automatic/precomputed buckets, separate INSERT and DELETE commits, sequence ordering, physical Parquet root fields, INPUT changelogs, value predicates that must not resurrect deleted rows, managed BLOB files, and required-field/other-engine/cross-partition negatives.

  • Existing filtered-write integration suite: 4 passed.

  • Python-binding Rust library: 104 passed. New source-transport and real upsert tests first fail on the old importer, then pass, including discarded duplicate values and invalid selected target NULLs.

  • Rust core library: 4319 passed, 7 ignored. Pinned Rust 1.98.0 core/binding all-target Clippy with -D warnings, rustfmt and diff checks passed.

  • Current Java reference tests TableWriteTest.testDeduplicateDeleteWithPrimaryKeyOnly and testDeleteRequiresSequenceField: 2 passed.

  • Companion Python regressions cover ordinary/dedicated-BLOB target constraints, physical schema flags, and real commit/read results. Local Python tests did not rebuild or install the Native extension.

No new dependency, public API, version probe or commit-failure cleanup is introduced.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 9430ee8 into apache:main Oct 19, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants