[core] Negate predicates like Java so NULL rows stay excluded

[JeonDaehong]

Purpose

Linked issue: close #1143

Predicate::negate wrapped leaves in Not, and Not was evaluated as !inner, so a NULL input made NOT(p = 'a') true. Reads returned those rows and delete_by_predicate deleted them. Java flips the leaf instead (Equal.negate() is NotEqual), which is false for NULL.

Brief change log

• Predicate::negate now does what Java Predicate.negate does: flip the leaf operator (=/!=, </>=, <=/>, IS NULL/IS NOT NULL, IN/NOT IN, BETWEEN/NOT BETWEEN) and push through AND/OR with De Morgan.
• DataFusion: the NOT BETWEEN coverage check in scan.rs now matches the new shape of the negated filter (x < lo OR x > hi instead of NOT(x >= lo AND x <= hi)).
• Leaves with no negated operator here (STARTS_WITH, ENDS_WITH, CONTAINS, LIKE, array ops) stay as Not(leaf). eval_row and the residual row filter now return false for those when the input is NULL, same as Java NotStartsWith/NotLike/... .

Tests

• negated_predicates_keep_null_rows and negated_data_predicate_keeps_null_rows cover the cases from the issue on the partition and DV delete paths and on reads. Both fail on main.
• Unit tests for negate and for NULL evaluation in eval_row and the residual filter.
• Changed existing tests:
  • test_file_index_nested_not_with_added_column_falls_back now expects [1] instead of [1, 2]. added is NULL in the old file, so for id 2 the predicate is NOT(true AND unknown). Java drops that row too.
  • test_not / test_display_not, three DataFusion pushdown tests, the global index exactness test and the stats fail-open test asserted the old Not shape. The last two now build Predicate::Not directly so they keep testing the Not branch.

fmt and clippy (1.98.0, -D warnings) pass. cargo test --no-fail-fast -p paimon -p paimon-datafusion -p paimon-c: 5804 passed. The 39 failures are all TableNotExist / table not found for the fixture tables CI provisions, plus 2 that need the lumina library. The ones I checked fail the same way on main.

Not covered here: a Predicate::Not around AND/OR built by hand (not through negate) is still evaluated as !inner. Nothing in core builds that today, but I can add it if you'd like.

API and Format

No API change. Predicate::negate returns a different but equivalent tree for non-NULL inputs.

Documentation

None.

[JeonDaehong]

@JingsongLi @leaves12138 could you take a look when you have time? Fixes #1143: delete_by_predicate with a negated predicate was also deleting rows where the column is NULL.

[JeonDaehong]

Moved to draft until CI is green. I couldn't run the docker fixtures and the Go/Python suites locally. Will mark it ready once they pass.

[JeonDaehong]

CI is green, ready for review.

[JingsongLi]

Requirement fit: SUPPORTED. Implementation: FINDINGS at 0a5bdd9. Fixing NULL rows being selected/deleted by negated predicates has clear end-to-end value. The ordinary NULL and added string-negation scenarios pass, but two additional public-API regression tests fail on this head and pass on base 9568f39: floating bitmap + vector Top-1 returns a row that violates the filter, and a NaN partition is silently excluded. Details are inline. Local verification also ran the core library suite (4146 passed, 6 ignored); after provisioning the actual Spark warehouse, all 358 DataFusion library tests passed. Both new regression tests were also run unchanged against the baseline worktree and passed there.

[JingsongLi]

[P1] Refine floating bitmap candidates before ranking normalized NOT filters

Turning NOT(f = 1) into NotEq now enters the floating bitmap scalar-index path. BitmapGlobalIndexReader::query deliberately returns all non-null rows for floating NotEq, but matching_ranges_with_exactness labels this leaf exact, so scalar_matched_row_ids skips refinement before vector ranking. I reproduced this through public write/commit/index/search APIs: two rows have f=[1,2], vectors [[1,0],[0,1]], a bitmap index on f, and an ivf-flat index on the vectors. Searching [1,0] with limit 1 and Predicate::negate(equal(f,1)) returns row 0 (f=1), which violates the filter; expected row 1. The same test passes on base 9568f39a, and explicitly constructing raw Predicate::Not still returns row 1 on this head. Propagate floating bitmap candidate inexactness to the scanner and refine before ranking; add this filtered Top-K regression test.

[JeonDaehong]

Thanks, reproduced. Fixed in eff224e in two places:

• negate now keeps Not for FLOAT/DOUBLE comparisons (see the NaN comment), so NOT(f = 1) stays out of the bitmap NotEq path.
• The same wrong Top-1 also came from a plain not_equal(f, 1) or greater_than(f, 1.5), with no negation involved. is_exact_predicate now treats FLOAT/DOUBLE leaves that the bitmap answers with every non-null row (is_bitmap_floating_residual_sensitive_op) as inexact, so they are refined before ranking.

floating_bitmap_candidates_are_refined_before_top_k covers all three filters. Before the change it returned row 0, now row 1.

[JingsongLi]

[P2] Preserve NaN rows when negating floating partition comparisons

These operator flips assume complementary ordering, but native datum_cmp uses partial_cmp for FLOAT/DOUBLE, so NaN makes both < and >= false. I reproduced a scan with rows (id,f)=[(0,0.0),(1,NaN),(2,2.0)] and Predicate::negate(less_than(f,1.0)): this head returns [1,2] when f is a data column but only [2] when partitioned by f, because the normalized GtEq prunes the NaN partition. The identical test returns [1,2] in both layouts on base 9568f39a. Java's floating comparison and the Arrow residual path retain the NaN row here. Make native floating comparison compatible with those semantics, or retain NULL-aware NOT until the complements are sound, and cover partitioned/nonpartitioned NaN reads.

[JeonDaehong]

Thanks, reproduced: the partitioned read returned [2]. I kept Not for FLOAT/DOUBLE comparisons in negate rather than changing datum_cmp, since that would also change plain < / > on NaN partitions, which seems like a separate change. Not(leaf) is false only for NULL, and NaN < 1.0 is false, so the NaN row is kept, as in Java.

negated_float_comparison_keeps_nan_rows checks the read and delete_by_predicate, partitioned and not.

[JeonDaehong]

@JingsongLi thanks for the careful review. Both are fixed in eff224e, details inline.

I also rebased onto main. After #1153, negate handles NestedLeaf the same way as Leaf, and the NULL guard for a kept Not is shared by eval_row and the Arrow residual (Predicate::not_null_guard).

Checked with the ci.yml steps: fmt, clippy (fulltext,vortex), msrv 1.94, paimon unit (4983 passed), rest-server, integration with make docker-up, DataFusion (+vortex), Python (579 passed) and Go bindings.

[JingsongLi]

Requirement fit: SUPPORTED. Implementation: FINDINGS at eff224e. The prior floating-bitmap Top-1 and NaN-partition negation regressions are fixed on their covered paths; both added regression tests pass. A new public-API BTree + vector-search probe now returns no rows on this head and passes unchanged on baseline 46e6871. Details inline.

Local validation: 4,250 core library tests, 14 native predicate read/delete tests, 358 DataFusion library tests, 46 SQL read tests, and 5 C API checks passed. The actual Spark warehouse was provisioned. Two additional SQL reads fail only because liblumina_py.dylib is unavailable; lumina-data 0.1.0 supplies only a Linux x86_64 wheel, so these two native-library checks could not be completed on this macOS ARM host. Formatting and clean integration with current main were also checked.

[JingsongLi]

[P2] Keep exact floating BTree answers exact

This bitmap-specific downgrade currently applies to BTree results as well. BTree already implements floating != and ranges using the same bit-preserving total ordering as the residual filter, so those answers are exact. With the supported global-index.filter.refine-from-data=false option, scalar_matched_row_ids excludes all candidates that this method marks inexact. I reproduced the regression using actual Parquet data with finite f=[1.0, 2.0], a scalar BTree on f, an ivf-flat vector index, and filtered Top-1 with f != 1.0: baseline 46e6871 returns id 1, while this head returns no rows. No NaN or predicate negation is involved. Please derive exactness from the contributing index entries/readers and downgrade the floating bitmap fallback specifically, retaining exact BTree results; add a regression with refinement disabled.

[JeonDaehong]

Right, the downgrade was keyed on the predicate, not on the index that answered it. Fixed in ae3660e. evaluate_leaf now marks a leaf inexact only when a bitmap entry answered a floating range or negation, so BTree results stay exact. I reverted the check in is_exact_predicate.

floating_bitmap_candidates_are_refined_before_top_k now also runs on BTree, with and without refine-from-data=false. Before this change, BTree + refine-from-data=false returned [].

I also rebased onto current main.