[common][core] Preserve identity-mapped iterators safely - #9702
[common][core] Preserve identity-mapped iterators safely#9702jianguotian wants to merge 2 commits into
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
Found one row-tracking regression in the identity-mapping shortcut; details and reproduction are inline.
| if (partitionInfo == null && isIdentityMapping(indexMapping, row.batch().getArity())) { | ||
| return this; |
There was a problem hiding this comment.
[P1] Keep row-tracking wrappers out of the reader's reusable batch
For an unpartitioned table with row tracking enabled, reading t$row_tracking can reach this branch with a full identity mapping: FormatReaderMapping.Builder.trimKeyFields() returns an explicit identity array even when the schema mapping is null. DataFileRecordReader then calls assignRowTracking(), which replaces entries in batch.columns in place. Returning the original iterator exposes the Parquet/ORC reader's reusable column array to those mutations, so each reused batch wraps the previous batch's wrappers. Since their isNullAt() always returns false, each metadata getLong() recursively traverses the accumulated wrappers, causing progressively slower reads and eventually StackOverflowError. Previously, createMappedVectors() plus copy() isolated these mutations in a separate column array.
I reproduced this on JDK 8 with a real Parquet file, batch size 1, and the same mapping(...).assignRowTracking(...) sequence: this implementation overflows when checking the row at approximately 20,000 batches, while the identical test with the parent implementation completes all 100,000 batches. The four existing PR tests pass but do not exercise this reuse path.
Please keep row-tracking decoration isolated from the format reader's column array, or retain the copy path when row tracking needs to modify the vectors, and add a regression covering repeated batch reuse.
Purpose
Preserve specialized
ColumnarRowIteratorimplementations when an index mapping is an exact no-op, while keeping row-tracking decoration isolated from reusable format-reader batches.The read path can supply an explicit full identity mapping even when all columns remain in their original order. Previously,
ColumnarRowIterator.mappingcalledcopyfor every non-null mapping, and the basecopyimplementation could erase capabilities carried by a specialized iterator.Returning the source iterator introduces a separate ownership requirement:
assignRowTrackingreplaces entries inbatch.columnsin place. A format reader may reuse that column array across batches, so row-tracking wrappers must be applied to an isolated array rather than to the reader-owned batch.This change:
[0, 1, ..., batchArity - 1];Tests
Validated commit:
151f38b6a91830770caf14f55e3a1b8bf690e1a5.ColumnarRowIteratorTestandDataFileRecordReaderTest, 5/5 passed.ColumnarRowIteratorTestandDataFileRecordReaderTest, 5/5 passed.verifyforpaimon-common,paimon-core, and required upstream modules passed Checkstyle, Spotless, Enforcer, RAT, and packaging.DataFileRecordReaderTestrepeats reusable identity-mapped batches and asserts that reader-owned columns remain unchanged and each batch is recycled exactly once.git diff --checkpassed.