Repository navigation
[python] Support structured-data local cache selection - #10225
XiaoHongbo-Hope wants to merge 3 commits into
Conversation
| # ParquetFile(filesystem=...) is unavailable in PyArrow 6. | ||
| self._parquet_source = file_io.filesystem.open_input_file( | ||
| file_path_for_pyarrow) | ||
| self._parquet_source = _open_pyarrow_input_file(file_io, file_path) |
There was a problem hiding this comment.
[P2] Forward the known file size to the direct Parquet reader
SplitRead already supplies DataFileMeta.file_size, but this call drops it when row selection, bounded VARIANT or selected MAP reads open their separate Parquet source. The resulting CachingInputStream calls the delegate's get_file_size() before reading cached blocks. With the actual Jindo handler, the same known-size selected-row read makes 0 backend metadata requests without the cache wrapper and 1 with it.
I also reproduced this with a fully cached Parquet file: perform a full scan with file_size supplied, remove the source and clear the parsed-footer cache, then read again with the same file_size and row_indices=[1]. The full warm scan succeeds, but the selected-row scan raises FileNotFoundError while querying the source size. Forwarding the known size makes the selected-row scan succeed as well.
Please pass file_size to _open_pyarrow_input_file here and add coverage combining a known size with selected-row warm reads.
b879f89 to
039f543
Compare
Summary
Add
structured-datatolocal-cache.whitelistfor supported Parquet, ORC, Avro, ROW and Mosaic reads without caching BLOB or video bodies. PyPaimon Vortex and Lance readers bypass this cache. Defaults anddata/*behavior are unchanged. Complements apache/paimon-rust#972.Validation
Focused cache and format-reader tests: 95 passed, 1 skipped, 64 subtests passed. Flake8 and
git diff --checkpassed. Includes a known-size selected-row warm-read regression.