Skip to content

[python] Support structured-data local cache selection - #10225

Draft
XiaoHongbo-Hope wants to merge 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/blob-meta-cache
Draft

XiaoHongbo-Hope wants to merge 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/blob-meta-cache

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add structured-data to local-cache.whitelist for supported Parquet, ORC, Avro, ROW and Mosaic reads without caching BLOB or video bodies. PyPaimon Vortex and Lance readers bypass this cache. Defaults and data/* behavior are unchanged. Complements apache/paimon-rust#972.

Validation

Focused cache and format-reader tests: 95 passed, 1 skipped, 64 subtests passed. Flake8 and git diff --check passed. Includes a known-size selected-row warm-read regression.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft September 27, 2026 01:22
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Cache BLOB metadata ranges separately from data [python] Support BLOB metadata caching Sep 27, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 27, 2026 04:42
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft September 27, 2026 10:40
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Support BLOB metadata caching [python] Support Parquet-only local caching Sep 27, 2026
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Support Parquet-only local caching [python] Support local-cache extension exclusions Sep 28, 2026
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Support local-cache extension exclusions [python] Support structured-data local cache selection Oct 19, 2026
# ParquetFile(filesystem=...) is unavailable in PyArrow 6.
self._parquet_source = file_io.filesystem.open_input_file(
file_path_for_pyarrow)
self._parquet_source = _open_pyarrow_input_file(file_io, file_path)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Forward the known file size to the direct Parquet reader

SplitRead already supplies DataFileMeta.file_size, but this call drops it when row selection, bounded VARIANT or selected MAP reads open their separate Parquet source. The resulting CachingInputStream calls the delegate's get_file_size() before reading cached blocks. With the actual Jindo handler, the same known-size selected-row read makes 0 backend metadata requests without the cache wrapper and 1 with it.

I also reproduced this with a fully cached Parquet file: perform a full scan with file_size supplied, remove the source and clear the parsed-footer cache, then read again with the same file_size and row_indices=[1]. The full warm scan succeeds, but the selected-row scan raises FileNotFoundError while querying the source size. Forwarding the known size makes the selected-row scan succeed as well.

Please pass file_size to _open_pyarrow_input_file here and add coverage combining a known size with selected-row warm reads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants