Skip to content

[core][WIP] Optimize Parquet point lookup I/O on OSS - #9695

Draft
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:wip-parquet-point-lookup-io-20260909
Draft

[core][WIP] Optimize Parquet point lookup I/O on OSS#9695
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:wip-parquet-point-lookup-io-20260909

Conversation

@leaves12138

Copy link
Copy Markdown
Contributor

Purpose

[WIP] Reduce remote I/O overhead for Parquet point lookups, particularly when a global BTree index already provides row IDs.

  • Add a default length-aware FileIO.newInputStream(Path, long) overload and forward it through the relevant FileIO proxies. Implementations without the optimization retain the existing path.
  • Use lazy, exact OSS range reads when file length is known, avoiding an initial HEAD/read-ahead and supporting bounded vectored reads. Preserve Hadoop read statistics and error/close semantics.
  • Reuse the current data-evolution schema instead of fetching it again.
  • Apply row-ID row-group selection before dictionary/Bloom predicate filtering, retaining the conservative fallback for missing row offsets.
  • Buffer at most 128 KiB of the tail per opened remote Parquet file, allowing footer and nearby page indexes to share a range read. Preserve native vectored reads outside that buffer and caller-provided range buffers.

This is a read-path change; it does not change the data/index file formats or disable predicate checks. The draft status is intentional for API and connector review.

Earlier synthetic tests on two fixed 100-billion-row snapshots, using identical UUIDs and Java 11 clients, showed:

Same-table comparison Before P50 Optimized P50
Full point lookup, table with page indexes 953.00 ms 177.63 ms
Full point lookup, table without page indexes 3674.48 ms 955.00 ms

For one representative page-index lookup, Parquet I/O decreased from 26 GETs + 1 HEAD to 7 GETs, and response-body bytes decreased from 11.71 MB to 4.25 MB. These are client API measurements, not SQL-service end-to-end latency or a production SLA. They were collected on the earlier benchmarked base; this draft is rebased onto current master.

Tests

  • 111 targeted tests passed on Linux for Parquet, data-evolution reads, OSS range streams, vectored reads, REST FileIO and caching FileIO.
  • Review regressions reproduced against the pre-fix implementation: cached vectored reads must fill and return the supplied buffer (including zero-length ranges); closing one failing request must not prevent other requests from being closed, including repeated identical exceptions and unchecked failures.
  • Current/historical schemas, missing row offsets, predicate filtering, EOF/truncation, close during read, random seeks, tail boundaries and Hadoop read statistics.
  • Earlier real-table validation: 4,048 checked point lookups, plus before/after whole-file reads of two 16,934,000-row Parquet files with identical ordered content SHA-256 digests.
  • After the review fixes and rebase, four real-snapshot point-lookup smoke checks passed using the freshly packaged bundle and OSS connector.
  • Spotless, Checkstyle and git diff --check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant