Repository navigation
[EPIC] Execution-time adaptive pruning for Iceberg readers #3343
Description
What's the feature are you trying to implement?
Summary
Add generic execution-time predicate support to iceberg-rust so readers can progressively avoid file, row-group, page, and payload reads as predicates become available during query execution.
The reader remains producer-agnostic: callers publish Iceberg predicates and generations, while iceberg-rust handles safe application through the existing reader pipeline.
This epic tracks the upstream reader infrastructure required by execution engines such as DataFusion Comet.
Related work
- Runtime predicate discussion: feat(datafusion): runtime filter / dynamic predicate pushdown into IcebergTableScan #2376
- Live predicate discussion: Explore live runtime predicate refresh during Arrow reads #3336
- Arrow 60 upgrade: Bump to Arrow 60 #3257
Work items
Dependencies
Reader foundations
- Resolve unknown delete-file sizes lazily — Support unknown delete-file sizes in externally constructed scan tasks #3344, PR feat(reader): resolve unknown delete-file sizes lazily #3345
- Reuse the inclusive statistics evaluator with borrowed metrics — Evaluate inclusive metrics from borrowed file statistics #3346, PR refactor(expr): evaluate inclusive metrics from borrowed statistics #3347
- Add the generic runtime predicate publication API — Support execution-time predicates in ArrowReader #3350, PR feat(reader): support execution-time predicates in ArrowReader #3351
- Apply runtime predicates at file-task startup — Support execution-time predicates in ArrowReader #3350, PR feat(reader): support execution-time predicates in ArrowReader #3351
- Add opt-in file statistics retention to scan tasks
Adaptive pruning
- Reject file tasks before footer and delete-file I/O
- Refresh predicates and prune unread row groups
- Represent static page selections per row group
- Preserve position and equality deletes during live pruning
- Refresh page pruning and row filters after predicate updates
- Preserve late materialization with runtime predicates
Correctness and robustness
- Bound refresh overhead and cover reader edge cases
- Add adaptive pruning observability and reader-level metrics
- Complete delete, split-task, schema-evolution, and fallback coverage
Correctness principles
- Runtime predicates are conservative restrictions.
- Previously skipped data is never required by a later predicate publication.
- Missing or unusable statistics retain candidates.
- Runtime predicate failures must not disable planned filtering or delete handling.
- Position deletes and metadata row positions preserve original file ordinals.
- Equality deletes remain active across predicate refreshes.
- Existing reader and schema adaptation paths should be reused rather than duplicated.
Definition of done
- No private Arrow/Parquet patch is required.
- Generic task-start and live runtime predicate APIs are upstream.
- Whole-file, row-group, page, and row-filter pruning use the existing reader pipeline.
- Position deletes, equality deletes, and combined deletes are supported.
- Stable predicates add minimal reader overhead.
- Each work item above is linked to its corresponding issue or PR.
Willingness to contribute
I can contribute to this feature independently
Activity
unikdahal commentedon Oct 4, 2026
take
unikdahal commentedon Oct 12, 2026
Hi all, a quick heads-up: I'll be away for about a month starting now, and I don't want this epic to block anyone in the meantime. If any part of it is useful to you, please feel free to pick it up, rework it, or take it in a different direction. There's no need to wait for me or ask first.
Here's where things stand, to make that easier:
FileScanTaskDeleteFilefor 0.11, modelling an unknown size asOption<u64>is cleaner than the0sentinel. The PR needs a rebase on fix(scan): validate deletion vector delete tasks #3145 plus that change. It's small, if someone wants an easy start.ArrowReaderBuildertoday, notTableScan::to_arrow, so wiring it intoTableScanwould be a natural follow-up.A few things I learned from an end-to-end prototype (iceberg-rust plus DataFusion Comet), in case they save someone time:
INlist for every file adds up quickly. A monotonic generation from the publisher keeps the cache simple and safe._pos/_row_idhave to keep original file ordinals through all of the pruning.I also just saw apache/datafusion-iceberg#68, which proposes a very similar hook. If that design, or its author, ends up driving this, I'm completely happy for #3351 to be closed in favour of it. What matters is that the feature lands, not whose PR it is.
Thanks to everyone who has weighed in on the related issues and PRs, and sorry for leaving this half-finished for a few weeks.
alexanderbianchi commentedon Oct 12, 2026
Thanks for all the summaries, they are useful. DataDog is very interested in dynamic filtering (ideally landing it before end of november, and ideally on the open source not in our fork that we try and keep slim) I'll be happy to keep an eye on any reviews of #3351 and dive in responding to them. I still need to do a review myself.
anuragmantri commentedon Oct 12, 2026
Thanks for the write up and summaries!
I picked up "Add opt-in file statistics retention to scan tasks" in #3422 as I needed it for an optimization in apache/datafusion-iceberg#55.
JeonDaehong commentedon Oct 12, 2026
Thanks @unikdahal, and enjoy the time off! I'll pick up #3345: rebase it on #3145 and switch the unknown size to Option. I'll open a follow-up PR crediting you and link it here.
JeonDaehong commentedon Oct 13, 2026
Opened #3433 for #3345.