Repository navigation
Enable parallel file-level scanning for IcebergTableScan Datafusion Integration #2220
Description
Is your feature request related to a problem or challenge?
As noted in #1604, Iceberg-DataFusion read performance is currently bottlenecked by single-threaded execution. While size-based planning is the proposed long-term solution, a more immediate improvement would be to parallelize over FileScanTask and leverage ArrowReaderBuilder during plan execution.
Describe the solution you'd like
Pre-calculates the FileScanTask streams and partitions them across the available DataFusion partitions, updating the IcebergTableScan struct and ExecutionPlan trait:
Pre-partitioning Scan Tasks: IcebergTableScan now accepts a grouped tasks: Vec<Vec> rather than computing streams eagerly.
Propagating Partition Counts: The compute_properties method now dynamically returns Partitioning::UnknownPartitioning(tasks.len()) instead of the hardcoded 1.
Parallel Stream Execution: The execute phase uses self.tasks.get(partition) to spawn an ArrowReaderBuilder specific only to the slice of tasks mapped to that discrete DataFusion partition index.
Willingness to contribute
I would be willing to contribute to this feature with guidance from the Iceberg Rust community
Activity
[-]feat(datafusion): Enable parallel file-level scanning for IcebergTableScan Datafusion Integration[/-][+]Enable parallel file-level scanning for IcebergTableScan Datafusion Integration[/+]YuriyGavrilov commentedon Mar 26, 2026
+1
toutane commentedon Mar 31, 2026
Hey 👋, I have a working POC for this, I drafted it in #2298
The implementation is slightly different from the approach described here: rather than modifying
IcebergTableScan, it introduces a newIcebergPartitionedScanexecution plan + a dedicatedIcebergPartitionedTableProvider. The provider collects allFileScanTasksat plan time, thenIcebergPartitionedScanmaps one DataFusion partition per task, each executing independently viaArrowReaderBuilder.This keeps the existing
IcebergTableScanuntouched and lets users opt into the parallel path explicitly by registering the partitioned provider.Let me know if this direction sounds good, thanks!
toutane commentedon Jun 19, 2026
Following up on this issue and the earlier POC in #2298, I opened a smaller PR focused on the first scoped step: #2671
This PR adds an opt-in eager scan planning path for the DataFusion integration. When
iceberg.enable_eager_scan_planningis enabled,IcebergTableProvider::scan()plansFileScanTasks during physical planning, groups them across DataFusiontarget_partitions, and exposes the resulting output partition count asUnknownPartitioning(N). Eachexecute(partition)call then reads only its assigned task group throughArrowReaderBuilder.Compared with the broader work discussed in #2298, this PR intentionally keeps the scope limited to file-level scan parallelism and output partition count reporting. It does not include hash partitioning, size-aware bin-packing, or row-group/sub-file planning.