Repository navigation
fix: project columns correctly when scanning iceberg table metadata - #47
Conversation
| check_record_batches( | ||
| multiple_projected_columns_result, | ||
| expect
537a745 to
a2c73d6
Compare
| let stream = | ||
| futures::stream::once(fut) | ||
| .try_flatten() | ||
| .map(move |result| -> Result<_> { | ||
| let batch = result?; | ||
| match &projection { | ||
| Some(indices) => Ok(batch.project(indices)?), | ||
| None => Ok(batch), | ||
| } | ||
| }); |
There was a problem hiding this comment.
🤔 This seems off. This implies that we might be paying the price of reading columns that we are going to end up dropping.
Probably worth checking that we are not reading unnecessary columns on the first place.
There was a problem hiding this comment.
I did try to see if I could push this projection down further into the scan so that we don't have to manually project them after reading all of the columns, but it looks like iceberg-rust doesn't support this - the scan of the tables happens here which calls this in iceberg-rust and that doesn't seem to take projections
Admittedly, I'm still trying to build a mental model of all of this and might be missing something, so I can spend some more time trying to make this work (or open PRs on iceberg-rust to add support for this)
There was a problem hiding this comment.
🤔 Ok, this might be worth at least creating an issue in iceberg-rust upstream requesting for this functionality so that we can leave a TODO comment in the code here for future contributors, and leave it like this for now.
WDYT?
There was a problem hiding this comment.
Yes, sounds good - I'll create an issue on that repo and add a comment above the changes in this PR
There was a problem hiding this comment.
👋 I've updated the PR with the latest changes from main, opened an issue in iceberg-rust apache/iceberg-rust#3391 , and added a TODO in the code here
a2c73d6 to
2c56abf
Compare
21d41de to
cc706f1
Compare
cc706f1 to
f1383a9
Compare
2 participants
Summary
Adds support for projecting columns when scanning Iceberg metadata tables.
Addresses #41
Changes
projectioninto the schema stored in the ExecutionPlan'spropertiesexecute