fix(arrow): read nested struct fields as null for files without ids or name mapping

[zhangfengcdt]

Which issue does this PR close?

• Closes ArrowReader returns nested field values for Parquet files without field ids or a name mapping, where Iceberg Java returns NULL #3436.

What changes are included in this PR?

For a Parquet file with no field ids and no name mapping, only top-level columns get fallback ids. RecordBatchTransformer then matched the file's id-less struct fields to the table's by position, so ArrowReader returned their stored values. iceberg-java matches nested struct fields by id, finds none, and reads them as absent.

Id-less struct fields now go through the existing absent-field path, as in Java's ReadBuilder.struct: identity partition constant, then null, or an error for a required field with no initial-default. This removes the positional fallback kept in #3255 until recursive name mapping, which #3384 added.

Map key and value now match by position, as in iceberg-java's map readers. Without that, an id-less map would fail on its required key. For files with ids the result is the same, since map entries are always key then value. Lists already matched their element by position. Structs with only some field ids still error.

Behavior a reviewer may not expect:

• A required nested field in such a file now errors with "required nested field ... is absent", matching Java's "Missing required field". test_read_parquet_without_field_ids_with_struct used required fields, so it now uses optional ones.
• A list of structs comes back with its structs and null fields. Spark returns an empty list there when it reads no nested leaf; this follows the spec and Java's readers instead.
• get_arrow_projection_mask_fallback still reads whole top-level columns, so nested values are read and then discarded. Pruning them is left out of this fix.
• An optional struct keeps its validity from the file and reads its fields as null. Spark's reader does the same; Java's GenericParquetReaders returns a null struct on every row instead.

Are these changes tested?

Yes. Unit tests in record_batch_transformer.rs for an id-less struct, an id-less nested struct and an id-less map with a struct value, and the ArrowReader test for a file without field ids, which now expects null struct fields. All four fail on main.

AI Disclosure

I used Claude Code to investigate the issue and write the fix and tests. Worth a close look: the claim that Java errors on a required nested field absent from the file, and the decision to match map key and value by position for files with ids too.

[JeonDaehong]

Checked the two points you flagged against iceberg-java (core/parquet/data built from main at 059e7ea94, which includes apache/iceberg#17320), reading Parquet files with no field ids and no name mapping through GenericParquetReaders:

file / read schema	iceberg-java	main	this PR
s: struct<a: required int>	Missing required field: a	stored values	required nested field a is absent ...
m: map<string, struct<x: int>>	{k1={x=null}}	stored values	{k1: {x: }}
s: struct<a: optional int>, row 3 s null	s null on every row	stored values	{a: }, {a: }, null

So the required-field error and matching map key/value by position both match Java. One difference that isn't in the description: for an optional nested field, Java returns a null struct on every row, because its presence selection only uses columns that have ids and none under s do. This PR keeps s present on rows 1 and 2 (it reads the whole top-level column), which I think is the more accurate result, but it differs from Java.

[zhangfengcdt]

Checked the two points you flagged against iceberg-java (core/parquet/data built from main at 059e7ea94, which includes apache/iceberg#17320), reading Parquet files with no field ids and no name mapping through GenericParquetReaders:

file / read schema iceberg-java main this PR
s: struct<a: required int> Missing required field: a stored values required nested field a is absent ...
m: map<string, struct<x: int>> {k1={x=null}} stored values {k1: {x: }}
s: struct<a: optional int>, row 3 s null s null on every row stored values {a: }, {a: }, null
So the required-field error and matching map key/value by position both match Java. One difference that isn't in the description: for an optional nested field, Java returns a null struct on every row, because its presence selection only uses columns that have ids and none under s do. This PR keeps s present on rows 1 and 2 (it reads the whole top-level column), which I think is the more accurate result, but it differs from Java.

Thanks @JeonDaehong for the review.

For the struct validity: Java's readers disagree here. Spark's reader keeps the struct present with null fields (the Comet repro in #3436 shows [0,[null,null]]), while GenericParquetReaders nulls the struct. This PR follows the Spark result, which is also how iceberg-rust treats a struct whose fields are all absent from a file that has ids.

I added a null struct row to promote_struct_without_source_field_ids_reads_null to pin it, and noted the difference in the description.

[comphead]

Verified against iceberg-java: ParquetSchemaUtil.addFallbackIds assigns ids to top-level fields only, so reading id-less nested fields as absent matches Java. CI is green and the branch merges cleanly with main. I also ran four single-line production mutations against the new and changed tests (details on the test comment below).

• get_arrow_projection_mask_fallback still decodes whole top-level columns, and for a struct column the nested values are now always discarded. Pruning is not free because the struct keeps the file's validity, so at least one leaf per struct must still be read. A tracked follow-up issue would help.
• This turns values that #3255 returned into nulls with no signal, which matches Java. A line in the release notes or the name-mapping docs saying migrated tables need a name mapping for nested fields would help users who hit it.
• Nit: PromotePlan::Map::entries is now always ChildPlan::FromSource, so the Add arm in apply_struct is only reachable from structs. Fine to leave, since splitting it would add more code than it removes.

[comphead]

When the field counts differ, this guard falls through to the _ arm, which reports "expected struct-typed map entries" even though the entries are structs. A separate message such as "map entries have N source fields, expected M" would point at the real problem.

[zhangfengcdt]

I split _ arm and also a count mismatch errors with "map entries have N source fields, expected M".

[comphead]

This test looks redundant. In a mutation spot check, every production mutation it caught was also caught by another test:

• Restoring the positional fallback in build_struct_children failed this test, promote_idless_map_struct_value_reads_null_fields, and the ArrowReader test.
• Making absent fields non-null failed this test along with most of the other promote tests.
• Passing None for struct validity in apply_struct did not fail it.

The outer struct is fully absent, so the inner struct is never visited and the test adds no path beyond promote_struct_without_source_field_ids_reads_null. Consider dropping it. Likewise promote_idless_map_keeps_data is now caught together with promote_idless_map_struct_value_reads_null_fields for the map change (mutation: build map entries by id again), so one map test with a struct value would be enough. This is a spot check on four mutations, not a coverage proof.

[zhangfengcdt]

I dropped the redundant test and now it is tested in promote_struct_without_source_field_ids_reads_null.

[JeonDaehong]

Thanks @zhangfengcdt, that makes sense: following Spark's reader, and the same rule iceberg-rust already uses for files with ids, is a fair choice, and the description now says so. Rechecked 3406cf2: arrow:: tests pass (261), and reading the same id-less files gives the same results as before (optional field: {a: null}, {a: null}, null; required field: the error; map: by position). I did not run Spark's reader myself. LGTM.

[zhangfengcdt]

Verified against iceberg-java: ParquetSchemaUtil.addFallbackIds assigns ids to top-level fields only, so reading id-less nested fields as absent matches Java. CI is green and the branch merges cleanly with main. I also ran four single-line production mutations against the new and changed tests (details on the test comment below).

• get_arrow_projection_mask_fallback still decodes whole top-level columns, and for a struct column the nested values are now always discarded. Pruning is not free because the struct keeps the file's validity, so at least one leaf per struct must still be read. A tracked follow-up issue would help.
• This turns values that fix(arrow): reconcile evolved nested structs by field id, not position #3255 returned into nulls with no signal, which matches Java. A line in the release notes or the name-mapping docs saying migrated tables need a name mapping for nested fields would help users who hit it.
• Nit: PromotePlan::Map::entries is now always ChildPlan::FromSource, so the Add arm in apply_struct is only reachable from structs. Fine to leave, since splitting it would add more code than it removes.

@comphead Thanks for the review. Both comments should be addressed now:

• Pruning: opened Prune nested leaves in the projection fallback for files without field ids #3480 to track reading only the needed leaves in the fallback projection, keeping one leaf per struct for validity.
• Release notes: added a CHANGELOG.md line saying tables migrated without field ids need a name mapping covering nested fields.

[comphead]

Thanks @zhangfengcdt love negative PRs

[comphead]

@kevinjqliu @CTTY @laskoviymishka @blackmwk appreciate you can take a look