Skip to content

JSON data-file helpers reject valid floats and silently corrupt non-finite partition values #3409

Open

Description

@hareshkh

Apache Iceberg Rust version

0.10.0

Describe the bug

JSON round-trips reject valid float partition values such as 123.456 because of an absolute f32::EPSILON check. NaN and ±Infinity silently become null for both float and double partitions, allowing incorrect metadata to be committed.

To Reproduce

  1. Take test_insert_into_partitioned from datafusion-iceberg PR #54 and run it against the PR’s base commit, which still uses the JSON helpers.
  2. Use 100.1 for the finite float partition value. The insert fails with Unable to convert raw literal (double) fail convert to type float.
  3. Change it to 1.5, keeping the NaN, Infinity, and -Infinity rows. The insert succeeds, but inspecting the committed manifest shows null for those three partition values. File and row counts still look correct.
  4. Repeat with a double partition column: non-finite values also silently become null.

Expected behavior

Preserve valid partition values. Reject unsupported values explicitly instead of silently converting them to null.

Willingness to contribute

None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions