Skip to content

Switch default cutout compression to zstd with BitRound quantization - #529

Open
fneum wants to merge 2 commits into
masterfrom
zstd-compression
Open

fneum wants to merge 2 commits into
masterfrom
zstd-compression

Conversation

@fneum

@fneum fneum commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Closes #508.

Changes proposed in this Pull Request

Cutout.prepare now compresses cutouts with zstd (complevel 3) after BitRound quantization to 14 mantissa bits (Klöwer et al., 2021). The maximum relative error is 2**-15 = 3.1e-5.

  • Default: {"compression": "zstd", "complevel": 3, "quantize_mode": "BitRound", "significant_digits": 14}. For the old behaviour, pass {"compression": "zlib", "complevel": 9, "shuffle": True}.
  • All variables: the encoding now applies to all variables of the cutout, not only to the new ones. xarray < 2026.9 drops the zstd setting of existing variables when it rewrites the file. Without this change, a second prepare call would store the earlier variables uncompressed.
  • Dependency: requires netCDF4>=1.7.4, whose wheels bundle the zstd plugin. conda-forge libnetcdf (tested with 4.10.1) also works.

Benchmark

Europe, January to March 2013, 2160×131×177, 3.2 GB in memory:

Setting Write Read Size
zlib 9 (old default) 143.9 s 10.6 s 1582 MB
zstd 3, lossless 10.5 s 6.4 s 1497 MB
zstd 3 + BitRound 14 (new) 11.8 s 6.0 s 797 MB

Compatibility

  • Existing zlib cutouts still read without changes. A new prepare call recompresses them with zstd.
  • netCDF4<1.7.4 (pip) cannot read the new cutouts. It fails with NetCDF: Filter error: undefined filter encountered.

Checklist

  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • Newly introduced dependencies are added to environment.yaml, environment_docs.yaml and setup.py (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

fneum added 2 commits October 7, 2026 14:21
Default compression in Cutout.prepare is now zstd (complevel 3) after
BitRound quantization to 14 mantissa bits (max relative error 2**-15).
The encoding applies to all variables, because xarray < 2026.9 drops the
zstd setting of existing variables when the cutout file is rewritten.
Require netCDF4>=1.7.4, whose wheels bundle the zstd plugin.

Closes #508
@fneum
fneum requested a review from euronion October 7, 2026 12:40

@euronion euronion left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting.

Did you compare the old and new cutout values as well, to be sure there is not further corruption/mix of values from e.g. NaN to 0?

We do not have the atlite version encoded somewhere in the cutouts, do we? So we cannot mark new cutouts incompatible with older versions of the library, can we? A feature we should implement.

@fneum

fneum commented Oct 7, 2026 •

Copy link
Copy Markdown
Member Author

Did you compare the old and new cutout values as well, to be sure there is not further corruption/mix of values from e.g. NaN to 0?

Yes, they were within numerical tolerances. Although I will have to look at the failing CI tests for older Python versions.

We do not have the atlite version encoded somewhere in the cutouts, do we? So we cannot mark new cutouts incompatible with older versions of the library, can we? A feature we should implement.

No, we don't have that yet. And yes, we should.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Switch from zlib to a modern compression standard

2 participants