Skip to content

migrate Zarr backend to ZarrCore and add ZarrZip extension support - #73

Open
lazarusA wants to merge 12 commits into
masterfrom
la/zarrcore
Open

lazarusA wants to merge 12 commits into
masterfrom
la/zarrcore

Conversation

@lazarusA

@lazarusA lazarusA commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

as hinted in #72

Plus is needed to properly implement the multi scales work on PyramidSchemes.jl

@codecov

codecov Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.66667% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.53%. Comparing base (49d7155) to head (f5d8740).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
ext/ZarrCoreExt.jl 89.65% 6 Missing ⚠️
ext/ZarrZipExt.jl 93.75% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master      #73      +/-   ##
==========================================
+ Coverage   52.04%   59.53%   +7.49%     
==========================================
  Files          12       13       +1     
  Lines         490      514      +24     
==========================================
+ Hits          255      306      +51     
+ Misses        235      208      -27     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ressed

create_dataset now forwards `compressor` (Zarr) and `compress` (NetCDF)
to the coordinate arrays, not only to the data variables. Other keyword
arguments such as filters stay data-variable only, since they can be
type specific.

NetCDF iscompressed always returned false because NetCDF.jl does not read
the deflate level back when opening a variable. Query it with
nc_inq_var_deflate instead.
Zarr v3 allows omitting dimension_names, which zarr-python does for
0-dimensional arrays. get_var_dims threw for such arrays because neither
dimension_names nor _ARRAY_DIMENSIONS was present. Return an empty list
for arrays without dimensions instead.
Comment thread ext/ZarrCoreExt.jl Outdated
@felixcremer

Copy link
Copy Markdown
Member

I changed to using the zarr_format function from ZarrCore and renamed the corresponding keyword to format. Why is this breaking?

Comment thread ext/ZarrCoreExt.jl
@lazarusA

lazarusA commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

by loading Zarr.jl should not affect end users, but in isolation is breaking, then in order to access other features, like Zip, you need to be explicit about what to load.

@felixcremer

Copy link
Copy Markdown
Member

The last change is needed until JuliaIO/Zarr.jl#359 lands.

@lazarusA

Copy link
Copy Markdown
Collaborator Author

Let's do this update only with the latest version of Zarr, then when a new version is patch-release we come back to this overload approach and enable your suggestion.

Now, we should do this transition.

Comment thread ext/ZarrCoreExt.jl
Comment thread src/datasets/datasetinterface.jl
Comment thread ext/ZarrCoreExt.jl Outdated
alex-s-gardner and others added 3 commits October 15, 2026 14:32
ZGroup has no metadata field in ZarrCore 0.11.1, so zarr_format(ds)
raised a FieldError and every add_var call failed. The format is the
group's zarr_format field, a ZarrFormat{N} that converts to Int.

Co-authored-by: Claude 🤖 <noreply@anthropic.com>
ZarrCore.zcreate takes the format as the zarr_format keyword. The tests
that call it directly passed format, which zcreate rejects; format is
the keyword of the YAXArrayBase functions only.

Co-authored-by: Claude 🤖 <noreply@anthropic.com>
The Zarr v3 zarr.json files carry the format version under the key
zarr_format; there is no key named format.

Co-authored-by: Claude 🤖 <noreply@anthropic.com>
alex-s-gardner added a commit to alex-s-gardner/ChunkManifests.jl that referenced this pull request Oct 15, 2026
Zarr 0.11 is an umbrella package: the array, store and codec types live in
ZarrCore, and the umbrella re-exports only some of their names. The
ZarrDatasets 0.1.7 release depends on ZarrCore and cannot load beside the
pre-split Zarr 0.10 that `[sources]` pinned, which is why CI is red on main
(`Method overwriting is not permitted during Module precompilation`).

- `Project.toml` requires `Zarr = "0.11"`, and `ZarrCore` is a dependency.
  The names ZarrCore owns are qualified with it (about 190 changed lines in
  32 files, nearly all `Zarr.X` -> `ZarrCore.X`), as ExplicitImports requires.
- `[sources]` points Zarr, ZarrCore and ZarrZlib at the fork branch
  `complex-int-dtype-v0.11`: Zarr.jl `main` plus complex-integer arrays,
  non-finite complex fill values and threaded chunk decoding. ZarrZlib is
  listed in `[extras]` only so `[sources]` can name it. The docs environment
  repeats the pin.

Not covered: the YAXArrays test is not run, because YAXArrayBase 0.7 requires
Zarr 0.10, so YAXArrays cannot be installed beside this package until
YAXArrayBase supports Zarr 0.11 (JuliaDataCubes/YAXArrayBase.jl#73). The docs
example for it is replaced by a note.

Tested locally on Julia 1.13 with 4 threads: 8434 tests pass, none fail, one
is marked broken as before, including Aqua and ExplicitImports; the docs
build. Julia 1.10 ignores `[sources]` and resolves registered Zarr 0.11.1,
which the CI lts jobs exercise.

Co-authored-by: Claude 🤖 <noreply@anthropic.com>
Rename the format keyword of create_dataset, create_empty and add_var back
to zarr_format, the name ZarrCore uses. With `format` as the keyword, a
`zarr_format` passed by the caller reached zcreate, which overrides the
group format and split the store into a v2 group with v3 arrays, losing
those arrays on reopening.

Read the group format with ZarrCore.zarr_format instead of extending it
with a method that returned an Int. This needs ZarrCore 0.11.2.

Skip the remote Zarr test with a warning when the server can not be
reached, instead of failing on TLS or rate limit errors.
@lazarusA

Copy link
Copy Markdown
Collaborator Author

I think is ok now. The only thing I should have done is to make public the zarr_format function in ZarrCore.jl, but that we do in another round, if you all agree on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants