Skip to content

Version 1.0: compiled kernels, TopoToolbox parity, and a GDAL installer - #12

Merged
gehilley merged 2 commits into
masterfrom
topoanalysis-1.0
Sep 22, 2026
Merged

gehilley merged 2 commits into
masterfrom
topoanalysis-1.0

Conversation

@gehilley

Copy link
Copy Markdown
Collaborator

Summary

  • Repairs ~70 defects, several of which prevented the package from importing at all on current NumPy and SciPy, and replaces the recursive basin traversals with iterative kernels. The public interface is unchanged; results that differ because they were wrong are listed in CHANGELOG.md under "Results that change".
  • Depression filling now uses the Priority-Flood algorithms of Barnes, Lehman and Mulla (2014), compiled to C++ (~50× faster), with an equivalent NumPy implementation that produces identical numbers when the extension cannot be built.
  • Adds TopoAnalysis.topotoolbox, reproducing TopoToolbox 2 (MATLAB) function by function, plus a parity table saying where exact agreement is and is not achievable.
  • Adds an installer for GDAL — both the C library and the Python bindings.

Why GDAL needs its own installer

There is no GDAL wheel on PyPI at any version for any platform: every release is an sdist whose setup.py shells out to gdal-config and compiles the bindings against a C library that must already exist. install.py and topoanalysis-gdal supply both halves, choosing between an existing C library, conda, a prebuilt Windows wheel, the system package manager, or a from-source build of PROJ and GDAL into the virtualenv that needs no administrator rights.

Two failure modes are detected explicitly:

  • Bindings built without numpy. GDAL sdists only declared numpy as a build dependency from 3.9 on; below that, pip's build isolation omits it and osgeo.gdal_array is silently skipped — osgeo imports cleanly and ReadAsArray() fails at run time. Ubuntu 22.04 (GDAL 3.4) and 24.04 (3.8) are both affected.
  • A stale PROJ_LIB/GDAL_DATA. Conda activation exports both, and they outrank the paths compiled into every other GDAL on the machine; a PROJ 9 library handed PROJ 8's proj.db refuses every lookup.

Test plan

Verified on macOS (Python 3.10):

  • python3 install.py → exit 0, GeoTIFF round-trip
  • python3 install.py --gdal-strategy source → PROJ + GDAL built from source into the venv, exit 0, 159 drivers
  • python3 install.py --no-gdal → exit 0, library works without GDAL
  • 350 tests passing with the C++ kernels, 347 with the NumPy kernels
  • sdist and wheel both carry the new modules; entry points registered

Not yet exercised anywhere — this is what CI is for: Linux (the apt route, lib64, loader paths), Windows (the prebuilt-wheel strategy), the conda and system strategies, and the Dockerfile. The workflow added here covers install.py on all three platforms, an Ubuntu job that reproduces the gdal_array trap, and a scheduled from-source build.

🤖 Generated with Claude Code

Repairs roughly 70 defects, several of which prevented the package from
importing at all on current NumPy and SciPy, and replaces the recursive
basin traversals -- which needed a million-frame recursion limit and
still overflowed on real DEMs -- with iterative kernels.

The public interface is unchanged: every class, method and keyword still
exists and still means what it did. A handful of results differ because
they were wrong; those are listed in CHANGELOG.md under "Results that
change".

Depression filling now uses the Priority-Flood algorithms of Barnes,
Lehman and Mulla (2014), compiled to C++ and roughly 50x faster, with an
equivalent NumPy implementation in TopoAnalysis.fastops that produces
identical numbers when the extension cannot be built. The compiled
kernels release the GIL.

Adds TopoAnalysis.topotoolbox, which reproduces TopoToolbox 2 (MATLAB)
function by function, and docs/topotoolbox_parity.md, which tabulates the
differing conventions and says where exact agreement is not achievable.

Adds an installer for GDAL, the one dependency pip cannot supply. There
is no GDAL wheel on PyPI at any version for any platform -- only an sdist
that compiles the bindings against a C library which must already exist
-- so "install GDAL" is two jobs that have to agree. install.py and the
topoanalysis-gdal command supply both halves, choosing between an
existing C library, conda, a prebuilt Windows wheel, the system package
manager, or a from-source build of PROJ and GDAL into the virtualenv
that needs no administrator rights. Two failure modes are detected
explicitly: bindings built without numpy, where osgeo imports cleanly and
ReadAsArray fails at run time, and a stale PROJ_LIB inherited from
another installation.

Tests: 350 passing with the C++ kernels, 347 with the NumPy kernels.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gehilley gehilley added the test-source-build Run the ~40min from-source GDAL build on this PR label Sep 22, 2026
The Windows route is the only GDAL strategy with nothing behind it: there
is no gdal-config and no source build on Windows, so a prebuilt wheel that
bundles the C library is the whole plan. It had never been exercised
anywhere. The new job installs GDAL through it and asserts that
ReadAsArray actually works rather than that osgeo merely imports.

The from-source job stays off the default path -- it is about forty
minutes -- but a pull request labelled `test-source-build` now runs it,
which is what a change to the build itself needs. Waiting for the weekly
schedule means finding out on Monday. `labeled` joins the pull_request
event types so applying the label is enough to start it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gehilley
gehilley merged commit 43e1e57 into master Sep 22, 2026
19 checks passed
@gehilley
gehilley deleted the topoanalysis-1.0 branch September 22, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test-source-build Run the ~40min from-source GDAL build on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant