no versions specified in requirements.txt cause unpredictable failures due to dependency upgrades.

[ladc]

I had made some modifications to our operational pysteps, tested the new version and this worked fine in our staging environment. Our production team created a new docker image on the production environment, but here the new prod runs crashed quite dramatically due to an error in a very freshly upgraded dependency.

This can happen at unpredictable moments because the requirements.txt file does not pin specific versions.

I have pinned our docker dependencies to a version that I know works. However, @jbelien pointed out to me that it would make more sense to simply specify the versions in the mainline pysteps requirements.txt instead.

I propose therefore to pin the versions of dependencies for reproducability.

To avoid having to update this manually every time packages are updated, one can run dependabot.

[ladc]

Additionally, dependencies are defined all over the place:

• environment.yml (and the corresponding _dev file)
• requirements.txt (and the corresponding _dev file)
• setup.py
• pyproject.toml

Which is the source of truth?

[jbelien]

I'm no Python expert but I think you could manage all your dependencies in 1 single file: pyproject.toml ; this could be your single source of truth and should help you manage your dependencies more easily. That would also allow you to enable Dependabot to upgrade your dependencies.

Documentation: https://packaging.python.org/en/latest/guides/writing-pyproject-toml/

[project]
# Your core dependencies (instead of requirement.txt)
dependencies = [
  ...
]

[project.optional-dependencies]
# Your dev dependencies (instead of requirements_dev.txt)
dev = [
  ...
]
# Your dependencies needed to generate the documentation (instead of doc/requirements.txt)
docs = [
  ...
]

You can then use

pip install -e . # Install core dependencies
pip install -e .[docs] # Install dependencies for the documentation generation
...

Those are just my 2 cents, as I said I'm no Python expert. 😄
I'm willing to create a PR to try to clean this up but it would need to be validated/finalized by one of you.