Generate Pypi-compatible indexes for our wheelhouses

[mboisson]

Using:
https://pypi.org/project/dumb-pypi/

We can generate a plain static file local Pypi index:

module load python/3.8
virtualenv dumb-pypi
source dumb-pypi/bin/activate
pip install dumb-pypi
pushd /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/; ls *.whl > ~/tmp/generic.lst; popd
dumb-pypi --package-list generic.lst --packages-url file:///cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/ --output-dir generic-index

To use that index:

unset PIP_CONFIG_FILE
pip install --index file:///home/mboisson/tmp/generic-index/simple torch
Looking in indexes: file:///home/mboisson/tmp/generic-index/simple
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/torch-1.9.1+computecanada-cp38-cp38-linux_x86_64.whl
Processing /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/generic/typing_extensions-4.3.0+computecanada-py3-none-any.whl
Installing collected packages: typing-extensions, torch
Successfully installed torch-1.9.1+computecanada typing-extensions-4.3.0+computecanada

What’s the significance of this ? We can generate and serve serverless instances of a local pypi. That means those can be used by tools that don’t support --find-links to point to a local directory containing wheels and only support pypi-style repositories (looking at you, poetry)

[mboisson]

Well, that does not work with poetry because of this:

  InvalidSchema

  No connection adapters were found for 'file:///home/mboisson/tmp/generic-index/simple/torch/'

  at test_poetry/lib/python3.8/site-packages/requests/sessions.py:792 in get_adapter
      788│             if url.lower().startswith(prefix.lower()):
      789│                 return adapter
      790│
      791│         # Nothing matches :-/
    → 792│         raise InvalidSchema(f"No connection adapters were found for {url!r}")
      793│
      794│     def close(self):
      795│         """Closes all adapters and as such the session"""
      796│         for v in self.adapters.values():`

references: psf/requests#2732

[mboisson]

It could be possible to generate the index, and then serve them locally using a local http.server:

python -m http.server $((UID % 60000 + 1024)) --directory /home/mboisson/tmp/local-index/ &

We must then add a symbolic link to the wheelhouse inside of that folder.

[mklasby]

+1, this would be a big help so that we can effectively use poetry in compute canada servers. I do think this issue is best addressed in poetry itself as there really should be support for --find-links similar to pip.

My current workarounds are to either download the necessary wheels to my project and edit the pyproject.toml file to point the dependency to the cc wheel or build the lock file on a local environment and poetry export to a requirements file that I can use pip with to rebuild on compute canada with the local wheelhouse

[ccoulombe]

I think poetry should instead provide the means to find wheels via find-links. If a PR is not already open, it would be the best solution.

Plus, poetry managed repository can be installed by pip with PEP-517.