Add Lium backend

[peterschmidt85]

This PR adds experimental Lium backend support for GPU runs and S3-backed volumes. It uses the online provider in dstackai/gpuhunt#259.

• Execution: Starts the stock dstack runner through Lium's root SSH, preserving the image's environment and working directory. Jobs use dstack's existing image-user detection and privilege drop, so non-root images work without changing the runner.
• Private registries: Creates a temporary credential when registry_auth is set. Each job gets a private template; the backend removes the template and credential after the pod is deleted.
• Volumes: Supports one S3-backed volume per pod, without size or region. Writable mounts require root; non-root jobs without volumes work normally. Local storage is unencrypted and deleted with the pod.

Validation: 211 targeted tests passed (36 skipped), Ruff, Pyright, strict docs build, and package build. Live E2E covered private-image authentication, non-root execution, volume persistence across rentals, cancellation, and cleanup. Temporary capacity failures recovered through normal no-capacity retries. Long-running reliability and RDMA remain unverified.

Before merging:

• Correct RAM and disk reporting in gpuhunt: advertised RAM must not exceed the allocation, and separate filesystem quotas must not be added together as one usable disk.
• Merge dstackai/gpuhunt#259 and publish a gpuhunt release.
• Replace the temporary Datura-ai/gpuhunt@lium-provider dependency with the released version, remove allow-direct-references, and recheck against that release.

[surcyf123]

Thanks @peterschmidt85. We tried this branch (eafa49b) with a local server, a lium backend and a nodes: 0..2 fleet. gpuhunt came from lium-provider at a405412, which includes both resource fixes. Everything below ran on real 1-GPU pods.

What worked

• Task on python:3.11-slim. On an RTX 3090 at $0.24/h it was RUNNING about 70 s after submit and exited 0. Inside the pod the limits matched the offer:
  • memory.max = 20 GiB (offer mem=20GB)
  • cpu.max = 10 CPUs (offer cpu=10)
  • / had 37 GB free (offer disk=30GB)
• PyTorch task on pytorch/pytorch:2.4.0-cuda12.1-cudnn9-runtime. It ran an fp16 matmul on the GPU and exited 0. Provisioning to BUSY took about 3m20s, mostly the image pull.
• Dev environment. It reached RUNNING, dstack attach worked, and ssh <run> reached the GPU. The working dir was /root. dstack stop deleted the pod and its template.
• Cancel during provisioning. We stopped a task about 30 s in, while the pod was still being created. Our account log shows the delete requested, the in-flight create then failing, and the pod reaching DELETED. No paid pod was left behind.
• Cleanup. After every run, Lium's API showed no leftover pods or private templates.

Observations

1. Default disk filters out cheap offers. dstack requests disk=100GB.. by default. Now that disk_size is only the container storage quota, that default cut 1-GPU offers from 16 to 10 at the time of the test, and dropped the cheapest one, the 30 GB 3090. The backend mounts the other 2/3 of the share as the /root volume. For dev environments that is also the working dir, so they had more room than advertised (69 GB on /root here). Lower bounds are the right default. We could also look on the Lium side at giving container storage a larger part of the share for templates that don't need /root persistence, if that would help dstack.
2. "Lost run connection". Two attached dstack apply runs ended with "Lost run connection. Timed out waiting for run final status" even though both exited 0. In the torch run the job hit TERMINATING (DONE_BY_RUNNER) at 03:17:53 and the run reached DONE at 03:18:15, about 4 s after the CLI's 30 s wait ran out. This looks like local server pacing rather than the backend. We mention it in case run status should not wait on the instance going BUSY → TERMINATING.
3. Not covered: we did not exercise private images, non-root images or volumes, since your E2E covered those.

We didn't find anything blocking on our side. Happy to try a specific scenario if you'd like.