[feat] introduce payload transfer abstraction and NIXL-UCX backend

[Ginray]

Motivation

SimpleStorage currently couples payload transfer with its ZMQ request path, which makes it difficult to add alternative high-throughput transports without changing storage routing and public APIs.

This PR introduces a small payload-transfer abstraction so that SimpleStorage can keep ZMQ as its control plane while allowing Host-to-Host payloads to use alternative transports such as NIXL-UCX.

The goals are to:

• keep existing SimpleStorage APIs and default ZMQ behavior compatible;
• make the payload transfer backend independently replaceable;
• improve large-payload transfer performance without redesigning the storage layer.

Summary

• Add PayloadTransfer as the payload backend interface for SimpleStorage.
• Keep the existing ZMQ payload path as the default backend.
• Add NIXL-UCX as an optional Host-to-Host payload backend.
• Keep storage routing independent from the payload data path while retaining ZMQ as the control plane.

Design

AsyncSimpleStorageManager selects the configured payload backend and delegates logical payload PUT/GET operations to PayloadTransfer.

AsyncSimpleStorageManager
        |
        | logical put/get
        v
PayloadTransfer
      /            \
ZmqPayloadTransfer  NixlPayloadTransfer

On the StorageUnit side, the selected PayloadTransfer handles backend-specific PUT/GET requests, while SimpleStorageUnit continues to own storage semantics, data access, CLEAR, metrics, and checkpoint operations.

For NIXL-UCX:

• ZMQ remains the control plane for prepare/commit/error coordination.
• Non-empty payload data is transferred through NIXL-UCX.
• Backend-specific endpoint and transfer metadata remain internal to the NIXL implementation.

Usage

The default remains ZMQ.

To enable NIXL-UCX:

backend:
  storage_backend: SimpleStorage
  SimpleStorage:
    payload_transfer:
      backend: nixl-ucx
      ucx_env_vars: {}

See docs/nixl_ucx_payload.md for installation, UCX configuration, and RDMA verification.

Compatibility

• ZMQ remains the default backend.
• Existing SimpleStorage PUT/GET/CLEAR callers and public APIs remain unchanged.
• The existing ZMQ wire protocol and external behavior remain unchanged.
• NIXL-UCX is opt-in.
• NIXL transfer failures are reported directly and do not silently fall back to ZMQ.

Validation

Two-node A2 SimpleStorage tests covered 33 payload sizes from 128 KiB to 1 GiB. All PUT/GET/CLEAR operations and data checks passed.

Native UCX is shown only as a benchmark reference and is not part of this PR.

Payload	ZMQ	Native UCX	NIXL-UCX
128 KiB	10.23 / 10.45	26.04 / 27.63	27.95 / 26.06
1 MiB	51.73 / 51.88	72.23 / 72.35	79.22 / 77.70
16 MiB	104.47 / 104.33	106.75 / 106.65	106.16 / 106.04
1 GiB	112.09 / 111.92	110.30 / 110.29	109.17 / 109.18

Median MiB/s, PUT / GET.

The fixed real DAPO payload produced identical digests and training inputs through ZMQ and NIXL-UCX. Both paths completed 20/20 VIME steps on 80 real DAPO problems without OOM or malformed output.

Follow-up

• Perform broader correctness, stability, and performance validation across larger-scale and sustained multi-node workloads.
• Add user-facing tutorials and complete the related feature documentation for the NIXL-UCX payload path.
• Prepare the feature for integration into main once it is considered stable.