[MSCCLPP-EP] C++ throughput expert-parallel runtime by Binyang2014 · Pull Request #903 · microsoft/mscclpp

Binyang Li (Binyang2014) · 2026-09-10T23:14:58Z

Summary

Adds throughput-mode expert-parallel dispatch/combine to the C++ MoERuntime, alongside the existing latency path. This PR targets binyli/ep-latency-cpp.

Implementation

GPU-only preparation: prepare(PrepareRequest) returns an opaque, reusable PrepareHandle. Counts, stable receive offsets, and per-token destination maps stay on the GPU. Dispatch prepares automatically when prepareHandle is empty, or reuses an explicit preparation while routing remains unchanged.
Unified token-based dispatch: the existing count/scan kernel builds rank-ordered {rank, offset} entries, deduplicating destinations without an extra kernel. Dispatch uses this map for every supported rank count, loads each input vector once, and fans it out to the selected peers. There is no RouteByTopk specialization or lane-ID-equals-rank-ID requirement.
Pipelined combine: reuses the same routing map to gather BF16 expert results through a two-stage TMA pipeline, accumulate in FP32, and write BF16 output using MSCCL++ vector types.
Configuration-sized storage: one registered symmetric allocation per rank plus a private workspace. Dispatch output and combine input share the receive buffer. TOKEN_MAJOR rows are dense, without gaps between source-rank batches; RANK_MAJOR retains explicit source-rank strides. Dispatch supports BF16 and FP8 E4M3, with optional expert-ID, weight, and scale outputs.
Shared launch and synchronization helpers: reuse IPC mappings and channel semaphores, with cooperative grid synchronization and common shared-memory/occupancy setup for latency and throughput.

API contract and scope

CUDA SM90 or newer; 1-64 ranks within one CUDA IPC domain; top-k in [1, 8]. This does not add a cross-IPC-domain/RDMA transport.
Operations enqueue asynchronously on one caller-provided stream, including preparation, dispatch, expert computation, and combine. Cross-stream overlap is not supported, and handles are not completion events. The caller must serialize host calls and complete local/peer GPU use before destroying the runtime.
Token input/output buffers must be 16-byte aligned. Active per-rank capacity may be smaller than the configured allocation capacity.

Validation and performance

The existing ThroughputCorrectness and ThroughputPerformance tests passed on eight H100 GPUs. Correctness includes both layouts and dispatch formats, automatic/explicit preparation, reuse and routing rebuilds, scan boundaries, duplicate/invalid routes, and non-power-of-two experts per rank. Preparation is covered by correctness rather than a separate test.

Performance measures eager dispatch + combine, using one initialized runtime across 1K/2K/4K tokens per rank, hidden size 7168, 128 experts, top-k 8, D/C block budgets of 24/32 and 130/128, BF16/FP8 dispatch, and automatic/cached preparation. Allocation, input conversion, and expert compute are outside timing; normal combine input staging is included.

The latest route-map simplification (3d7964d9 versus 6c8d9cc3, both within this PR) has the following measured D+C latency changes across the 24 profiles, comparing three baseline runs with two updated runs. Positive values mean slower:

Preparation	Per-profile change range	Median change
Automatic	+0.38% to +1.31%	+0.69%
Cached	-0.38% to +0.82%	-0.06%

Physical 32/64-GPU qualification has not been performed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-09-10T23:15:07Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Move the peer barrier into shared device helpers, update throughput call sites, and remove the redundant warp synchronization between signaling and waiting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace the custom launch wrapper with direct CUDA launches and remove rank-count kernel specializations while preserving cooperative launches and contributor/warp tuning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Move receive-pool layout calculations into config.hpp and update runtime allocation and buffer accessors to use the new storage view without changing capacities or offsets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Expose plain preparation handles without host count readbacks and preserve automatic preparation for existing dispatch callers. Unify symmetric storage and peer mappings, reuse latency-style workload and launch helpers, and clarify payload and workspace layouts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Size throughput buffers from runtime configuration and keep preparation handles opaque. Fuse stable token offsets into route counting, remove redundant preparation work, and tune token groups and BF16/FP8 metadata paths. Clarify latency payload and combine-buffer naming. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove the byte-copy fallback and reject unaligned token input/output buffers before launching GPU work. Document the alignment contract, update native coverage, and retain combine provenance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use bf16x8/f32x8 conversions while preserving FP32 accumulation and int4 transfers. Consolidate throughput coverage into correctness, prepare and performance entry points, and benchmark eager dispatch with one initialized runtime across all profiles. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Move count exchange into throughput_prepare.cu without changing its launch sequence. Measure eager dispatch-plus-combine round trips using one registered runtime, with BF16 expert-result setup outside timing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep only throughput correctness and eager dispatch-plus-combine performance entry points. Cover explicit preparation in the correctness path and remove the separate preparation checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Generate compact, rank-ordered token routes in the existing prepare kernel. Share the map between dispatch and combine, removing RouteByTopk and rank-count-specific dispatch selection without changing receive placement or combine reduction order. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-09-15T19:00:06Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Qinghua Zhou (seagater) · 2026-09-15T20:51:28Z

+  EP_HOST_ASSERT(maxTokensPerRank_ > 0);
+  EP_HOST_ASSERT(static_cast<int64_t>(hidden_) * sizeof(Bf16) % sizeof(int4) == 0);
+
+  if (!isSupportedRanks(numRanks_) || numRanksPerIpcDomain_ < numRanks_) {


Should we also check cudaDevAttrComputeCapabilityMajor >= 9, and EP_THROW a clear message? The dynamic shared request also exceeds MaxSharedMemoryPerBlockOptin in lower arch like sm_80.

I think during compilation, we force to use sm>=90. So the code will no run on arch like sm_80

Qinghua Zhou (seagater) · 2026-09-15T20:55:03Z

+      contributorSlots[contributor] = warpBroadcast(slot, contributor);
+    }
+
+    auto* outputRow = output + static_cast<int64_t>(token) * hiddenInt4;


combine output requires 16-byte alignment, should we check the 16-byte alignment for combine output and dispatch input?

Binyang Li (Binyang2014) · 2026-09-18T17:40:29Z

Copilot pls help me resolve the conflict

Co-authored-by: Binyang2014 <9415966+Binyang2014@users.noreply.github.com>

Copilot · 2026-09-18T17:48:06Z

Copilot pls help me resolve the conflict

Binyang Li (Binyang2014) · 2026-09-18T23:17:24Z

+  ThroughputWorkspaceLayout(void* workspace, int maxTokensPerRank, int numRanks, int numExperts, int numTopk) {
+    size_t offset = 0;
+    auto place = [&](size_t bytes, size_t alignment) -> void* {
+      offset = configAlign<size_t>(offset, alignment);
+      void* ptr = workspace == nullptr ? nullptr : reinterpret_cast<uint8_t*>(workspace) + offset;
+      offset += bytes;
+      return ptr;
+    };


What's the difference between playload layout and workspace layout

Binyang Li (Binyang2014) · 2026-09-18T23:21:40Z

+}
+
+Workload ThroughputRuntimeContext::makeWorkload(int numTokens, int maxTokensPerRank, DispatchDataType dataType) const {
+  return {.epoch_ = 0,


epoch start from1?

Binyang Li (Binyang2014) · 2026-09-18T23:22:51Z

+                                                  context.numExperts_, context.numTopk_);
+  if (reusePreparation) {
+    // Replays still need a peer handshake before overwriting the previous payload.
+    throughputSynchronizePeers(context.deviceContext_, request.stream);


What's this

Binyang Li (Binyang2014) · 2026-09-18T23:48:51Z

Copilot resolve the merge conflicts in this pull request

Co-authored-by: Binyang2014 <9415966+Binyang2014@users.noreply.github.com>

Copilot · 2026-09-18T23:54:20Z

Copilot resolve the merge conflicts in this pull request

已在 9ca96a0 合并最新 binyli/ep-latency-cpp 并推送，冲突已消除。保留基分支的 latency epoch 更新及 throughput 原有行为，API 文档已区分两种模式的句柄生命周期。C++ lint、敏感信息扫描和 CodeQL 通过，独立审查未发现问题；环境缺少 CUDA/GPU，未能完成编译和 GPU 测试。

Distinguish peer-visible payload storage from private routing workspace and rename the metadata row stride. Default latency-only epoch fields so throughput does not initialize or track unused packet/dispatch generations. Document strict dispatch/combine ordering and explain the peer handshake needed when reusable preparation skips the collective prepare phase. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restore C++ throughput expert-parallel runtime

0dbb99f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Binyang Li (Binyang2014) and others added 5 commits September 10, 2026 23:25

Add throughput EP correctness test

ccd357c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge branch 'binyli/ep-latency-cpp' into binyli/ep-throughput-cpp

4b9def7

Simplify EP barrier synchronization

4803979

Move the peer barrier into shared device helpers, update throughput call sites, and remove the redundant warp synchronization between signaling and waiting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Binyang Li (Binyang2014) commented Sep 11, 2026

View reviewed changes

Comment thread src/ext/ep/include/config.hpp Outdated

Binyang Li (Binyang2014) commented Sep 11, 2026

View reviewed changes

Comment thread src/ext/ep/include/config.hpp Outdated

Binyang Li (Binyang2014) commented Sep 11, 2026

View reviewed changes

Comment thread src/ext/ep/include/config.hpp Outdated

Binyang Li (Binyang2014) commented Sep 11, 2026

View reviewed changes

Comment thread src/ext/ep/include/config.hpp Outdated

Binyang Li (Binyang2014) commented Sep 11, 2026

View reviewed changes

Comment thread src/ext/ep/include/device_context.hpp Outdated

Binyang Li (Binyang2014) commented Sep 11, 2026

View reviewed changes

Comment thread src/ext/ep/include/device_context.hpp Outdated

Binyang Li (Binyang2014) commented Sep 11, 2026

View reviewed changes

Comment thread src/ext/ep/include/moe_runtime_context.hpp Outdated