[MSCCLPP-EP] Add C++ latency expert-parallel runtime by Binyang2014 · Pull Request #895 · microsoft/mscclpp

Binyang Li (Binyang2014) · 2026-09-03T00:49:20Z

Summary

add the public C++ MoERuntime API and latency dispatch/combine request types
add the current low-latency expert-major and rank-major CUDA dispatch/combine kernels
support BF16 and FP8 dispatch plus rank-local-reduce and direct-send combine modes
build the implementation as a standalone mscclpp_ep shared library with no Python dependency
add 8-GPU SM90+ C++ initialization, correctness, and CUDA-graph performance tests

Scope

empty ThroughputDispatchRequest and ThroughputCombineRequest types reserve names for the follow-up implementation
throughput request fields, prepare, notify, host runtime, and CUDA kernels are intentionally excluded
all Python bindings, wrappers, benchmarks, and tests are intentionally excluded for follow-up PRs

C++ test coverage

runtime initialization and unsupported throughput-mode validation
expert-major BF16 and FP8 with rank-local-reduce and direct-send combine
rank-major BF16 with rank-local-reduce and direct-send combine
CUDA-graph D+C performance for expert-major and rank-major, 32 tokens/rank, hidden 7168, 8 experts, top-8
tests skip unless launched with exactly eight local GPUs at SM90 or newer

Validation

./tools/lint.sh
C++-only mscclpp_ep builds for MSCCLPP_GPU_ARCHS=90 and native
Python-enabled configure builds both mscclpp_ep and mscclpp_py
external C++ consumer compiles and links against the installed public API
install contains libmscclpp_ep.so and the public EP headers
mpirun -np 8 ./build-ep-tests/bin/mp_unit_tests --filter=MoERuntimeTest: 4/4 passed
graph D+C: expert-major 51.08 us, rank-major 40.95 us

Feature/EP regression comparison

identical 8-GPU C++ harness, hidden 7168, 32 tokens/rank, BF16, rank-local-reduce, 50 D+C pairs/graph, 100 replays
split branch averages: expert-major 50.47 us, rank-major 40.75 us
feature/ep averages: expert-major 50.58 us, rank-major 41.21 us
six dispatch/combine correctness combinations pass on both branches

Add the public MoERuntime request API, latency dispatch/combine host runtime, and expert-major and rank-major CUDA kernels as a standalone mscclpp_ep library that does not depend on Python bindings. Preserve the unified throughput-facing API surface while rejecting throughput mode until its follow-up implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-09-03T00:49:29Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Callers can derive the topology distinction from isAvailable(), numRanks(), and numNvlRanks(); keep the initial C++ latency surface focused on the capability check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove throughput-only prepare and notify methods and strip throughput request fields while retaining empty request types as placeholders for the follow-up implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace the duplicate EP CUDA_CHECK macro with MSCCLPP_CUDATHROW and remove the unused throughput-only extended launch helper from the latency implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove unused device helpers, align helper naming and inline annotations with MSCCL++ conventions, cap latency routing at top-8, and keep rank-major combine on its TMA path without the unreachable fallback state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Cover latency runtime initialization, unsupported throughput requests, expert-major and rank-major dispatch/combine correctness across BF16 and FP8, and CUDA-graph D+C performance for 32 tokens per rank. Remove dead and redundant runtime state found during the kernel review. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use 100 graph replays to reduce variance in the expert-major and rank-major 32-token D+C measurements. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rename the mode-owned host context types to LatencyRuntimeContext and ThroughputRuntimeContext, and clarify the combine input buffer documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-09-03T16:55:44Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Rename the leader-only destination variable from completionRank to dstRank so the rank-major token-store path is easier to follow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Binyang Li (Binyang2014) · 2026-09-03T17:38:49Z

/azp run mscclpp-ut

azure-pipelines · 2026-09-03T17:39:05Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Keep dispatch metadata opaque, derive fixed request configuration from the runtime, and use EP-local exceptions with caller-managed GPU lifetimes. Update EP tests for randomized top-8 routing with 16 experts per GPU and weighted rank-major benchmark inputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Caio Rocha (caiocbr) · 2026-09-16T03:28:27Z

+  MoEMode mode_;
+  int rank_;
+  int numRanks_;
+  int numNvlRanks_;


Could we rename numNvlRanks_ to numRanksPerNode_? It is derived from getNranksPerNode(), so the current name implies an NVLink topology property that is not actually established here. We could potentially remove this field instead, since the current latency implementation uses numRanksPerIpcDomain_ for its communication-domain and availability decisions.

Caio Rocha (caiocbr) · 2026-09-16T03:34:30Z

+  DispatchHandle launchLatencyDispatch(const LatencyDispatchRequest& request);
+  void launchLatencyCombine(const LatencyCombineRequest& request);
+
+  std::shared_ptr<mscclpp::Bootstrap> bootstrap_;


Since we pass the communicator through the constructor, wouldn’t it make sense to retain it directly? I realize that most of the time we essentially use only the bootstrap, but in other codebases we generally keep the communicator rather than retaining only the bootstrap.

Caio Rocha (caiocbr) · 2026-09-16T03:53:00Z

+///
+/// The caller must keep all buffers referenced by this struct (inputs and
+/// outputs) valid until the corresponding combine operation has finished.
+struct LatencyDispatchRequest {


Would it be clearer and safer to split this into layout-specific request types (for example, ExpertMajorDispatchRequest and RankMajorDispatchRequest) and make LatencyDispatchRequest a variant or shared/common portion? The current struct permits invalid combinations and relies on runtime assertions even though several fields are mutually exclusive:

Expert-major only: outputScales (FP8 only), outputSrcInfo, and outputLayoutRange.

Rank-major only: outputTopkIdx, outputTopkWeights, and invalidTokenExpertId.

Both layouts: output, outputCount, input, topkIdx, topkWeights, numTokens, maxTokensPerRank, dispatchDataType, numBlocks, and stream; however, outputCount has layout-dependent indexing, and FP8 dispatchDataType is currently supported only by expert-major.

Separate request types would make these requirements explicit in the type system, document the different output shapes, and avoid exposing fields that callers must set to nullptr for the selected layout.

Caio Rocha (caiocbr) · 2026-09-16T03:53:00Z

+  /// Requested dispatch payload format.
+  DispatchDataType dispatchDataType;
+  /// Dispatch grid block count.
+  int numBlocks;


Does numBlocks need to be part of every dispatch request? This looks like an implementation-specific tuning parameter whose valid and optimal value depends on the selected algorithm, device occupancy, rank count, and shared-memory use. Could the runtime/algorithm choose a standard value by default (for example, treating 0 as automatic), while retaining an optional override for benchmarking or advanced tuning? That would keep ordinary callers from needing to understand the kernel's control-block and all-block-residency constraints.

Caio Rocha (caiocbr) · 2026-09-16T03:53:00Z

+  /// Handle returned by the matching dispatch.
+  DispatchHandle handle;
+  /// Combine grid block count.
+  int numBlocks;


Similarly, should combine block count default to an algorithm-selected value rather than being mandatory in every request? A default such as 0 meaning automatic would let the selected combine implementation derive a safe value from occupancy, layout, and device properties, while still allowing an explicit override for performance tuning.

Caio Rocha (caiocbr) · 2026-09-16T05:01:20Z

+namespace mscclpp {
+namespace ep {
+
+constexpr int DispatchNWarps = 16;


Could you clarify why dispatch uses 16 warps (512 threads) rather than the maximum 32 warps (1024 threads) per block? Is this an intentional performance choice based on benchmarking—for example, to balance register/shared-memory pressure, block residency, or TMA throughput? Using fewer threads can certainly perform better here, but it would be helpful to document the rationale and which workload or GPU configurations were used to select 16.

Caio Rocha (caiocbr) · 2026-09-16T05:01:20Z

+#define EP_STATIC_ASSERT(cond, reason) static_assert(cond, reason)
+#endif
+
+class EPException : public std::exception {


Do we need an EP-specific exception type and this separate exception header? MSCCL++ already has the public mscclpp::Error hierarchy with typed ErrorCode values, along with MSCCLPP_CUDATHROW, MSCCLPP_CUTHROW, and MSCCLPP_ASSERT_DEVICE. It may be more consistent to use mscclpp::Error(..., ErrorCode::InvalidUsage) for invalid requests and ErrorCode::InternalError for failed internal invariants, keeping only minimal EP helper macros if needed. This would also avoid the current public-API mismatch where public methods document EPException, but that type is defined only in this private src header and therefore cannot be caught by users through the installed public headers.

Caio Rocha (caiocbr) · 2026-09-16T05:01:20Z

+    EP_HOST_ASSERT(outputTopkWeights == allocationLayout.rankMajorTopkWeightsBuffer_);
+  }
+
+  const Workload workload{.epoch_ = context.epoch_ + 1,


Should we update the context epoch here—or perhaps use an atomic add—to prevent two kernels from being called with the same epoch if one don't update before the other arrives?

Binyang Li (Binyang2014) marked this pull request as draft September 3, 2026 02:41

Binyang Li (Binyang2014) and others added 4 commits September 3, 2026 02:49

Remove redundant EP internode availability API

fd153f6

Callers can derive the topology distinction from isAvailable(), numRanks(), and numNvlRanks(); keep the initial C++ latency surface focused on the capability check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Limit initial EP API to latency mode

2b3208b

Remove throughput-only prepare and notify methods and strip throughput request fields while retaining empty request types as placeholders for the follow-up implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Reuse MSCCL++ CUDA error handling in EP

ff69416

Replace the duplicate EP CUDA_CHECK macro with MSCCLPP_CUDATHROW and remove the unused throughput-only extended launch helper from the latency implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Binyang Li (Binyang2014) changed the title ~~Add C++ latency expert-parallel runtime~~ [MSCCLPP-EP] Add C++ latency expert-parallel runtime Sep 3, 2026

Binyang Li (Binyang2014) and others added 3 commits September 3, 2026 05:41

Increase EP graph benchmark iterations

72a9570

Use 100 graph replays to reduce variance in the expert-major and rank-major 32-token D+C measurements. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Clarify EP runtime context naming

a910c1e

Rename the mode-owned host context types to LatencyRuntimeContext and ThroughputRuntimeContext, and clarify the combine input buffer documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Binyang Li (Binyang2014) marked this pull request as ready for review September 3, 2026 16:55

Binyang Li (Binyang2014) requested a review from a team September 3, 2026 16:56

Clarify rank-major store destination

49075b5

Rename the leader-only destination variable from completionRank to dstRank so the rank-major token-store path is easier to follow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Mahdieh Ghazi (mahdiehghazim) reviewed Sep 4, 2026

View reviewed changes

Binyang Li (Binyang2014) and others added 2 commits September 8, 2026 09:49

Merge branch 'main' into binyli/ep-latency-cpp

766e326

Binyang Li (Binyang2014) requested review from Caio Rocha (caiocbr), Changho Hwang (chhwang) and Qinghua Zhou (seagater) September 8, 2026 23:47

Mahdieh Ghazi (mahdiehghazim) reviewed Sep 9, 2026

View reviewed changes

Comment thread include/mscclpp/ext/ep/types.hpp

Comment thread include/mscclpp/ext/ep/types.hpp

Binyang Li (Binyang2014) added 2 commits September 10, 2026 22:25

WIP

9e09d3a

Merge branch 'main' into binyli/ep-latency-cpp

1dad926

Mahdieh Ghazi (mahdiehghazim) approved these changes Sep 14, 2026

View reviewed changes

Caio Rocha (caiocbr) reviewed Sep 16, 2026

View reviewed changes

Uh oh!

Conversation

Binyang Li (Binyang2014) commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope

C++ test coverage

Validation

Feature/EP regression comparison

Uh oh!

azure-pipelines Bot commented Sep 3, 2026

Uh oh!

azure-pipelines Bot commented Sep 3, 2026

Uh oh!

Binyang Li (Binyang2014) commented Sep 3, 2026

Uh oh!

azure-pipelines Bot commented Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Binyang Li (Binyang2014) commented Sep 3, 2026 •

edited

Loading