[Bug] ALLREDUCE assert failed

[zhangandy0727-jpg]

torchrun --nproc_per_node=8 --master_port=29503 test/torch/correctness_test.py --collective allreduce --nelem 10556587 --dtype float

github.com/microsoft/mscclpp/apps/nccl/src/allreduce.hpp:679: void allreduce11(const void *, void *, void *, mscclpp::BaseMemoryChannelDeviceHandle *, mscclpp::SwitchChannelDeviceHandle *, unsigned long, unsigned long, int, int) [with T = float]: block: [31,0,0], thread: [943,0,0] Assertion sizePerRank % alignment == 0 failed.

any plan to provide a fallback version of allreduce ?

[Binyang2014]

Thanks for reporting this, will take a look for this issue

[seagater]

Hi zhangandy0727-jpg, while we are debugging this issue, you may try the fallback to nccl allreduce:

Here is an example of the runtime parameters to fallback to nccl allreduce.
mpirun -np 8 --bind-to numa --allow-run-as-root -x LD_PRELOAD=/root/mscclpp/build/apps/nccl/libmscclpp_nccl.so -x NCCL_DEBUG=WARN -x MSCCLPP_ENABLE_NCCL_FALLBACK=TRUE -x MSCCLPP_NCCL_LIB_PATH=/root/nccl/build/lib/libnccl.so -x MSCCLPP_FORCE_NCCL_FALLBACK_OPERATION="allreduce"

You can also find some usage examples in our .azure-pipeline/templates/nccl-test.yaml:
https://github.com/microsoft/mscclpp/blob/main/.azure-pipelines/templates/nccl-test.yaml#L223C9-L223C285

[seagater]

The issue could be related to the misaligned size 10556587 for nvls algorithm. We will fix it soon.

[mahdiehghazim]

Fixed by #697. The assertion for misaligned element counts in the NVLS AllReduce kernel has been resolved.

Closing this issue. Please reopen if you still see the problem with the latest MSCCL++ release.