For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Mojo struct
DistributedReduceScatterSum
struct DistributedReduceScatterSum
Implemented traits
AnyType,
ImplicitlyDestructible
Methods
execute
static def execute[dtype: DType, rank: Int, target: StringSlice[StaticConstantOrigin], _trace_name: StringSlice[StaticConstantOrigin], axis: Int = 0](outputs: _FusedOutputVariadicTensors[static_specs=outputs.static_specs], inputs: VariadicTensors[Input, static_specs=inputs.static_specs], signal_buffers: VariadicTensors[MutableInput, static_specs=signal_buffers.static_specs], dev_ctxs_input: DeviceContextList)
Distributed reduce-scatter operation implementation for sum reduction.
Limitations: - Maximum of 8 GPUs supported (matches MAX_GPUS in comm/sync.mojo) - Tensor element count must be multiple of SIMD width - Requires identical tensor shapes across all participating GPUs
Args:
- outputs (
_FusedOutputVariadicTensors[static_specs=outputs.static_specs]): Output tensors (one per GPU) to store scattered reduced results. - inputs (
VariadicTensors[Input, static_specs=inputs.static_specs]): Input tensors (one per GPU) containing values to reduce. - signal_buffers (
VariadicTensors[MutableInput, static_specs=signal_buffers.static_specs]): Preallocated synchronization buffers for cross-GPU coordination. - dev_ctxs_input (
DeviceContextList): Device contexts for participating GPUs.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!