For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Mojo struct
DistributedAllGather
struct DistributedAllGather
Registers the mo.distributed.allgather graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[dtype: DType, rank: Int, target: StringSlice[ImmStaticOrigin], _trace_name: StringSlice[ImmStaticOrigin], group_size: Int = Int(0)](outputs: VariadicTensors[IOSpec[_, _].Output, static_specs=outputs.static_specs], inputs: VariadicTensors[IOSpec[_, _].Input, static_specs=inputs.static_specs], signal_buffers: VariadicTensors[IOSpec[_, _].MutableInput, static_specs=signal_buffers.static_specs], dev_ctxs_input: DeviceContextArray)
Distributed allgather operation implementation.
Parameters:
- dtype (
DType): Element type of the input and output tensors. - rank (
Int): Tensor rank (number of dimensions) of the inputs and outputs. - target (
StringSlice[ImmStaticOrigin]): Target device string for tracing. - _trace_name (
StringSlice[ImmStaticOrigin]): Trace name for profiling. - group_size (
Int): Number of devices per allgather group; must be at least 1 and must evenly divide the total number of devices (defaults to 0).
Args:
- outputs (
VariadicTensors[IOSpec[_, _].Output, static_specs=outputs.static_specs]): Output tensors (one per GPU) to store gathered results. - inputs (
VariadicTensors[IOSpec[_, _].Input, static_specs=inputs.static_specs]): Input tensors (one per GPU) containing values to gather. - signal_buffers (
VariadicTensors[IOSpec[_, _].MutableInput, static_specs=signal_buffers.static_specs]): Device buffer values used for synchronization. - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!