For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Mojo struct
DistributedAllGatherRMSNorm
struct DistributedAllGatherRMSNorm
Registers the mo.composite.distributed.allgather_rms_norm graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[dtype: DType, rank: Int, target: StringSlice[ImmStaticOrigin], _trace_name: StringSlice[ImmStaticOrigin]](outputs_normed: VariadicTensors[IOSpec[_, _].Output, static_specs=outputs_normed.static_specs], outputs_residual: VariadicTensors[IOSpec[_, _].Output, static_specs=outputs_residual.static_specs], inputs: VariadicTensors[IOSpec[_, _].Input, static_specs=inputs.static_specs], signal_buffers: VariadicTensors[IOSpec[_, _].MutableInput, static_specs=signal_buffers.static_specs], gammas: VariadicTensors[IOSpec[_, _].Input, static_specs=gammas.static_specs], epsilons: VariadicTensors[IOSpec[_, _].Input, static_specs=epsilons.static_specs], weight_offsets: VariadicTensors[IOSpec[_, _].Input, static_specs=weight_offsets.static_specs], dev_ctxs_input: DeviceContextArray)
Fused all-gather + RMSNorm (bf16 in/out, no quantization).
All-gathers inputs (per-device [shard_i, cols] row-shards) into the
full replicated [rows, cols] stream and RMSNorms every gathered row in
one launch: normed to outputs_normed, gathered residual to
outputs_residual (both full and replicated).
Limitations: - Maximum of 8 GPUs supported (matches MAX_GPUS in comm/sync.mojo). - Full-world all-gather only (no device grouping); requires P2P.
Parameters:
- dtype (
DType): Element type of the input/output tensors. - rank (
Int): Tensor rank of the inputs and outputs. - target (
StringSlice[ImmStaticOrigin]): Target device string for tracing. - _trace_name (
StringSlice[ImmStaticOrigin]): Trace name for profiling.
Args:
- outputs_normed (
VariadicTensors[IOSpec[_, _].Output, static_specs=outputs_normed.static_specs]): Per-device normed output (full replicated[rows, cols]). - outputs_residual (
VariadicTensors[IOSpec[_, _].Output, static_specs=outputs_residual.static_specs]): Per-device gathered residual (full replicated, the residual stream). - inputs (
VariadicTensors[IOSpec[_, _].Input, static_specs=inputs.static_specs]): Per-device input row-shards to gather. - signal_buffers (
VariadicTensors[IOSpec[_, _].MutableInput, static_specs=signal_buffers.static_specs]): Per-device synchronization buffers. - gammas (
VariadicTensors[IOSpec[_, _].Input, static_specs=gammas.static_specs]): Per-device RMSNorm gamma weights (in_dtype, length cols). - epsilons (
VariadicTensors[IOSpec[_, _].Input, static_specs=epsilons.static_specs]): Per-device RMSNorm epsilon scalars (float32). - weight_offsets (
VariadicTensors[IOSpec[_, _].Input, static_specs=weight_offsets.static_specs]): Per-device gamma offset scalars (in_dtype). - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!