For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).

Mojo function

batched_matmul_dynamic_scaled_fp8

def batched_matmul_dynamic_scaled_fp8[c_type: DType, a_type: DType, b_type: DType, a_scales_type: DType, b_scales_type: DType, //, input_scale_granularity: StringSlice[ImmStaticOrigin], weight_scale_granularity: StringSlice[ImmStaticOrigin], m_scale_granularity: Int, n_scale_granularity: Int, k_scale_granularity: Int, transpose_b: Bool = False, target: StringSlice[ImmStaticOrigin] = StringSlice("cpu")](c: TileTensor[c_type, Storage=c.Storage, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[a_type, Storage=a.Storage, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[b_type, Storage=b.Storage, address_space=b.address_space, linear_idx_type=b.linear_idx_type], a_scales: TileTensor[a_scales_type, Storage=a_scales.Storage, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type], b_scales: TileTensor[b_scales_type, Storage=b_scales.Storage, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type], ctx: DeviceContext)

Dispatches a batched blockwise scaled FP8 matrix multiplication to the SM100 blockwise kernel on Blackwell hardware or falls back to the naive per-batch implementation on H100.

Parameters:

c_type (DType): Output tensor element dtype.
a_type (DType): LHS input tensor element dtype.
b_type (DType): RHS input tensor element dtype.
a_scales_type (DType): LHS scales tensor element dtype.
b_scales_type (DType): RHS scales tensor element dtype.
input_scale_granularity (StringSlice[ImmStaticOrigin]): Scale granularity mode for the LHS input (only "block" is supported).
weight_scale_granularity (StringSlice[ImmStaticOrigin]): Scale granularity mode for the RHS input (only "block" is supported).
m_scale_granularity (Int): M-direction scale granularity (must be 1).
n_scale_granularity (Int): N-direction scale granularity (64 or 128).
k_scale_granularity (Int): K-direction scale granularity (64 or 128).
transpose_b (Bool): Whether the RHS input is transposed.
target (StringSlice[ImmStaticOrigin]): Target platform string.

Args:

c (TileTensor[c_type, Storage=c.Storage, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): Rank-3 output tensor of shape (batch, m, n).
a (TileTensor[a_type, Storage=a.Storage, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): Rank-3 LHS input tensor of shape (batch, m, k).
b (TileTensor[b_type, Storage=b.Storage, address_space=b.address_space, linear_idx_type=b.linear_idx_type]): Rank-3 RHS input tensor of shape (batch, k, n).
a_scales (TileTensor[a_scales_type, Storage=a_scales.Storage, address_space=a_scales.address_space, linear_idx_type=a_scales.linear_idx_type]): Rank-3 LHS scales tensor.
b_scales (TileTensor[b_scales_type, Storage=b_scales.Storage, address_space=b_scales.address_space, linear_idx_type=b_scales.linear_idx_type]): Rank-3 RHS scales tensor.
ctx (DeviceContext): Device context used to dispatch the kernel.