For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).

Mojo function

enqueue_apple_int8_matmul

def enqueue_apple_int8_matmul[c_type: DType = DType.bfloat16, *, has_bias: Bool = False](c: TileTensor[c_type, Storage=c.Storage, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[DType.int8, Storage=a.Storage, address_space=a.address_space, linear_idx_type=a.linear_idx_type], b: TileTensor[DType.int8, Storage=b.Storage, address_space=b.address_space, linear_idx_type=b.linear_idx_type], a_scale: TileTensor[DType.float32, Storage=a_scale.Storage, address_space=a_scale.address_space, linear_idx_type=a_scale.linear_idx_type], b_scale: TileTensor[DType.float32, Storage=b_scale.Storage, address_space=b_scale.address_space, linear_idx_type=b_scale.linear_idx_type], bias: TileTensor[c_type, Storage=bias.Storage, address_space=bias.address_space, linear_idx_type=bias.linear_idx_type], ctx: DeviceContext, _use_i32_override: Optional[Bool] = None)

Enqueue the W8A8 matmul: out = dequant(a @ b^T).

a is int8 (M, K), b int8 (N, K) (transpose_b), a_scale (M,) per-row, b_scale (N,) per-column, bias (N,) in c_type (used iff has_bias; pass a length-1 dummy otherwise). C is (M, N).

Parameters:

c_type (DType): Output element type (bf16 / fp16 / fp32).
has_bias (Bool): If True, add bias[col] after dequant.

Args:

c (TileTensor[c_type, Storage=c.Storage, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): Output TileTensor of shape (M, N) in c_type, written mutable.
a (TileTensor[DType.int8, Storage=a.Storage, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): Int8 activation TileTensor of shape (M, K).
b (TileTensor[DType.int8, Storage=b.Storage, address_space=b.address_space, linear_idx_type=b.linear_idx_type]): Int8 weight TileTensor of shape (N, K) (used transposed).
a_scale (TileTensor[DType.float32, Storage=a_scale.Storage, address_space=a_scale.address_space, linear_idx_type=a_scale.linear_idx_type]): Per-row fp32 activation scale, shape (M,).
b_scale (TileTensor[DType.float32, Storage=b_scale.Storage, address_space=b_scale.address_space, linear_idx_type=b_scale.linear_idx_type]): Per-column fp32 weight scale, shape (N,).
bias (TileTensor[c_type, Storage=bias.Storage, address_space=bias.address_space, linear_idx_type=bias.linear_idx_type]): Per-column bias in c_type, shape (N,) (used iff has_bias; pass a length-1 dummy otherwise).
ctx (DeviceContext): Device context used to enqueue the kernel.
_use_i32_override (Optional[Bool]): Optional internal override forcing (True) or disabling (False) the int32 accumulation path; auto-selected when None.

Raises:

If the attached GPU is not Apple M5 (compute_capability == 5).