For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).

Mojo function

enqueue_apple_fp4_gemv

def enqueue_apple_fp4_gemv[c_type: DType = DType.float32, elementwise_lambda_fn: Optional[def[dtype: DType, width: SIMDLength, *, alignment: Int = Int(1)](IndexList[Int(2)], SIMD[dtype, width]) capturing thin -> None] = None](c: TileTensor[c_type, Storage=c.Storage, address_space=c.address_space, linear_idx_type=c.linear_idx_type], a: TileTensor[DType.bfloat16, Storage=a.Storage, address_space=a.address_space, linear_idx_type=a.linear_idx_type], packed: TileTensor[DType.uint8, Storage=packed.Storage, address_space=packed.address_space, linear_idx_type=packed.linear_idx_type], scales: TileTensor[DType.float8_e4m3fn, Storage=scales.Storage, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type], n: Int, k: Int, ctx: DeviceContext)

Enqueue the M=1 W4A16 GEMV: out = a @ dequant(packed, scales)^T.

One warp per output column N (ceil(N*WARP_SIZE / block_dim) threadgroups of block_dim threads). a is the bf16 activation [1, K], packed the FP4 weight [N, K//2] (lo-nibble first), scales the FP8-E4M3 block scales [N, ceil(K/16)]. Assumes M == 1 and K % NVFP4_SF_VECTOR_SIZE == 0 (true for every NVFP4-quantized Linear); the caller (enqueue_apple_fp4_matmul) gates the M == 1 route.

Parameters:

c_type (DType): Output element type (fp16, bf16, fp32). Accumulation is fp32.
elementwise_lambda_fn (Optional[def[dtype: DType, width: SIMDLength, *, alignment: Int = Int(1)](IndexList[Int(2)], SIMD[dtype, width]) capturing thin -> None]): Optional fused epilogue (AMD's (row, col) contract), applied on the width-1 store.

Args:

c (TileTensor[c_type, Storage=c.Storage, address_space=c.address_space, linear_idx_type=c.linear_idx_type]): Output tile tensor [1, N] receiving the GEMV result.
a (TileTensor[DType.bfloat16, Storage=a.Storage, address_space=a.address_space, linear_idx_type=a.linear_idx_type]): Bf16 activation tile tensor [1, K], the single activation row.
packed (TileTensor[DType.uint8, Storage=packed.Storage, address_space=packed.address_space, linear_idx_type=packed.linear_idx_type]): FP4-packed weight tile tensor [N, K//2] (lo-nibble first).
scales (TileTensor[DType.float8_e4m3fn, Storage=scales.Storage, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type]): FP8-E4M3 block scales tile tensor [N, ceil(K/16)].
n (Int): Number of output columns (rows of the transposed weight).
k (Int): Inner dimension length; must be a multiple of NVFP4_SF_VECTOR_SIZE (16).
ctx (DeviceContext): DeviceContext used to enqueue the kernel on the device.