For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).

Mojo module

fp4_gemv

Apple M5 weight-only NVFP4 (W4A16) GEMV: out = x @ dequant(W)^T at M=1.

Apple silicon GPU (Metal 4, compute_capability == 5). The batch-1 decode regime (M == 1) is a matrix-vector product, NOT a matrix-matrix product: there is exactly one activation row, so there is no MMA to feed. Routing it through the prefill cooperative-SMEM MMA path (AppleM5Fp4MatMul) wastes the simdgroup matrix unit on a rank-1 update and is bandwidth-inefficient. This kernel is the GEMV instead -- register-resident, no SMEM, no barrier(), no _mma_apple.

WHY it wins at decode: Llama-8B batch-1 decode is weight-read-bandwidth-bound (the profiling campaign measured ~540 GB/s, ~88% of the M5 ~614 GB/s peak, on the bf16 path). W4A16 reads ~1/4 the weight bytes (packed nibbles + a small fp8 scale stream vs a full bf16 weight), so the wall-clock drops roughly with the byte count -- measured 1.53x over a bf16 GEMV at the down-proj shape (N=4096, K=14336) on M5 Max. The dequant-to-bf16 gives NO compute speedup (there is no MMA here); the entire win is reading fewer bytes. See Kernels/claude_kb/entries/kernels/apple-m5-fp4-matmul.md and patterns/apple-m5-quantized-matmul-ceiling (the co-issue penalty applies only at core saturation -- the GEMV is under-occupancy, so it does not bite here).

Structure (mirrors linalg/gemv.mojo::gemv_kernel_vector, B-load replaced by an inline FP4 decode): one warp owns one output column n (= one row of the transpose_b weight W[N, K]); its 32 lanes stride down K, each lane owning whole 16-col FP8 scale blocks (strided by WARP_SIZE blocks). Per block: one coalesced width-8 packed-byte load -> expand to 16 E2M1 nibbles -> decode_e2m1_to_f16 (F16-domain inject; cast f32) -> * |block_scale| -> FMA into an fp32 accumulator against the activation. A warp.sum reduces the 32 partials to the output element. The decode stays in the float16 domain because the M5 flushes f32/bf16 denormals to zero on arithmetic inputs (a magic-constant f32/bf16 decode silently zeroes every +-0.5, see patterns/apple-m5-denormal-flush-to-zero) but preserves f16 subnormals, so the f16 inject decodes +-0.5 exactly (verified on-device).

Two hard M5 constraints, both satisfied by the width-16 decode:

The decode's 16-bit SIMD width is exactly NVFP4_SF_VECTOR_SIZE = 16 nibbles per block; the Metal backend crashes on >= 24-lane 16-bit-element SIMD arithmetic (see known-limitations/apple-m5-wide-16bit-simd-codegen-crash). Staying at 16 lanes dodges it. All accumulation is fp32 (any width is safe there).
The dequant arithmetic (decode_e2m1_to_f16(nibble).cast[f32]() * abs(scale), fp32 accumulate) is BIT-IDENTICAL to the materialize->dense oracle (fp4_dequant.mojo) -- all 16 E2M1 values are exact in f16->f32, so the f16 decode equals decode_e2m1_to_f32 bit-for-bit -- so the GEMV is a within-tolerance match of the dense path (differing only by the fp32-vs-MMA reduction order and the final cast).

Functions

enqueue_apple_fp4_gemv: Enqueue the M=1 W4A16 GEMV: out = a @ dequant(packed, scales)^T.
fp4_gemv_kernel: One warp per output column; 32 lanes stride down K decoding FP4 -> fp32.

Functions​

Functions