For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).

Mojo module

fp8_gemv

Apple M5 weight-only FP8 (W8A16) GEMV: out = x @ dequant(W)^T at M=1.

Apple silicon GPU (Metal 4, compute_capability == 5). The batch-1 decode regime (M == 1) is a matrix-vector product, NOT a matrix-matrix product: there is exactly one activation row, so there is no MMA to feed. This kernel is the GEMV -- register-resident, no SMEM, no barrier(), no _mma_apple.

This is the FP8 sibling of fp4_gemv.mojo (the shipped W4A16 GEMV), transliterated op-for-op and SIMPLER: the weight is stored float8_e4m3fn (one byte per element, NOT two packed E2M1 nibbles), so the FP4 nibble-unpack + E2M1-f16-decode + per-16 FP8 block-scale collapse to a single width load + a native float8_e4m3fn -> f32 widening cast. Two FP4-specific M5 hazards vanish with it:

No denormal-flush-to-zero footgun: there is no magic-constant f16 decode injecting +-0.5 (patterns/apple-m5-denormal-flush-to-zero); the E4M3 -> f32 cast is the hardware's own conversion, exact for every E4M3 code on M5 (the same cast the FP4 GEMV already uses on its FP8-E4M3 block scales).
No >=24-lane 16-bit-SIMD-arithmetic codegen crash (known-limitations/apple-m5-wide-16bit-simd-codegen-crash): the only 16-bit-domain arithmetic in the FP4 kernel was the E2M1 nibble expansion, which does not exist here. The FP8 load widens straight to f32; all arithmetic is f32 (any width is safe there).

WHY it wins at decode: batch-1 decode on M5 is weight-read-bandwidth-bound (the profiling campaign measured ~530 GB/s of the ~614 GB/s peak on the bf16 path). W8A16 reads 1 byte/weight instead of the 2 bytes of a bf16 weight, so the wall-clock drops roughly with the byte count. The float8_e4m3fn -> f32 cast gives NO compute speedup (there is no MMA here); the entire win is reading fewer bytes. See Kernels/claude_kb/entries/kernels/apple-m5-fp4-matmul.md and patterns/apple-m5-gpu-perf-model (the co-issue penalty applies only at core saturation -- the GEMV is under-occupancy, so it does not bite here).

Structure (mirrors fp4_gemv.mojo, itself mirroring linalg/gemv.mojo::gemv_kernel_vector): one warp owns one output column n (= one row of the transpose_b weight W[N, K]); its 32 lanes stride down K in whole TILE_K-wide chunks (strided by WARP_SIZE chunks). Per chunk: one coalesced width-TILE_K FP8 load through the Fp8WeightLoader expert object -> native E4M3 -> f32 widen; one width-TILE_K bf16 activation load -> f32; FMA into an fp32 accumulator. A warp.sum reduces the 32 partials to the output element. A scalar (width-1) tail handles K % TILE_K != 0 (the non-aligned edge; empty on the model path, where K is a large power of two).

Scale contract (mirrors the FP4 W4A16 path exactly): the modelopt static FP8 checkpoint carries ONE per-tensor scalar weight_scale (and a per-tensor input_scale that cancels for a bf16 activation -- see the routing branch in quant_ops._matmul_float8). This kernel produces the RAW x @ W_fp8^T; the scalar weight_scale is folded by the graph lowering as a post-matmul multiply (the FP8 analog of NVFP4's weight_scale_2, applied in quant_ops._matmul_float8). Because weight_scale is a scalar it factors out of the sum, so the post-matmul fold is EXACT (not merely within tolerance).

Every DRAM -> register weight transition has an OWNER (Fp8WeightLoader, KB new-primitives/amd-tile-io-expert-objects; the FP8 sibling of Fp4WeightLoader in matmul2d_fp4.mojo): it holds the weight [N, K] TileTensor view and does ALL addressing through TileTensor width-loads / indexing -- no raw pointer arithmetic. The bf16 activation is a plain row-major [1, K] tensor; its width load is an inline TileTensor access (no special decode to own), matching fp4_gemv.mojo.

Structs

Fp8WeightLoader: Owner of the FP8 weight -> f32 register transition for the W8A16 GEMV.

Functions

enqueue_apple_fp8_gemv: Enqueue the M=1 W8A16 GEMV: out = a @ W_fp8^T (raw, unscaled).
enqueue_apple_fp8_matmul: Enqueue the W8A16 matmul: out = a @ W_fp8^T (raw, unscaled).
enqueue_fp8_materialize: Enqueues fp8_materialize_kernel: FP8 weight -> dense [N, K] out_type.
fp8_gemv_kernel: One warp per output column; 32 lanes stride down K widening FP8 -> f32.
fp8_materialize_kernel: Materializes the FP8 weight into a dense [N, K] out_type buffer.

Structs​

Functions​

Structs

Functions