For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).

Mojo function

selective_scan_fwd_gpu

def selective_scan_fwd_gpu[kernel_dtype: DType, DSTATE: Int, output_LT: TensorLayout, x_LT: TensorLayout, out_z_LT: TensorLayout, u_LT: TensorLayout, delta_LT: TensorLayout, A_LT: TensorLayout, B_LT: TensorLayout, C_LT: TensorLayout, D_LT: TensorLayout, z_LT: TensorLayout, delta_bias_LT: TensorLayout](total_batch_dim: Int, batch: Int, dim: Int, seqlen: Int, group_size: Int, delta_softplus: Int8, output: TileTensor[kernel_dtype, output_LT, MutUntrackedOrigin], x: TileTensor[kernel_dtype, x_LT, MutUntrackedOrigin], out_z: TileTensor[kernel_dtype, out_z_LT, MutUntrackedOrigin], u: TileTensor[kernel_dtype, u_LT, MutUntrackedOrigin], delta: TileTensor[kernel_dtype, delta_LT, MutUntrackedOrigin], A: TileTensor[kernel_dtype, A_LT, MutUntrackedOrigin], B: TileTensor[kernel_dtype, B_LT, MutUntrackedOrigin], C: TileTensor[kernel_dtype, C_LT, MutUntrackedOrigin], D: TileTensor[kernel_dtype, D_LT, MutUntrackedOrigin], z: TileTensor[kernel_dtype, z_LT, MutUntrackedOrigin], delta_bias: TileTensor[kernel_dtype, delta_bias_LT, MutUntrackedOrigin], output_strides: IndexList[Int(3)], x_strides: IndexList[Int(4)], out_z_strides: IndexList[Int(3)], u_strides: IndexList[Int(3)], delta_strides: IndexList[Int(3)], A_strides: IndexList[Int(2)], B_strides: IndexList[Int(4)], C_strides: IndexList[Int(4)], D_strides: IndexList[Int(1)], z_strides: IndexList[Int(3)], delta_bias_strides: IndexList[Int(1)])

GPU kernel for selective scan forward pass.

Each thread processes one (batch, dim) pair and iterates through the sequence.

Parameters:

kernel_dtype (DType): Element type of the input and output tensors.
DSTATE (Int): Number of SSM state elements per (batch, dim) pair.
output_LT (TensorLayout): Memory layout of the output tensor.
x_LT (TensorLayout): Memory layout of the x checkpoint tensor.
out_z_LT (TensorLayout): Memory layout of the out_z gated output tensor.
u_LT (TensorLayout): Memory layout of the u input tensor.
delta_LT (TensorLayout): Memory layout of the delta time-step tensor.
A_LT (TensorLayout): Memory layout of the A recurrence matrix.
B_LT (TensorLayout): Memory layout of the B input projection tensor.
C_LT (TensorLayout): Memory layout of the C output projection tensor.
D_LT (TensorLayout): Memory layout of the D skip connection tensor.
z_LT (TensorLayout): Memory layout of the z gating tensor.
delta_bias_LT (TensorLayout): Memory layout of the delta_bias tensor.

Args:

total_batch_dim (Int): Total (batch, dim) pairs launched.
batch (Int): Number of sequences.
dim (Int): Hidden dimension (channels per sequence position).
seqlen (Int): Sequence length.
group_size (Int): Dims per group sharing B and C.
delta_softplus (Int8): When nonzero, apply softplus to delta.
output (TileTensor[kernel_dtype, output_LT, MutUntrackedOrigin]): Output tensor (batch, dim, seqlen).
x (TileTensor[kernel_dtype, x_LT, MutUntrackedOrigin]): Checkpoint tensor (batch, dim, n_chunks, 2*DSTATE).
out_z (TileTensor[kernel_dtype, out_z_LT, MutUntrackedOrigin]): Gated output (batch, dim, seqlen) when z is present.
u (TileTensor[kernel_dtype, u_LT, MutUntrackedOrigin]): Input tensor (batch, dim, seqlen).
delta (TileTensor[kernel_dtype, delta_LT, MutUntrackedOrigin]): Time-step tensor (batch, dim, seqlen).
A (TileTensor[kernel_dtype, A_LT, MutUntrackedOrigin]): Recurrence matrix (dim, DSTATE).
B (TileTensor[kernel_dtype, B_LT, MutUntrackedOrigin]): Input projection (batch, n_groups, DSTATE, seqlen).
C (TileTensor[kernel_dtype, C_LT, MutUntrackedOrigin]): Output projection (batch, n_groups, DSTATE, seqlen).
D (TileTensor[kernel_dtype, D_LT, MutUntrackedOrigin]): Skip vector (dim,).
z (TileTensor[kernel_dtype, z_LT, MutUntrackedOrigin]): Gating tensor (batch, dim, seqlen).
delta_bias (TileTensor[kernel_dtype, delta_bias_LT, MutUntrackedOrigin]): Bias vector (dim,).
output_strides (IndexList[Int(3)]): Stride tuple for output.
x_strides (IndexList[Int(4)]): Stride tuple for x.
out_z_strides (IndexList[Int(3)]): Stride tuple for out_z.
u_strides (IndexList[Int(3)]): Stride tuple for u.
delta_strides (IndexList[Int(3)]): Stride tuple for delta.
A_strides (IndexList[Int(2)]): Stride tuple for A.
B_strides (IndexList[Int(4)]): Stride tuple for B.
C_strides (IndexList[Int(4)]): Stride tuple for C.
D_strides (IndexList[Int(1)]): Stride tuple for D.
z_strides (IndexList[Int(3)]): Stride tuple for z.
delta_bias_strides (IndexList[Int(1)]): Stride tuple for delta_bias.