Mojo function

topk_sampling_from_prob

topk_sampling_from_prob[dtype: DType, out_idx_type: DType, block_size: Int = 1024](ctx: DeviceContext, probs: LayoutTensor[dtype, layout, origin, address_space=address_space, element_layout=element_layout, layout_int_type=layout_int_type, linear_idx_type=linear_idx_type, masked=masked, alignment=alignment], output: LayoutTensor[out_idx_type, layout, origin, address_space=address_space, element_layout=element_layout, layout_int_type=layout_int_type, linear_idx_type=linear_idx_type, masked=masked, alignment=alignment], top_k_val: Int, deterministic: Bool = False, rng_seed: UInt64 = 0, rng_offset: UInt64 = 0, indices: OptionalReg[LayoutTensor[out_idx_type, Layout.row_major(-1), MutAnyOrigin]] = None, top_k_arr: OptionalReg[LayoutTensor[out_idx_type, Layout.row_major(-1), MutAnyOrigin]] = None)

Top-K sampling from probability distribution.

Performs stochastic sampling from a probability distribution, considering only the top-k most probable tokens. Uses rejection sampling with ternary search to efficiently find appropriate samples.

Args:

ctx (DeviceContext): Device context for kernel execution.
probs (LayoutTensor): Input probability distribution [batch_size, d].
output (LayoutTensor): Output sampled indices [batch_size].
top_k_val (Int): Default top-k value (number of top tokens to consider).
deterministic (Bool): Whether to use deterministic sampling.
rng_seed (UInt64): Random seed for Random number generator.
rng_offset (UInt64): Random offset for Random number generator.
indices (OptionalReg): Optional row indices for batch indexing [batch_size].
top_k_arr (OptionalReg): Optional per-row top-k values [batch_size].

Raises:

Error: If tensor ranks or shapes are invalid.