Mojo function

generic_fused_qkv_matmul_kv_cache_paged_ragged

generic_fused_qkv_matmul_kv_cache_paged_ragged[dtype: DType, weight_dtype: DType, target: StringSlice[StaticConstantOrigin] = StringSlice("cpu"), group_size: Optional[Int] = None, has_zp: Optional[Bool] = None](hidden_state: LayoutTensor[dtype, hidden_state.layout, hidden_state.origin, element_layout=hidden_state.element_layout, layout_int_type=hidden_state.layout_int_type, linear_idx_type=hidden_state.linear_idx_type, masked=hidden_state.masked, alignment=hidden_state.alignment], input_row_offsets: LayoutTensor[DType.uint32, input_row_offsets.layout, input_row_offsets.origin, element_layout=input_row_offsets.element_layout, layout_int_type=input_row_offsets.layout_int_type, linear_idx_type=input_row_offsets.linear_idx_type, masked=input_row_offsets.masked, alignment=input_row_offsets.alignment], weight: LayoutTensor[weight_dtype, weight.layout, weight.origin, element_layout=weight.element_layout, layout_int_type=weight.layout_int_type, linear_idx_type=weight.linear_idx_type, masked=weight.masked, alignment=weight.alignment], kv_collection: PagedKVCacheCollection[kv_collection.dtype_, kv_collection.kv_params_, kv_collection.page_size, kv_collection.scale_dtype_, kv_collection.quantization_granularity_], layer_idx: UInt32, output: LayoutTensor[dtype, output.layout, output.origin, element_layout=output.element_layout, layout_int_type=output.layout_int_type, linear_idx_type=output.linear_idx_type, masked=output.masked, alignment=output.alignment], ctx: DeviceContextPtr)

Performs a fused QKV matmul. Q outputs are written to the output argument while K and V outputs are written in-place into k_cache and v_cache.

Args:

hidden_state (LayoutTensor): Tensor with shape (sum(seq_lens), num_heads * head_size).
input_row_offsets (LayoutTensor): Tensor with shape (batch_size + 1,). The value at each index is the start_idx of the corresponding batch in hidden_state.
weight (LayoutTensor): Tensor with shape (num_heads * head_size, num_kv_heads * head_size).
kv_collection (PagedKVCacheCollection): The object storing the KVCache for this layer.
layer_idx (UInt32): The current layer, used to retrieve the KVCache object from kv_collection.
output (LayoutTensor): The pre-allocated output buffer for Q projections. K and V projections are written in-place to k_cache and v_cache. Shape: (sum(seq_lens), num_heads * head_size).
ctx (DeviceContextPtr): The call context pointer, passed by the graph compiler.