Mojo function
flash_attention_kv_cache
flash_attention_kv_cache[dtype: DType, cache_t: KVCacheT, //](q: NDBuffer[dtype, 4, origin, shape, strides], k: cache_t, v: cache_t, mask: NDBuffer[dtype, rank, origin, shape, strides], scale: SIMD[float32, 1], output: NDBuffer[dtype, 4, origin, shape, strides], sink_weights: OptionalReg[NDBuffer[dtype, 1, MutableAnyOrigin]] = OptionalReg[NDBuffer[dtype, 1, MutableAnyOrigin]]({:i1 0, 1}))
flash_attention_kv_cache[dtype: DType, cache_t: KVCacheT, mask_t: MHAMask, //](q: NDBuffer[dtype, 4, origin, shape, strides], k: cache_t, v: cache_t, mask: mask_t, scale: SIMD[float32, 1], output: NDBuffer[dtype, 4, origin, shape, strides], sink_weights: OptionalReg[NDBuffer[dtype, 1, MutableAnyOrigin]] = OptionalReg[NDBuffer[dtype, 1, MutableAnyOrigin]]({:i1 0, 1}))
flash_attention_kv_cache[dtype: DType, cache_t: KVCacheT, mask_t: MHAMask, //](q: NDBuffer[dtype, 3, origin, shape, strides], q_input_row_offsets: NDBuffer[uint32, 1, origin, shape, strides], kv_input_row_offsets: NDBuffer[uint32, 1, origin, shape, strides], k: cache_t, v: cache_t, mask: mask_t, scale: SIMD[float32, 1], output: NDBuffer[dtype, 3, origin, shape, strides], sink_weights: OptionalReg[NDBuffer[dtype, 1, MutableAnyOrigin]] = OptionalReg[NDBuffer[dtype, 1, MutableAnyOrigin]]({:i1 0, 1}))
Entrypoint for ragged tensors.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!