Mojo function
generic_flash_attention_kv_cache_ragged_sink
generic_flash_attention_kv_cache_ragged_sink[collection_t: KVCollectionT, dtype: DType, //, *, target: StringSlice[StaticConstantOrigin], mask_str: StringSlice[StaticConstantOrigin], score_mod_str: StringSlice[StaticConstantOrigin], local_window_size: Int = -1](q: NDBuffer[dtype, 3, origin, shape, strides], input_row_offsets: ManagedTensorSlice[io_spec, static_spec=static_spec], kv_collection: collection_t, layer_idx: SIMD[uint32, 1], scale: SIMD[float32, 1], output: NDBuffer[dtype, 3, origin, shape, strides], context: DeviceContextPtr, sink_weights: NDBuffer[dtype, 1, MutableAnyOrigin])
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!