Mojo struct
PagedKVCacheCollectionFA3Fallback
struct PagedKVCacheCollectionFA3Fallback[type_: DType, kv_params_: KVCacheStaticParams, page_size: Int]
Aliases
type = type_:kv_params = kv_params_:CacheType = PagedKVCacheFA3Fallback[type_, kv_params_, page_size]:
Fields
- blocks (
NDBuffer[type_, 5, MutableAnyOrigin]): - cache_lengths (
NDBuffer[int32, 1, MutableAnyOrigin]): - lookup_table (
NDBuffer[int32, 3, MutableAnyOrigin]): - max_seq_length (
SIMD[uint32, 1]): - max_cache_length (
SIMD[uint32, 1]):
Implemented traits
AnyType,
Copyable,
KVCollectionT,
Movable,
UnknownDestructibility
Methods
__init__
__init__(out self, blocks: NDBuffer[type_, 5, origin], cache_lengths: NDBuffer[int32, 1, origin], lookup_table: NDBuffer[int32, 3, origin], max_seq_length: SIMD[uint32, 1], max_cache_length: SIMD[uint32, 1])
__copyinit__
__copyinit__(out self, other: Self)
__moveinit__
__moveinit__(out self, owned other: Self)
copy
copy(self) -> Self
Explicitly construct a copy of self.
Returns:
A copy of this value.
get_key_cache
get_key_cache(self, layer_idx: Int) -> PagedKVCacheFA3Fallback[type_, kv_params_, page_size]
get_value_cache
get_value_cache(self, layer_idx: Int) -> PagedKVCacheFA3Fallback[type_, kv_params_, page_size]
cache_length
cache_length(self, bs_idx: Int) -> Int
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!