Python class
KVCacheParamInterface
KVCacheParamInterface
class max.nn.kv_cache.KVCacheParamInterface(*args, **kwargs)
Bases: Protocol
Interface for KV cache parameters.
bytes_per_block
property bytes_per_block: int
Number of bytes per cache block.
data_parallel_degree
data_parallel_degree: int
enable_kvcache_swapping_to_host
enable_kvcache_swapping_to_host: bool
get_symbolic_inputs()
get_symbolic_inputs()
Returns the symbolic inputs for the KV cache.
-
Return type:
-
FlattenableInputSymbols
host_kvcache_swap_space_gb
n_devices
n_devices: int
page_size
page_size: int
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!