Python class
KVCacheParamInterface
KVCacheParamInterface
class max.nn.kv_cache.KVCacheParamInterface(*args, **kwargs)
Bases: Protocol
Interface for KV cache parameters.
bytes_per_block
property bytes_per_block: int
Number of bytes per cache block.
data_parallel_degree
data_parallel_degree: int
get_symbolic_inputs()
get_symbolic_inputs(prefix='')
Returns the symbolic inputs for the KV cache.
-
Parameters:
-
prefix (str)
-
Return type:
host_kvcache_swap_space_gb
kv_connector
kv_connector: KVConnectorType | None
n_devices
n_devices: int
num_eagle_speculative_tokens
num_eagle_speculative_tokens: int = 0
page_size
page_size: int
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!