For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).

Python class

KVCacheParamInterface

`KVCacheParamInterface`

class max.nn.kv_cache.KVCacheParamInterface(*args, **kwargs)

source

Bases: Protocol

Interface for KV cache parameters.

`bytes_per_block`

property bytes_per_block: int

source

Number of bytes per cache block.

`data_parallel_degree`

data_parallel_degree: int

source

`get_symbolic_inputs()`

get_symbolic_inputs(prefix='')

source

Returns the symbolic inputs for the KV cache.

Parameters:: prefix (str)
Return type:: KVCacheInputs[TensorType, BufferType]

`host_kvcache_swap_space_gb`

host_kvcache_swap_space_gb: float | None

source

`kv_connector`

kv_connector: KVConnectorType | None

source

`n_devices`

n_devices: int

source

`num_eagle_speculative_tokens`

num_eagle_speculative_tokens: int = 0

source

`page_size`

page_size: int

source

KVCacheParamInterface​

bytes_per_block​

data_parallel_degree​

get_symbolic_inputs()​

host_kvcache_swap_space_gb​

kv_connector​

n_devices​

num_eagle_speculative_tokens​

page_size​

`KVCacheParamInterface`

`bytes_per_block`

`data_parallel_degree`

`get_symbolic_inputs()`

`host_kvcache_swap_space_gb`

`kv_connector`

`n_devices`

`num_eagle_speculative_tokens`

`page_size`