For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Python function
compute_num_host_blocks
compute_num_host_blocks()
max.nn.kv_cache.compute_num_host_blocks(params)
Computes the number of blocks that can be allocated on the host.
The host (CPU/disk) tier is a single pool shared across all data-parallel
replicas via one connector (SERVOPT-1501), sized at
host_kvcache_swap_space_gb total (independent of
data_parallel_degree). The connector is replica-agnostic, so a block
offloaded by one replica can be served as a cache hit for another.
-
Returns:
-
The total number of blocks that can be allocated in the shared host pool.
-
Parameters:
-
params (KVCacheParamInterface)
-
Return type:
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!