For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Python class
KVCacheMemory
KVCacheMemory
class max.nn.kv_cache.KVCacheMemory(buffer)
Bases: object
A single KV cache shard as a 2-D uint8 view.
buffer has shape [num_pages, bytes_per_page] with dtype
uint8. This is the form consumed by the offload engine and KV
connectors. ReplicatedKVCacheMemory subclasses this for
caches that are replicated across TP shards (MLA).
-
Parameters:
-
buffer (Buffer)
all_buffers
Returns every shard buffer backing this unit (rank-0 + peers).
buffer
buffer: Buffer
copy_block_to()
copy_block_to(dst, dst_block_id, src_block_id)
Copies one page from this unit into dst (device-to-device).
dst must be the same logical unit on another replica (identical
page layout, same number of shards). The copy may cross devices; the
driver issues a peer-to-peer (or host-bounced) transfer. Used to
materialize a prefix-cache block on a different DP replica
(SERVOPT-1500).
-
Parameters:
-
- dst (KVCacheMemory) – Destination memory unit (same logical unit, another replica).
- dst_block_id (int) – Destination page index.
- src_block_id (int) – Source page index in this unit.
-
Return type:
-
None
total_num_pages
property total_num_pages: int
Returns the total number of pages.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!