Skip to main content

/

Python module

registry

`estimate_kv_cache_size()`

max.kv_cache.registry.estimate_kv_cache_size(params, max_batch_size, max_seq_len, num_layers, available_cache_memory, devices, **kwargs)

Parameters:

params (KVCacheParams)
max_batch_size (int | None)
max_seq_len (int)
num_layers (int)
available_cache_memory (int)
devices (Sequence[Device])
kwargs (Any)

Return type:

`infer_optimal_batch_size()`

max.kv_cache.registry.infer_optimal_batch_size(params, max_seq_len, num_layers, available_cache_memory, devices, **kwargs)

Parameters:

params (KVCacheParams)
max_seq_len (int)
num_layers (int)
available_cache_memory (int)
devices (Sequence[Device])
kwargs (Any)

Return type:

`load_kv_manager()`

max.kv_cache.registry.load_kv_manager(params, max_batch_size, max_seq_len, num_layers, devices, session, available_cache_memory=None, page_size=512)

Parameters:

params (KVCacheParams)
max_batch_size (int | None)
max_seq_len (int)
num_layers (int)
devices (Sequence[Device])
session (InferenceSession)
available_cache_memory (int | None)
page_size (int | None)

Return type:

PagedKVCacheManager | NullKVCacheManager

estimate_kv_cache_size()
infer_optimal_batch_size()
load_kv_manager()

View source

View source

Was this page helpful?

Thank you! We'll create more content like this.

Thank you for helping us improve!