Python class
KVCacheInputs
KVCacheInputs
class max.nn.kv_cache.KVCacheInputs(inputs)
Bases: object
KVCacheInputs is a sequence of KVCacheInputsPerDevice.
The number of KVCacheInputsPerDevice in the sequence is equal to the number of devices used to run the model. For example, if the model is run with DP=2 + TP=4 then there will be 8 items in the list.
-
Parameters:
-
inputs (Sequence[KVCacheInputsPerDevice])
inputs
inputs: Sequence[KVCacheInputsPerDevice]
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!