struct
Q4_KEncoding
The Q4_K quantization encoding.
Because this holds the quantized data in a special packing format, it currently does not print float values at runtime—it's just a bag of bits in uint8 format.
Implemented traits
AnyType
,
QuantizationEncoding
Methods
quantize
static quantize(_tensor: Tensor[float32]) -> Tensor[uint8]
Quantizes the full-precision tensor tensor
to Q4_K.
The quantize method is not yet implemented. However, since Q4_K quantized ops are supported, Q4_KEncoding is still provided to allow code to be generic over quantization encoding type.
id
static id() -> String
Identifier for the Q4_K quantized encoding.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!
If you'd like to share more information, please report an issue on GitHub
😔 What went wrong?