Skip to main content

/

Docs

Log in

Mojo struct

Q5_KEncoding

The Q5_K quantization encoding.

Because this holds the quantized data in a special packing format, it currently does not print float values at runtime—it's just a bag of bits in uint8 format.

Implemented traits​

AnyType, QuantizationEncoding

Methods​

quantize​

static quantize(tensor: Tensor[float32]) -> Tensor[uint8]

Quantizes the full-precision tensor tensor to Q5_K.

Args:

  • ​tensor (Tensor[float32]): Full-precision tensor to quantize. The innermost dimension of the tensor must be a factor of 256.

Returns:

Quantized Q5_K tensor. The tensor datatype is uint8 because this is simply a bytes buffer. Each scalar is actually stored with 5 bits.

Raises:

If the last dimension size is not a factor of 256.

id​

static id() -> String

Identifier for the Q5_K quantized encoding.

Was this page helpful?