struct

Q4_0Encoding

The Q4_0 quantization encoding.

Q4_0 is a block quantization scheme originally designed for GGML in which each element (number) is reduced to an unsigned, fixed-point, 4-bit value. Multiple quantized elements are packed together in a block, all using the same float16 scale.

The packing scheme requires that the innermost dimension is a factor of 32. When the tensor is quantized to Q4_0, each block of 32 scalar values is packed into 18 bytes. The first two bytes specify the float16 quantization scale, and the other 16 bytes hold the 32 values (one byte holds two 4-bit values).

Because this holds the quantized data in a special packing format, it currently does not print float values at runtime—it's just a bag of bits in uint8 format.

Implemented traits

AnyType, QuantizationEncoding

Methods

quantize

static quantize(tensor: Tensor[float32]) -> Tensor[uint8]

Quantizes the full-precision tensor to Q4_0.

Args:

• ​tensor (Tensor[float32]): Full-precision tensor to quantize. The innermost dimension

Returns:

Quantized Q4_0 tensor. The tensor datatype is uint8 because this is simply a bytes buffer. Each scalar is actually stored with 4 bits.

Raises:

If the last dimension size is not a factor of 32.

id

static id() -> String

Identifier for the Q4_0 quantized encoding.