For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).

Mojo function

update_frequency_data

def update_frequency_data[token_type: DType, //, target: StringSlice[ImmStaticOrigin]](compressed_frequency_data: TileTensor[DType.int32, Storage=compressed_frequency_data.Storage, linear_idx_type=compressed_frequency_data.linear_idx_type], frequency_offsets: TileTensor[DType.uint32, Storage=frequency_offsets.Storage, linear_idx_type=frequency_offsets.linear_idx_type], new_tokens: TileTensor[token_type, Storage=new_tokens.Storage, linear_idx_type=new_tokens.linear_idx_type], ctx: DeviceContext)

Update the frequency data for the given new tokens.

The frequency data is stored in a CSR format. This kernel expects there will be enough padding for each sequence to store the new tokens.

Parameters:

token_type (DType): Element type of the new_tokens tensor (inferred).
target (StringSlice[ImmStaticOrigin]): Target device to dispatch the kernel to.

Args:

compressed_frequency_data (TileTensor[DType.int32, Storage=compressed_frequency_data.Storage, linear_idx_type=compressed_frequency_data.linear_idx_type]): Mutable 2D CSR frequency data where column 0 is the token id and column 1 is the token count within the sequence, updated in place.
frequency_offsets (TileTensor[DType.uint32, Storage=frequency_offsets.Storage, linear_idx_type=frequency_offsets.linear_idx_type]): 1D tensor of starting indices into compressed_frequency_data for each sequence in the batch.
new_tokens (TileTensor[token_type, Storage=new_tokens.Storage, linear_idx_type=new_tokens.linear_idx_type]): 1D tensor of new token ids, one per sequence in the batch.
ctx (DeviceContext): Device context used to dispatch the kernel.