For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Python class
IdentityPipelineTokenizer
IdentityPipelineTokenizer
class max.pipelines.IdentityPipelineTokenizer(*args, **kwargs)
Bases: PipelineTokenizer[TokenGeneratorContext, str, TextGenerationRequest]
A pass-through tokenizer that returns prompts unchanged.
decode()
async decode(encoded, **kwargs)
Returns the encoded string unchanged (identity decoding).
encode()
async encode(prompt, add_special_tokens=False)
Returns the prompt unchanged (identity encoding).
eos
property eos: int
Returns the end-of-sequence token ID (0 for identity).
expects_content_wrapping
property expects_content_wrapping: bool
Returns whether this tokenizer expects content wrapping.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!