Python class
PixelGenerationTokenizer
PixelGenerationTokenizer
class max.pipelines.lib.PixelGenerationTokenizer(model_path, pipeline_config, subfolder, *, subfolder_2=None, revision=None, max_length=None, secondary_max_length=None, trust_remote_code=False, default_num_inference_steps=50, **unused_kwargs)
Bases: PipelineTokenizer[PixelContext, tuple[ndarray[tuple[Any, …], dtype[int64]], ndarray[tuple[Any, …], dtype[bool]]], OpenResponsesRequest]
Encapsulates creation of PixelContext and specific token encode/decode logic.
-
Parameters:
-
- model_path (str) – Path to the model/tokenizer.
- pipeline_config (PipelineConfig) – Pipeline configuration with ModelManifest metadata.
- subfolder (str) – Subfolder within the model path for the primary tokenizer.
- subfolder_2 (str | None) – Optional subfolder for a second tokenizer (e.g. text encoder).
- revision (str | None) – Git revision/branch to use.
- max_length (int | None) – Maximum sequence length for the primary tokenizer.
- secondary_max_length (int | None) – Maximum sequence length for the secondary tokenizer, if used.
- trust_remote_code (bool) – Whether to trust remote code from the model.
- default_num_inference_steps (int)
decode()
async decode(encoded, **kwargs)
Decodes token arrays to text (not implemented for this tokenizer).
encode()
async encode(prompt, add_special_tokens=True, *, use_secondary=False, images=None)
Transforms the provided prompt into a token array.
eos
property eos: int
Returns the end-of-sequence token ID.
expects_content_wrapping
property expects_content_wrapping: bool
Returns whether this tokenizer expects content wrapping.
new_context()
async new_context(request, input_image=None)
Creates a new PixelContext object, leveraging necessary information from OpenResponsesRequest.
-
Parameters:
-
- request (OpenResponsesRequest)
- input_image (Image | None)
-
Return type:
postprocess()
async postprocess(output)
Post-processes pipeline output.
Accepts either a raw numpy array or a GenerationOutput. For raw numpy arrays, denormalizes from [-1, 1] to [0, 1]. For GenerationOutput, returns as-is (denormalization is handled in the pipeline variant before encoding to OutputImageContent).
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!