For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Python class
TextGenerationInputs
TextGenerationInputs
class max.pipelines.modeling.types.TextGenerationInputs(batches, num_steps, input_tokens=-1, batch_type=BatchType.TG)
Bases: PipelineInputs, Generic[TextGenerationContextType]
Input parameters for text generation pipeline operations.
This class encapsulates the batch of contexts and number of steps required for token generation in a single input object, replacing the previous pattern of passing batch and num_steps as separate parameters.
-
Parameters:
batch_echo
List indicating whether echo is enabled for each context in the batch.
batch_top_log_probs
List of requested top log probabilities per context in the batch.
batch_type
batch_type: BatchType = 'TG'
Type of batch.
batches
Variable list of batches, with each batch being a list of contexts.
There can be multiple batches when using data parallelism, in which each batch is mapped to a different device replica.
enable_echo
property enable_echo: bool
True if any context in the batch has echo enabled.
enable_log_probs
property enable_log_probs: bool
True if any context in the batch requests log probabilities.
flat_batch
property flat_batch: list[TextGenerationContextType]
Flattened list of contexts across all replicas.
input_tokens
input_tokens: int = -1
Number of input tokens.
num_steps
num_steps: int
Number of steps to run for.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!