For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

VisionPreprocessCache

VisionPreprocessCache

class max.pipelines.lib.VisionPreprocessCache(max_bytes, *, idle_seconds=0.0, clock=<built-in function monotonic>)

source

Bases: Generic[_T]

LRU cache of preprocessed media payloads, bounded by total bytes.

Model-agnostic: a tokenizer supplies a content key and a function that preprocesses one media item, and get_or_preprocess() does the rest. Everything that is easy to get wrong per model – freezing the arrays that are about to be shared, charging the budget for what a payload really retains, evicting, reclaiming on idle, and pickling into the model worker – lives here rather than in each architecture.

Keyed on the same raw-encoded-bytes digest that VisionEncoderCache uses, so both caches hit and miss together for a given image.

This sits upstream of the vision encoder cache: it is consulted in the tokenizer, before preprocessing, whereas the encoder cache is consulted in the model worker after preprocessing has already run. A hit therefore skips the resize, rescale and patchify – work the encoder cache cannot avoid no matter how often it hits.

For images the decode itself is not saved on the serving path, because the API server already decodes every image once at admission and hands the tokenizer the decoded image; offline callers, which pass raw bytes through to the tokenizer, save the decode too. For video, which is never decoded at admission, a hit skips the whole decode.

Bounded by bytes rather than by entry count (unlike BoundedCache) because a preprocessed entry’s size tracks the resized image area: a thumbnail and a full-budget image differ by more than an order of magnitude, so an entry count bounds host memory far too loosely to be a safe default.

Parameters:

• max_bytes (int) – Host-memory budget for cached payloads. 0 disables the cache, in which case put() is a no-op and get() always misses.
• idle_seconds (float) – Drop an entry once it has gone this long without being used, so a burst of traffic does not hold host memory for the rest of the process’s life. Swept lazily on the next lookup or insert, which needs no thread and no timer: the reclaim exists to stop the cache holding memory it is not earning, and a process that has stopped touching it is not competing for that memory either. 0 keeps entries until the budget evicts them.
• clock (Callable[[], float]) – Monotonic seconds source, for tests to age entries without sleeping.

clear()

clear()

source

Drops every entry, returning the bytes freed.

Return type:

int

collect()

collect()

source

Drops entries unused for idle_seconds, returning bytes freed.

Runs on every lookup and every insert, so no caller has to schedule it. A no-op when idle_seconds is 0.

Return type:

int

enabled

property enabled: bool

source

Whether caching is enabled (max_bytes > 0).

for_images()

classmethod for_images(runtime)

source

Builds the preprocessed-image cache this deployment configured.

Parameters:

runtime (PipelineRuntimeConfig)

Return type:

VisionPreprocessCache[_T]

for_videos()

classmethod for_videos(runtime)

source

Builds the preprocessed-video cache this deployment configured.

Budgeted separately from for_images() because a video entry is an order of magnitude larger than an image one, so a shared budget would let a single video evict many images.

Parameters:

runtime (PipelineRuntimeConfig)

Return type:

VisionPreprocessCache[_T]

get()

get(key)

source

Look up a payload by content key, refreshing LRU order.

Also sweeps idle entries, because lookups are what a cache that is working does: a conversation resending its image hits every turn and never inserts, so sweeping only on insert would never reclaim anything on the workload this cache exists for.

The entry being looked up is refreshed before the sweep, so a request can never be answered by preprocessing something this same call just threw away. An entry past its deadline that a request wants is served, not treated as a miss: the payload is a pure function of the content key, so it cannot be stale, and discarding one we still hold only to preprocess the identical bytes again would spend CPU to free nothing. The hit resets the clock, which is the point – the entry turned out not to be idle.

Parameters:

key (int)

Return type:

_T | None

get_or_preprocess()

get_or_preprocess(key, preprocess)

source

Returns the cached payload for key, else preprocesses and caches.

The one entry point a tokenizer needs. On a miss it runs preprocess, freezes every array in the result, charges the budget for what those arrays retain, and stores it – unless the payload is larger than the whole budget, which put() drops rather than cache so that one oversized item cannot flush every useful entry. Such a payload is still returned, and still frozen, so a caller cannot tell from the result whether it was retained.

Two requests for the same uncached item may both preprocess it: this deliberately holds no lock across preprocess, since serializing on one would make every miss wait behind an unrelated one. The duplicate insert is accounted for correctly.

The payload’s arrays should own their data. A view pins the whole buffer it looks into, so caching one out of a batched processor call keeps that entire batch alive; copy it (~numpy.ascontiguousarray) before returning it from preprocess if the processor slices a shared buffer.

Parameters:

• key (int | None) – The item’s content digest, or None when the caller has none – no caching is enabled, so preprocess runs and its result is returned untouched, exactly as it would without a cache.
• preprocess (Callable[[], _T]) – Preprocesses the one media item, called only on a miss.

Returns:

The preprocessed payload, from the cache when it was there.

Return type:

_T

hits

property hits: int

source

Lookups served from the cache.

idle_seconds

property idle_seconds: float

source

never).

Type:

How long an entry may go unused before it is reclaimed (0

misses

property misses: int

source

Lookups that had to preprocess.

put()

put(key, value, nbytes)

source

Insert a payload, evicting least-recently-used entries to fit.

A payload larger than the whole budget is dropped rather than cached, so one oversized image cannot flush every useful entry.

Parameters:

• key (int) – The content key to key on.
• value (_T) – The preprocessed payload to retain.
• nbytes (int) – Host bytes value retains, used against the budget.

Return type:

None

total_bytes

property total_bytes: int

source

Host bytes currently retained by cached payloads.