Generate image descriptions with Llama 3.2 Vision

Judy Heflin

Technical Writer

max

llama3-vision

multimodal

a100

The MAX framework simplifies the process to create an endpoint for multimodal models that handle both text and images, such as Llama 3.2 11B Vision Instruct, which excels at tasks such as image captioning and visual question answering. This tutorial walks you through installing the necessary tools, configuring access, and serving the model locally with an OpenAI-compatible endpoint.

GPU required

To run the model in this tutorial, your system must have a compatible GPU.

System requirements:

Mac

Linux

WSL

GPU

Set up your environment

Create a Python project to install our APIs and CLI tools:

pip

1. Create a project folder:

   mkdir vision-tutorial && cd vision-tutorial

   mkdir vision-tutorial && cd vision-tutorial
2. Create and activate a virtual environment:

   python3 -m venv .venv/vision-tutorial \
     && source .venv/vision-tutorial/bin/activate

   python3 -m venv .venv/vision-tutorial \
     && source .venv/vision-tutorial/bin/activate
3. Install the modular Python package:
   Nightly

   pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://dl.modular.com/public/nightly/python/simple/

   pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://dl.modular.com/public/nightly/python/simple/

   Stable

   pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://modular.gateway.scarf.sh/simple/

   pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://modular.gateway.scarf.sh/simple/

uv

1. If you don't have it, install uv:

   curl -LsSf https://astral.sh/uv/install.sh | sh

   curl -LsSf https://astral.sh/uv/install.sh | sh

   Then restart your terminal to make uv accessible.

2. Create a project:

   uv init vision-tutorial && cd vision-tutorial

   uv init vision-tutorial && cd vision-tutorial
3. Create and start a virtual environment:

   uv venv && source .venv/bin/activate

   uv venv && source .venv/bin/activate
4. Install the modular Python package:
   Nightly

   uv pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://dl.modular.com/public/nightly/python/simple/ \
     --index-strategy unsafe-best-match

   uv pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://dl.modular.com/public/nightly/python/simple/ \
     --index-strategy unsafe-best-match

   Stable

   uv pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://modular.gateway.scarf.sh/simple/ \
     --index-strategy unsafe-best-match

   uv pip install modular \
     --extra-index-url https://download.pytorch.org/whl/cpu \
     --extra-index-url https://modular.gateway.scarf.sh/simple/ \
     --index-strategy unsafe-best-match

conda

1. If you don't have it, install conda. A common choice is with brew:

   brew install miniconda

   brew install miniconda
2. Initialize conda for shell interaction:

   conda init

   conda init

   If you're on a Mac, instead use:

   conda init zsh

   conda init zsh

   Then restart your terminal for the changes to take effect.

3. Create a project:

   conda create -n vision-tutorial

   conda create -n vision-tutorial
4. Start the virtual environment:

   conda activate vision-tutorial

   conda activate vision-tutorial
5. Install the modular conda package:
   Nightly

   conda install -c conda-forge -c https://conda.modular.com/max-nightly/ modular

   conda install -c conda-forge -c https://conda.modular.com/max-nightly/ modular

   Stable

   conda install -c conda-forge -c https://conda.modular.com/max/ modular

   conda install -c conda-forge -c https://conda.modular.com/max/ modular

pixi

1. If you don't have it, install pixi:

   curl -fsSL https://pixi.sh/install.sh | sh

   curl -fsSL https://pixi.sh/install.sh | sh

   Then restart your terminal for the changes to take effect.

2. Create a project:

   pixi init vision-tutorial \
     -c https://conda.modular.com/max-nightly/ -c conda-forge \
     && cd vision-tutorial

   pixi init vision-tutorial \
     -c https://conda.modular.com/max-nightly/ -c conda-forge \
     && cd vision-tutorial
3. Install the modular conda package:
   Nightly

   pixi add modular

   pixi add modular

   Stable

   pixi add "modular==25.3"

   pixi add "modular==25.3"
4. Start the virtual environment:

   pixi shell

   pixi shell

Configure Hugging Face access

To get the model used below, you must have a Hugging Face user access token and approved access to the Llama 3.2 11B Vision Instruct Hugging Face repo.

To create a Hugging Face user access token, see Access Tokens. Within your local environment, save your access token as an environment variable.

export HF_TOKEN="hf_..."

export HF_TOKEN="hf_..."

Generate a sample description

You can generate an image description using the max generate command. Downloading the Llama 3.2 11B Vision Instruct model weights takes some time.

You may need to alter the —-max-length and —-max-batch-size parameters depending on the amount of memory you have access to. The following command is optimized for a p4d.24xlarge instance with one NVIDIA A100 GPU and 96 vCPUs.

max generate \
  --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
  --prompt "<|image|><|begin_of_text|>What is in this image?" \
  --image_url="https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg" \
  --max-new-tokens 100 \
  --max-batch-size 1 \
  --max-length 108172

max generate \
  --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
  --prompt "<|image|><|begin_of_text|>What is in this image?" \
  --image_url="https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg" \
  --max-new-tokens 100 \
  --max-batch-size 1 \
  --max-length 108172

When using the max CLI tool with multimodal input, you must provide both a --prompt and an --image_url. Additionally, the prompt should be in a valid format for the model used. For Llama 3.2 Vision 11B Instruct, you must include the <|image|> tag in the prompt if the input includes an image to reason about. For more information about Llama 3.2 Vision prompt templates, see Vision Model Inputs and Outputs.

Serve the Llama 3.2 Vision model

You can alternatively serve the Llama 3.2 Vision model and make multiple requests to a local endpoint. If you already tested the model with the max generate command, you do not have to wait for the model to download again.

Serve the model with the max serve command:

max serve \
  --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
  --max-length 108172 \
  --max-batch-size 1

max serve \
  --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
  --max-length 108172 \
  --max-batch-size 1

The endpoint is ready when you see this message printed in your terminal:

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

Test the endpoint

After the server is running, you can test it by opening a new terminal window and sending a curl request.

When making requests with max serve, you do not need to include model-specific image tags within your prompt.

The following request includes an image URL and a question to answer about the provided image:

curl -N http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "meta-llama/Llama-3.2-11B-Vision-Instruct",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What is in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
          }
        }
      ]
    }
  ],
  "max_tokens": 300
}' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '\n' | sed 's/\\n/\n/g'

curl -N http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "meta-llama/Llama-3.2-11B-Vision-Instruct",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What is in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
          }
        }
      ]
    }
  ],
  "max_tokens": 300
}' | grep -o '"content":"[^"]*"' | sed 's/"content":"//g' | sed 's/"//g' | tr -d '\n' | sed 's/\\n/\n/g'

This sends an image along with a text prompt to the model, and you should receive a response describing the image. You can test the endpoint with any local base64-encoded image or any image URL.

If you make significant changes to the provided request template, you might receive less accurate responses. Some parts of the text prompt get ignored for certain input combinations. We've identified the problem and will have a fix in a subsequent nightly release.

Next steps

Now that you have successfully deployed Llama 3.2 Vision, you can:

• Experiment with different images and prompts
• Explore deployment configurations and additional features, such as function calling, prefix caching, and structured output
• Deploy the model to a containerized cloud environment for scalable serving

Deploy Llama 3 on GPU with MAX Serve

Learn how to deploy Llama 3 on GPU with MAX Serve.

Deploy Llama 3 on GPU-powered Kubernetes clusters

Learn how to deploy Llama 3 using Kubernetes, MAX, and NVIDIA GPUs

Did this tutorial work for you?

Thank you! We'll create more content like this.

Thank you for helping us improve!

😔 What went wrong?

Some code doesn’t work

It's missing a step; I don't know what to do

I don't understand the purpose

Other

Submit