Skip to main content

/

Docs

Nightly

v25.1

Log in

GPU Debugging

The MAX SDK provides support for debugging Mojo code running on GPU using CUDA-GDB. You can either debug using the cuda-gdb command-line interface, or through VS Code, using the Mojo and NVIDIA extensions.

GPU debugging setup​

To debug Mojo code on GPU, you need to be able run Mojo code on GPU. Currently this requires a Linux system with a supported GPU. For details, see GPU requirements.

If you're using VS Code, you can run it on the same system where your GPU code is running (the "target system"), or on a separate system using remote debugging.

To debug on GPU, you need to choose to debug with CUDA-GDB when you start a debugging session. Note that CUDA-GDB has very limited debugging capabilities for Mojo code running on the CPU.

To set up for GPU debugging:

  1. Install the NVIDIA CUDA Toolkit 12.4 or later on the target system. Make sure that the cuda-gdb binary is in your $PATH environment variable. For example, if you have CUDA Toolkit 12.8 installed, add /usr/local/cuda12.8/bin to your $PATH.

  2. If using VS Code, install Nsight Visual Studio Code Edition from the Visual Studio Marketplace.

Using the classic debugger backend​

CUDA-GDB includes two debugger backends, called the "classic debugger" and the "universal debugger." By default, Mojo uses the CUDA-GDB universal debugger. However, some systems require the classic debugger, instead. If you find that the debugger is losing its connection with the process being debugged, you may need to use the classic debugger. To use the classic debugger backend:

  • On the target system, set the environment variable CUDBG_USE_LEGACY_DEBUGGER to 1 in your shell configuration file (for example, .bashrc, .zshrc or config.fish). Source the file or start a new shell.

  • When creating a launch configuration for GPU debugging, add the following settings to the launch.json configuration:

    "legacyDebugger": true,
    "initCommands": [
        "set environment CUDBG_USE_LEGACY_DEBUGGER=1"
    ],
    "legacyDebugger": true,
    "initCommands": [
        "set environment CUDBG_USE_LEGACY_DEBUGGER=1"
    ],

Start GPU debugging from the command line​

To start a GPU debugging session in VS Code from the command line, run the following command on the target system:

mojo debug --cuda-gdb --break-on-launch --vscode myproject.mojo
mojo debug --cuda-gdb --break-on-launch --vscode myproject.mojo

To use the CUDA-GDB command line debugger, omit the --vscode argument.

The --break-on-launch flag is optional but very useful: it stops execution as soon as the GPU kernel launches, allowing you to set breakpoints inside the GPU code.

Start GPU debugging from VS Code​

The easiest way to start GPU debugging from VS Code is to add a Launch configuration. For example, the following launch configuration starts debugging the current Mojo file using CUDA-GDB.

  {
    "type": "mojo-cuda-gdb",
    "request": "launch",
    "name": "Mojo: Debug current Mojo file with CUDA-GDB",
    "description": "Launch and debug the Mojo file that is active on the editor when the debug session starts, using CUDA-GDB.",
    "mojoFile": "${file}",
    "args": [],
    "env": [],
    "cwd": "${workspaceFolder}",
    "breakOnLaunch": true,
    "legacyDebugger": true,
    "initCommands": [
        "set environment CUDBG_USE_LEGACY_DEBUGGER=1"
    ],
  },
  {
    "type": "mojo-cuda-gdb",
    "request": "launch",
    "name": "Mojo: Debug current Mojo file with CUDA-GDB",
    "description": "Launch and debug the Mojo file that is active on the editor when the debug session starts, using CUDA-GDB.",
    "mojoFile": "${file}",
    "args": [],
    "env": [],
    "cwd": "${workspaceFolder}",
    "breakOnLaunch": true,
    "legacyDebugger": true,
    "initCommands": [
        "set environment CUDBG_USE_LEGACY_DEBUGGER=1"
    ],
  },

The last two settings, legacyDebugger and initCommands should only be included if required on your system to maintain a stable connection to the process being debugged, as described in Using the classic debugger backend.

Issuing CUDA-GDB commands​

Some features of the debugger are only available via CUDA-GDB commands, so it's worth familiarizing yourself with CUDA-GDB even if you're using VS Code as a frontend.

If you're running in VS Code, you can enter CUDA-GDB commands in the debug console by prefixing them with a single backtick (`). Note that the console may automatically add a second backtick at the end of your command, which prevents it from being recognized as a CUDA-GDB command. Be sure to remove the second backtick before submitting the command.

The examples in the following section are raw CUDA-GDB commands, without the backtick.

A good starting point for learning about CUDA-GDB is the CUDA-GDB User Manual

Tips and tricks​

The following sections provide tips for some common tasks in GPU debugging.

Setting breakpoints in GPU code​

There are a few quirks to setting breakpoints in GPU code. If you set a breakpoint in GPU code before the GPU kernel launches, the breakpoint will show up in a different function (a CPU function).

When the debugger pauses at this first breakpoint, click Continue to resume execution, and the debugger should stop at the correct location in the GPU code. When paused at the first breakpoint, you can add more breakpoints in the GPU code, however, the breakpoints won't show up in the left gutter until the GPU kernel launches.

Symbol breakpoints aren't supported when debugging with CUDA-GDB.

On the CUDA-GDB command line, you can set a breakpoint using the break command (which can be abbreviated to b):

break filename.mojo:line_number
b filename.mojo:line_number

Stepping not supported on GPU​

The step commands—Step Over, Step Into, and Step Out do not work reliably on GPU. Instead we recommend adding breakpoints and using Continue to move between breakpoints.

Changing kernel focus​

You can use CUDA-GDB commands to change the current kernel focus: that is, the block and thread index that you're currently inspecting. Use the cuda to inspect the current focus or change focus:

cuda block thread
block (0,0,0), thread (0,0,0)
cuda block 0,0,0 thread 1,0,0
[Switching focus to CUDA kernel 0, grid 1, block (0,0,0), thread (1,0,0), device 0, sm 0, warp 0, lane 1]
cuda block thread
block (0,0,0), thread (0,0,0)
cuda block 0,0,0 thread 1,0,0
[Switching focus to CUDA kernel 0, grid 1, block (0,0,0), thread (1,0,0), device 0, sm 0, warp 0, lane 1]

For more information, see Kernel focus in the CUDA-GDB documentation.

Inspecting registers​

Inspect the values of registers using the info registers command.

`info registers $R0 $R1
info registers $R0 $R1
R0             0xebfffda8          -335544920
R1             0xfffda8            16776616
`info registers $R0 $R1
info registers $R0 $R1
R0             0xebfffda8          -335544920
R1             0xfffda8            16776616

For more information, see Registers in the CUDA-GDB documentation.

Was this page helpful?