This guide will walk you through Magic, a command-line-based package management tool by Modular
designed for fast, efficient, and scalable project management in Mojo and MAX environments.
Whether you're managing dependencies across multiple platforms, setting up environments for specific tasks,
or working with Python-based projects, magic
CLI simplifies these processes and more.
Built on the powerful Pixi,
Magic leverages its capabilities to provide seamless environment management and package handling for MAX and Mojo applications.
In this tutorial, we'll guide you through everything from setting up your first project
and understanding the magic
CLI to running Mojo code and building a FastAPI application.
By the end, you'll have a solid grasp of how to configure and use magic effectively for
streamlined project management, dependency handling, and environment setup.
Step 1: Create a projectβ
The first step to using Magic is creating a new project. Run the following command:
magic init hello-magic --format mojoproject
magic init hello-magic --format mojoproject
And navigate to the project directly:
cd hello-magic
cd hello-magic
Your project structure will look like this:
βββ .gitattributes
βββ .gitignore
βββ .magic
βββ magic.lock
βββ mojoproject.toml
βββ .gitattributes
βββ .gitignore
βββ .magic
βββ magic.lock
βββ mojoproject.toml
.magic
directory is used to store environment configurations and manages the dependencies for your project. This helps Magic keep your project isolated and ensures that different versions of dependencies won't conflict with other projects. The.magic/envs
sub-directory specifically stores the virtual environments for your project. Unlike other package managers, Magic keeps your environment separate and clean, making it easy to manage and switch between different projects.mojoproject.toml
is a single TOML configuration file.magic.lock
is a critical file for ensuring reproducibility. It captures the exact versions of every dependency in your project. This ensures that when you or someone else runs your project in the future or on a different machine, Magic will install the exact same versions of packages. This avoids the common "it works on my machine" issue, providing consistency, especially in complex projects across different platforms. Please find more details in Pixi lockfile.
Inspect mojoproject.toml
β
The mojoproject.toml
file defines your project's configuration.
It contains sections like project metadata, dependencies, and channels, all in a single TOML file.
Here's an example:
[project]
authors = ["Modular <hello@modular.com>"]
channels = ["conda-forge", "https://conda.modular.com/max"]
description = "Add a short description here"
name = "hello-magic"
platforms = ["osx-arm64"]
version = "0.1.0"
[dependencies]
max = ">=24.4.0,<25"
[project]
authors = ["Modular <hello@modular.com>"]
channels = ["conda-forge", "https://conda.modular.com/max"]
description = "Add a short description here"
name = "hello-magic"
platforms = ["osx-arm64"]
version = "0.1.0"
[dependencies]
max = ">=24.4.0,<25"
The key sections are:
[project]
: Basic metadata such as project name, version, and description so prefer using conda packages as much as possible.channels
: Defines the channels from which to install dependencies. A channel is a repository from which Magic fetches a package and its dependencies.name
: Project name.platforms
: The supported platforms for the project. At the time of writing this tutorial, I am using MacOS (M3) so by default Magic includesosx-arm64
. Other supported platforms for using MAX and Mojo arelinux-64
andlinux-aarch64
.[dependencies]
: Lists conda packages required for the project.
Switch to MAX nightly buildsβ
Nightly builds are useful when you want to work with the latest features or fixes that are not yet part of the official release. These builds can help you stay on the cutting edge, but they might come with more risks in terms of stability. If youβre developing a project where you need the latest functionality or youβre contributing to the Mojo standard library development, working with nightly builds can give you early access to new improvements.
To use the nightly builds, you can either initialize a project with a nightly channel:
magic init hello-magic-nightly --format mojoproject -c conda-forge -c https://conda.modular.com/max-nightly
magic init hello-magic-nightly --format mojoproject -c conda-forge -c https://conda.modular.com/max-nightly
Or change an existing project channel to use https://conda.modular.com/max-nightly
instead of
https://conda.modular.com/max
.
If we inspect the mojoproject.toml
, we should see the following changes:
[project]
...
channels = ["conda-forge", "https://conda.modular.com/max-nightly"]
name = "hello-magic-nightly"
...
[dependencies]
max = ">=24.6.0.dev2024090905,<25"
[project]
...
channels = ["conda-forge", "https://conda.modular.com/max-nightly"]
name = "hello-magic-nightly"
...
[dependencies]
max = ">=24.6.0.dev2024090905,<25"
Pin or wildcard a dependency versionβ
Magic allows us to pin a particular dependency upon availability as follows:
[dependencies]
max = "==24.6.0.dev2024090905"
[dependencies]
max = "==24.6.0.dev2024090905"
This ensures only version 24.6.0.dev2024090905
is used in our project.
It is also possible to use the wildcard "*"
for a dependency version too:
[dependencies]
max = "*"
[dependencies]
max = "*"
This allows magic
CLI to pick up the latest available MAX version upon invocation.
It is among the best practices to either pin to a particular version or provide a lower and upper bound for versions to avoid sudden breaking changes.
Step 2: Run Mojo via Magicβ
Next, let's see how to run a Mojo code via Magic. From the root hello-magic
repository,
create touch hello.mojo
file with this content:
def main():
print("hello, magic!")
def main():
print("hello, magic!")
To run mojo
via the magic
CLI:
magic run mojo hello.mojo
magic run mojo hello.mojo
Alternatively, the magic shell
command opens an interactive shell within the Magic environment.
This environment ensures that all your dependencies are correctly configured, and you can run commands
like mojo
directly without worrying about manually activating environments.
Think of this as an isolated workspace that automatically sets everything up for you.
This is especially useful when working on larger projects where precise control over environments is crucial.
magic shell
magic shell
and then we can run the mojo
command as usual:
mojo hello.mojo
mojo hello.mojo
To exit from the shell, simply run:
exit
exit
Set a task command for ease of useβ
You can define tasks in mojoproject.toml
.
This allows you to reuse common commands by simply referring to task names.
Add the task to your project either from CLI:
magic task add hello mojo hello.mojo
magic task add hello mojo hello.mojo
Or directly include the following in your mojoproject.toml
[tasks]
hello = "mojo hello.mojo"
[tasks]
hello = "mojo hello.mojo"
Now, we can run the task by its name:
magic run hello
magic run hello
Include more platformsβ
We can adjust the platforms
part of mojoproject.toml
to include more supported platforms:
platforms = ["osx-arm64", "linux-64"]
platforms = ["osx-arm64", "linux-64"]
Clean up the environmentβ
Magic uses virtual environment to manage dependencies and execute our application code. The environment
used is located under .magic/envs
. This location is not configurable. To learn more,
please check out the details in pixi environment.
To remove any outdated dependencies or reset the environment for a fresh start, use the following command to clean up the local environment:
magic clean
magic clean
Step 3: Add dependenciesβ
If you have followed along, make sure you navigate to the hello-magic
. Dependencies are critical to any project.
In magic, you can easily add them via channels. For example, to add Python as a dependency:
magic add python
magic add python
Or with a particular lower and upper bound versions:
magic add "python>=3.8,<3.12"
magic add "python>=3.8,<3.12"
This will include python
as dependencies section of mojoproject.toml
:
[dependencies]
max = ">=24.4.0,<25"
python = ">=3.8,<3.12"
[dependencies]
max = ">=24.4.0,<25"
python = ">=3.8,<3.12"
Magic prioritizes conda packages by default over other options such as PyPI that we will see an example of later.
Include and customize channelsβ
Channels in Magic define where your packages come from. By default, Magic uses popular channels like conda-forge to install dependencies that are not part of MAX. We can add other channels if you need more specific packages or versions. Channels are also prioritized, meaning that if a package exists in multiple channels, Magic will install it from the highest-priority channel. This is particularly useful when you need certain optimizations (e.g., official PyTorch builds) or when you're working in an organization with custom packages. Managing your channels carefully ensures that the correct versions and packages are always installed.
To better understand the role of channel, let's run magic add pytorch
.
By default, this will install pytorch from the conda-forge
channel which we can
verify where it's installed from using the search command:
magic search pytorch
magic search pytorch
We can include a new channel either from the CLI:
magic project channel add pytorch
magic project channel add pytorch
Or manually append it to the existing list:
channels = ["conda-forge", "https://conda.modular.com/max", "pytorch"]
channels = ["conda-forge", "https://conda.modular.com/max", "pytorch"]
To make the channel prioritization happen we need to remove and add pytorch
again:
magic remove pytorch
magic remove pytorch
then
magic add pytorch
magic add pytorch
Inspecting the package metadata magic search pytorch
will show that it is coming from the official PyTorch channel.
Step 4: Add PyPI dependenciesβ
Magic also supports adding packages from PyPI. This is useful especially when a package is not available as a conda package.
For the sake of example, we install fastapi
using PyPI as follows:
magic add --pypi "fastapi[standard]"
magic add --pypi "fastapi[standard]"
This for example will add a separate [pypi-dependencies]
section in our mojoproject.toml
:
[pypi-dependencies]
fastapi = { version = ">=0.114.0, <0.115", extras = ["standard"] }
[pypi-dependencies]
fastapi = { version = ">=0.114.0, <0.115", extras = ["standard"] }
To test, create a main.py
file with the following content:
from fastapi import FastAPI
app = FastAPI()
@app.get("/")
def root():
return {"message": "Hello, magic!"}
from fastapi import FastAPI
app = FastAPI()
@app.get("/")
def root():
return {"message": "Hello, magic!"}
Now to run the app using fastapi
CLI, execute the following command:
magic run python -m fastapi dev main.py
magic run python -m fastapi dev main.py
Or
magic run fastapi dev main.py
magic run fastapi dev main.py
and follow the prompt. We should see the server is running http://127.0.0.1:8000 which we can open it a browser and we should see the output:
{"message": "Hello, magic!"}
{"message": "Hello, magic!"}
To save in typing, we can add the following task to enable magic run dev-server
:
[tasks]
hello = "mojo hello.mojo"
dev-server = "fastapi dev main.py"
[tasks]
hello = "mojo hello.mojo"
dev-server = "fastapi dev main.py"
Step 5: Use a nested packageβ
Magic makes it easy to include local packages in your project. For example, you can write a local Mojo script and use it within your FastAPI application.
First, create a new local package within our hello-magic
:
magic init local --format mojoproject
magic init local --format mojoproject
and include pytorch
as the first channel like we did before with this code:
channels = ["pytorch", "conda-forge", "https://conda.modular.com/max"]
channels = ["pytorch", "conda-forge", "https://conda.modular.com/max"]
And include the following Mojo code to local/zero.mojo
:
from python import Python, PythonObject
def zero() -> PythonObject:
torch = Python.import_module("torch")
return torch.zeros(1)
def main():
print(zero())
from python import Python, PythonObject
def zero() -> PythonObject:
torch = Python.import_module("torch")
return torch.zeros(1)
def main():
print(zero())
Now, include the following to main.py
:
import subprocess
from fastapi import FastAPI, HTTPException
@app.get("/zero")
def zero():
try:
p = subprocess.Popen(
["magic", "run", "mojo", "local/zero.mojo"],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
)
while True:
output = p.stdout.readline()
if output == "" and p.poll() is not None:
raise HTTPException(
status_code=500, detail="Failed to produce zero"
)
return {"message": f"answer is {output}"}
except subprocess.SubprocessError as e:
raise HTTPException(status_code=500, detail="Failed to execute subprocess")
import subprocess
from fastapi import FastAPI, HTTPException
@app.get("/zero")
def zero():
try:
p = subprocess.Popen(
["magic", "run", "mojo", "local/zero.mojo"],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
)
while True:
output = p.stdout.readline()
if output == "" and p.poll() is not None:
raise HTTPException(
status_code=500, detail="Failed to produce zero"
)
return {"message": f"answer is {output}"}
except subprocess.SubprocessError as e:
raise HTTPException(status_code=500, detail="Failed to execute subprocess")
and invoke magic run dev-server
navigate to the /zero
endpoint
http://127.0.0.1:8000/zero. We should see
{"message":"answer is tensor([0.])\n"}
{"message":"answer is tensor([0.])\n"}
Above, we are running magic run mojo local/zero.mojo
from a subprocess.
Another way is to first build the binary separately with magic run mojo build local/zero.mojo
and run it with magic run bash local/zero
. This better takes advantage of the Mojo compiler whereas mojo local/zero.mojo
uses the just-in-time (JIT) feature of the Mojo compiler.
Step 6: Setup a test environmentβ
Testing is crucial in development, especially when dealing with complex dependencies. Using Magic, you can set up a dedicated testing environment that isolates your testing dependencies from your development dependencies. This ensures that your development environment remains clean and focused, while your test environment has everything it needs to run unit tests, integration tests, etc. Isolating these environments also helps prevent any accidental conflicts or issues during testing.
To add a specific testing dependencies in a dedicated environment using Magic, first run:
magic task add test "pytest" --feature test
magic task add test "pytest" --feature test
which includes the following:
[feature.test.tasks]
test = "pytest"
[feature.test.tasks]
test = "pytest"
Then we need to explicitly add the default
environment:
magic project environment add default --solve-group default
magic project environment add default --solve-group default
After than, we can include the test
environment via:
magic project environment add test --feature test --solve-group default
magic project environment add test --feature test --solve-group default
This adds the following configuration:
[environments]
default = { solve-group = "default" }
test = { features = ["test"], solve-group = "default" }
[environments]
default = { solve-group = "default" }
test = { features = ["test"], solve-group = "default" }
Finally, add pytest
as a dependency for the test environment via --feature
:
magic add pytest --pypi --feature test
magic add pytest --pypi --feature test
which includes the following in mojoproject.toml
:
[feature.test.pypi-dependencies]
pytest = ">=8.3.2,<9"
[feature.test.pypi-dependencies]
pytest = ">=8.3.2,<9"
We can now run our tests:
magic run test
magic run test
This is a simple way to maintain a clean separation between development and test environments, ensuring reproducibility and minimizing conflicts.
Best practices for using Magicβ
To ensure smooth project management with Magic, here are a few best practices:
- Prefer conda packages over PyPI: PyPI packages may be less stable than their conda counterparts.
- Pin dependencies: For production environments, always try to pin dependencies to specific versions or at least provide upper and lower bounds with specific versions to avoid unexpected changes that could break your project.
- Use multi-environments: Maintain separate environments for development, testing, and production to keep dependencies organized and prevent conflicts.
- Regular cleanup: Regularly use the
magic clean
command to clean up your environments and keep your project running smoothly. This is particularly useful when working with multiple environments or testing new dependencies to avoid clutter.
Next stepsβ
You've now completed the essential steps to get started with Magic.
With its powerful dependency management, task automation, and multi-environment capabilities,
Magic helps streamline the development process for Mojo and MAX projects.
From setting up projects to managing complex dependencies across different platforms,
magic
CLI is a tool designed to handle it all.
As you continue exploring its features, you'll unlock even more ways to simplify your workflow.
- Discover more features of Magic by looking at the help manual
magic -h
and the documentations. - Check out the frequently asked questions section.
- Explore more Pixi project configuration and Pixi configuration.
Here are some other topics to explore next:
Run inference with MAX Engine Python API
Learn how to run inference with MAX Engine.
Explore the use of Magic in MAX examples
Explore varieties of MAX examples.
Report feedback, including issues on our MAX GitHub tracker.