Python module
hf_utils
Utilities for interacting with HuggingFace Files/Repos.
HuggingFaceRepo
class max.pipelines.lib.hf_utils.HuggingFaceRepo(repo_id, revision='main', trust_remote_code=False, repo_type=None)
A class for interacting with HuggingFace Repos.
encoding_for_file()
encoding_for_file(file)
-
Parameters:
-
file (str | Path)
-
Return type:
-
SupportedEncoding
file_exists()
file_exists(filename)
files_for_encoding()
files_for_encoding(encoding, weights_format=None)
-
Parameters:
-
- encoding (SupportedEncoding)
- weights_format (WeightsFormat | None)
-
Return type:
-
dict[WeightsFormat, list[Path]]
formats_available
property formats_available: list[WeightsFormat]
info
property info: ModelInfo
repo_id
repo_id: str
The HuggingFace repo id. While it’s called repo_id, it can be a HF remote or local path altogether.
repo_type
repo_type: RepoType | None = None
The type of repo. This is inferred from the repo_id.
revision
revision: str = 'main'
The revision to use for the repo.
size_of()
size_of(filename)
supported_encodings
property supported_encodings: list[SupportedEncoding]
trust_remote_code
trust_remote_code: bool = False
Whether to trust remote code.
weight_files
property weight_files: dict[WeightsFormat, list[str]]
download_weight_files()
max.pipelines.lib.hf_utils.download_weight_files(huggingface_model_id, filenames, revision=None, force_download=False, max_workers=8)
- Provided a HuggingFace model id, and filenames, download weight files
- and return the list of local paths.
-
Parameters:
-
- huggingface_model_id (str) – The huggingface model identifier, ie. modularai/Llama-3.1-8B-Instruct-GGUF
- filenames (list[str]) – A list of file paths relative to the root of the HuggingFace repo. If files provided are available locally, download is skipped, and the local files are used.
- revision (str | None) – The HuggingFace revision to use. If provided, we check our cache directly without needing to go to HuggingFace directly, saving a network call.
- force_download (bool) – A boolean, indicating whether we should force the files to be redownloaded, even if they are already available in our local cache, or a provided path.
- max_workers (int) – The number of worker threads to concurrently download files.
-
Return type:
-
list[Path]
generate_local_model_path()
max.pipelines.lib.hf_utils.generate_local_model_path(repo_id, revision)
Generate the local filesystem path where a HuggingFace model repo is cached.
This function uses HuggingFace’s official snapshot_download with local_files_only=True to resolve the local cache path for a model repository.
-
Parameters:
-
Returns:
-
The absolute path to the cached model files for the specified revision.
-
Return type:
-
Raises:
-
FileNotFoundError – If the model is not found in the local cache
is_diffusion_pipeline()
max.pipelines.lib.hf_utils.is_diffusion_pipeline(repo)
Check if a HuggingFace repository is a diffusion pipeline.
Diffusion pipelines typically have a model_index.json file that describes the pipeline components.
-
Parameters:
-
repo (HuggingFaceRepo) – The HuggingFaceRepo to check.
-
Returns:
-
True if the repository appears to be a diffusion pipeline, False otherwise.
-
Return type:
try_to_load_from_cache()
max.pipelines.lib.hf_utils.try_to_load_from_cache(repo_id, filename, revision)
Wrapper around huggingface_hub.try_to_load_from_cache. We also validate that the repo exists.
validate_hf_repo_access is called before this function to ensure the repo exists.
validate_hf_repo_access()
max.pipelines.lib.hf_utils.validate_hf_repo_access(repo_id, revision)
Validate repository access and raise clear, user-friendly errors.
Results are cached to avoid redundant HuggingFace API calls when the same repository is validated multiple times within a process.
-
Parameters:
-
Raises:
-
ValueError – With user-friendly error messages for various access issues
-
Return type:
-
None
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!