dlmhub

A small, provider-agnostic model hub for Python libraries. dlmhub gives your users a MyClass.load_from_hub("org/model") API without hard-wiring your project to Hugging Face Hub (or any single provider).

Why

Many libraries want "download my pretrained files, cache them locally, load them" behavior, but end up tightly coupled to huggingface_hub. dlmhub extracts that pattern into a standalone, dependency-light package where:

• Providers are pluggable. Hugging Face and GitHub releases ship as built-ins; register your own (S3, a private server, git-lfs, ...) with a few lines.
• The cache directory is fully parametrable. Set it globally, per class, or per call — whichever fits your app.
• The mixin is the only requirement. Add HubMixin to any class, declare the files it needs, implement how to build an instance from them, and you're done.

Install

pip install -e .            # from a local checkout
# or, once published:
pip install dlmhub

Quickstart

from dlmhub import HubMixin

class MyTokenizer(HubMixin):
    files = ["vocab.json", "merges.txt"]

    @classmethod
    def _load_from_files(cls, local_files, **kwargs):
        # local_files: {"vocab.json": Path(...), "merges.txt": Path(...)}
        ...
        return cls(...)

tok = MyTokenizer.load_from_hub("org/model")                  # Hugging Face by default
tok = MyTokenizer.load_from_hub("org/repo", provider="github") # GitHub releases
tok = MyTokenizer.load_from_hub("org/model", variant="fp16")   # variant-prefixed files
tok = MyTokenizer.load_from_hub("org/model", token="hf_...")   # private/gated repos

Configuring the cache directory

Resolution order, highest priority first:

1. Per call: MyTokenizer.load_from_hub(..., cache_dir="/data/cache")
2. Per class: MyTokenizer.CACHE_DIR = Path("/data/cache")
3. Globally, in code: dlmhub.config.set_cache_dir("/data/cache")
4. Globally, via environment: DLMHUB_CACHE_DIR=/data/cache
5. Built-in default: ~/.cache/dlmhub

import dlmhub
dlmhub.config.set_cache_dir("/data/model_cache")

Adding a custom provider

from dlmhub.providers import Provider, default_registry

# Simple case: just a URL template with {model} and {file} placeholders.
default_registry.register(Provider(
    name="my-s3",
    url_pattern="https://my-bucket.s3.amazonaws.com/{model}/{file}",
))

# Full control: supply your own download logic (auth, SDKs, checksums...).
def _download_from_internal_server(model, file, dest_path, headers, **kwargs):
    ...  # write bytes to dest_path yourself

default_registry.register(Provider(
    name="internal",
    download_fn=_download_from_internal_server,
))

Need full isolation instead of touching the shared registry? Create your own ProviderRegistry() and assign it to MyTokenizer.PROVIDER_REGISTRY.

Compact hub-path strings

For CLIs/config files, a "provider:model[:variant]" string can be parsed with:

from dlmhub import process_hub_path

process_hub_path("huggingface:org/model:fp16")
# {"provider": "huggingface", "model": "org/model", "variant": "fp16"}

Project layout

src/dlmhub/
    __init__.py     public API surface
    config.py       global, overridable configuration (cache dir, ...)
    providers.py    Provider / ProviderRegistry, built-in HF + GitHub providers
    hub.py          HubMixin: load_from_hub() and download orchestration
    hubpath.py      "provider:model:variant" string parsing
    fileutils.py    small filesystem helpers (list files/subfolders, read lines)
    exceptions.py   DlmHubError and friends
tests/

License

Copyright (C) 2026 Abdelkrime Aries

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.