hubdl

English | 简体中文

hubdl is a multi-hub model and dataset downloader for Linux servers, distributed on PyPI.

It combines:

• Hugging Face listing + aria2 workflow (inspired by hfd)
• ModelScope API listing (from battle-tested local msd tooling)
• A shared core: integrity-aware resume, hard rate limits, path sandbox, directory flock, L3 SHA-256 and L4 repository reports

Disclaimer: hubdl is not affiliated with Hugging Face, Inc. or ModelScope / Alibaba DAMO.
You are responsible for complying with each model's license and each Hub's terms of service.

Install & use

Official install: create a venv, then install from PyPI. Install system aria2 for downloads.
Do not install into the system/global Python (sudo pip is discouraged).

# 1) System packages (Debian / Ubuntu)
sudo apt update
sudo apt install -y python3 python3-venv aria2 ca-certificates

# 2) Install from PyPI into a venv
python3 -m venv .venv
source .venv/bin/activate          # Windows: .\.venv\Scripts\Activate.ps1
pip install -U pip
pip install hubdl

# 3) Verify
hubdl --version
hubdl hf gpt2 --local-dir ./gpt2 --dry-run

Package: https://pypi.org/project/hubdl/ · CLI entry points: hubdl, hfdx, msd

Table of contents

• Install & use
• Features
• CLI entry points
• Requirements
• Installation
• Quick start
• Architecture
• CLI reference
• Profiles
• Workflow & modes
• Integrity (L0–L4)
• Delete policy
• Metadata directory
• Environment variables
• Hub differences
• Security
• Troubleshooting
• Development
• Roadmap
• Credits & license

Features

CLI entry points

Also accepted: hubdl huggingface …, hubdl modelscope ….

Current version: 0.1.2 (PyPI · CHANGELOG)
Metadata directory: <local-dir>/.hubdl/
Manifest format: hubdl-manifest-v1 — size TAB sha256|- TAB path

Requirements

System (Linux recommended)

# Debian / Ubuntu
sudo apt update
sudo apt install -y python3 python3-venv python3-pip aria2 ca-certificates
# optional
sudo apt install -y wget

Python

No third-party runtime dependencies (stdlib only). Dev extras: pytest, pytest-cov.

Installation

Release process for maintainers: docs/RELEASE.md · docs/PUBLISHING.md.

From PyPI (official)

sudo apt install -y python3 python3-venv aria2 ca-certificates   # Linux, once
python3 -m venv .venv
source .venv/bin/activate   # Windows: .\.venv\Scripts\Activate.ps1
pip install -U pip
pip install hubdl
hubdl --version
hfdx --help
msd --help

• PyPI: https://pypi.org/project/hubdl/
• Console scripts: hubdl, hfdx, msd
• Python runtime dependencies: none (stdlib only). Downloader: system aria2c.

From a GitHub Release wheel (offline)

Download the .whl from Releases:

python3 -m venv .venv && source .venv/bin/activate
pip install -U pip
pip install ./hubdl-0.1.2-py3-none-any.whl
hubdl --version

From Git (optional)

python3 -m venv .venv && source .venv/bin/activate
pip install -U pip
pip install "git+https://github.com/mrhegit/hubdl.git@v0.1.2"
# or track main:
# pip install "git+https://github.com/mrhegit/hubdl.git"
hubdl --version

Editable install (contributors)

git clone https://github.com/mrhegit/hubdl.git && cd hubdl
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
hubdl --version
pytest -q

Without activating the venv: .venv/bin/hubdl (Linux) or .venv\Scripts\hubdl.exe (Windows).
.venv/ is gitignored and must not be committed.

Quick start

Assumes Install & use is done and the venv is active (hubdl --version works).

Hugging Face

# Plan only (no download, no delete)
hubdl hf gpt2 --local-dir ./gpt2 --dry-run
# equivalent alias
hfdx gpt2 --local-dir ./gpt2 --dry-run

# Full download (L3 default = auto)
hubdl hf Qwen/Qwen2.5-0.5B-Instruct --local-dir ./qwen05

# Private / gated model
export HF_TOKEN=hf_xxxxxxxx
hubdl hf meta-llama/Llama-3.2-1B \
  --local-dir ./llama \
  --hf_username YOUR_HF_NAME

# Mirror endpoint
export HF_ENDPOINT=https://hf-mirror.com
hubdl hf gpt2 --local-dir ./gpt2

ModelScope

hubdl ms Qwen/Qwen2.5-0.5B-Instruct --local-dir ./qwen05 --dry-run
# equivalent alias
msd Qwen/Qwen2.5-0.5B-Instruct --local-dir ./qwen05 --dry-run

# Shared LAN: hard cap 35 Mb/s, single connection
hubdl ms deepseek-ai/DeepSeek-V4-Flash-0731 \
  --local-dir /data1/models/DeepSeek-V4-Flash-0731 \
  --profile lan-shared

Verify only

# Re-check integrity (verify-only maps auto → full)
hubdl hf owner/model --local-dir ./m --verify-only --verify-hash on
hubdl ms owner/model --local-dir ./m --verify-only --offline

Background job (nohup)

mkdir -p /data1/model_download_logs
LOG=/data1/model_download_logs/hubdl_$(date +%Y%m%d_%H%M%S).log
nohup hubdl ms deepseek-ai/DeepSeek-V4-Flash-0731 \
  --local-dir /data1/models/DeepSeek-V4-Flash-0731 \
  --profile lan-shared \
  --progress none \
  > "$LOG" 2>&1 &
echo $! > /data1/model_download_logs/hubdl.pid

Architecture

Option A: dual CLI + shared library.

hubdl/
├── src/hubdl/
│   ├── cli.py                 # hubdl / hfdx / msd entrypoints
│   ├── core/                  # shared pipeline
│   │   ├── sandbox.py         # path jail, reserved dirs
│   │   ├── plan.py            # size + optional L3
│   │   ├── verify.py          # L3 + L4 report
│   │   ├── download.py        # aria2 / wget
│   │   ├── cache.py           # hash_cache.tsv
│   │   ├── httputil.py        # host pin, retries
│   │   └── pipeline.py        # orchestration
│   └── hubs/
│       ├── huggingface.py     # list + resolve URLs
│       └── modelscope.py
├── scripts/                   # repo wrappers without install
└── tests/

CLI reference

Positional

Common options

Rate limit note

Spoken “35 megabit/s” → --rate-mbps 35 → roughly 4480K in aria2 (N × 1024 / 8 KiB/s).

Profiles

Workflow & modes

download (default)

fetch list → plan (size + optional L3)
           → scrub orphan .aria2 (size-ok payload) → re-plan if needed
           → download round=1 → plan
           → if still need: download round=2 → plan
           → verify L3+L4

ModelScope multi-conn notes (aria2 only):

dry-run

fetch → plan → exit 0

verify-only

fetch | offline → verify → exit 0/2

(auto verify-mode becomes full so stale plan.json is not trusted lightly.)

Exit codes

Integrity (L0–L4)

--verify-hash auto (HF default): enable L3 if any of:

• share of files with sha ≥ 30%, or
• all files ≥ 20 MiB have sha, or
• files covering top ~90% of total bytes all have sha

ModelScope default is on (API usually provides Sha256).

Files without a digest are recorded as size_ok_nohash when L3 is enabled for the run.

Delete policy

Deletes only occur inside --local-dir after sandbox checks.

Metadata directory

Path: <local-dir>/.hubdl/ (mode preferably 0700).

Do not publish or artifact-upload .hubdl/ without scrubbing secrets.

Environment variables

Proxy notes

Priority: CLI --proxy > HUBDL_PROXY > https_proxy / http_proxy / ALL_PROXY.

Only HTTP(S) forward proxies are supported (aria2 / urllib / wget common capability).
SOCKS is rejected — point hubdl at a local HTTP port (Clash/V2Ray mixed / HTTP port), not socks5://.

# Recommended: local HTTP proxy (API + download, full path)
export https_proxy=http://127.0.0.1:7890
hubdl hf gpt2 --local-dir ./gpt2

# Explicit CLI (overrides env)
hubdl ms org/model --local-dir ./m --proxy http://127.0.0.1:7890 --no-proxy localhost,127.0.0.1

# Force direct when the shell has a global proxy
hubdl hf gpt2 --local-dir ./gpt2 --proxy none
# same: export https_proxy=none

• Child downloaders do not inherit ambient proxy env; hubdl injects or clears explicitly.
• aria2 also clears protocol-specific options so conf/https_proxy cannot override --proxy none.
• Credentials in the proxy URL are redacted in logs (still visible to same-uid ps); prefer unauthenticated localhost proxies.
• last_run.json stores scheme only (http / https / none).

Hub differences

Capabilities are not 100% symmetric; that is intentional.

Security

Summary (see also SECURITY.md):

1. Sandbox: repository paths cannot target .hubdl, .hfdx, .msd, .hfd, or .git.
2. HTTP pin: requests must stay on the configured endpoint host; cross-host redirects are blocked (reduces token exfiltration).
3. aria2: --check-integrity=true; URL list is wiped after download.
4. Tokens: prefer env vars; do not commit tokens or pack .hubdl/ into CI artifacts.
5. Mirrors: a mirror can still observe your token on the first hop — treat mirrors as trusted parties.
6. fix_bad: can delete overflow/hash-bad files inside local_dir only; use --no-fix-bad when unsure.

Troubleshooting

tail -f <local-dir>/.hubdl/download.log
cat <local-dir>/.hubdl/last_run.json
cat <local-dir>/.hubdl/verify_report.json
cat <local-dir>/.hubdl/plan.json

Development

source .venv/bin/activate   # or Windows Activate.ps1
pip install -e ".[dev]"
pytest -q
# with coverage:
pytest --cov=hubdl --cov-report=term-missing
python -m hubdl hf --help

Project layout for contributors mirrors the Architecture section.

Tests

pytest -q
pytest --cov=hubdl --cov-report=term-missing

CI runs on Python 3.10–3.12 (GitHub Actions). Before a PR:

1. pytest -q
2. hubdl hf gpt2 --dry-run --local-dir /tmp/hubdl-gpt2 (network)
3. hubdl ms Qwen/Qwen2.5-0.5B-Instruct --dry-run --local-dir /tmp/hubdl-qwen --no-verify-hash

Roadmap

• Shared core + HF / MS backends
• Dual CLI aliases (hfdx, msd)
• Official PyPI distribution (pip install hubdl)
• CI + Trusted Publishing (OIDC) to PyPI / GitHub Release
• Stronger single-line progress HUD (hfd-style)
• Optional migration from legacy .msd / .hfdx meta dirs
• Broader integration tests against mocked Hub APIs
• Optional standalone binary packaging

Credits & license

Inspired by / thanks to:

• hfd by padeoe — Hugging Face listing + aria2 workflow
• Local msd / hfdx tooling — L3/L4, sandbox, operational flags

License: Apache License 2.0
Attribution: NOTICE
Security policy: SECURITY.md

Model weights and Hub content remain under their respective licenses.

Language