MirrorURL
MirrorURL is a Python command-line tool and library for mirroring and incrementally syncing public HTTP(S) directory trees to local disk. It also downloads individual file URLs or exact URL lists, with resumable transfers, parallel downloads, caching and integrity checks. URL-scope, private-network and filesystem protections guard discovery and transfers.
See the changelog for release notes.
Who it is for
MirrorURL is for people and organizations that regularly need to maintain local copies of files published over public HTTP(S):
- Researchers maintaining local copies of scientific data archives.
- Organizations downloading bulk files from public directory indexes.
- Maintainers of software, package, artifact and documentation mirrors.
- Sysadmins and data engineers maintaining local datasets.
- Public-data preservation communities.
- Users moving from
wget --mirror,lftp mirroror custom scripts.
Scientific mission archives used in the examples illustrate workflows that also apply to these other sources. Recursive mirroring requires an HTML directory index; direct file and URL-list downloads do not require a listing.
Features
- Recursive discovery of remote directory trees (BFS, depth/exclude limits, cycle-safe).
- True parallel downloads — multiple files and multiple chunks per file concurrently.
- Adaptive async concurrency that tunes itself to server RTT, throughput, and error rate.
- Resumable & partial downloads with HTTP range requests and chunk assembly.
- Integrity checks — size/timestamp comparison, ETag handling, verified byte ranges, and opt-in
--verify-contentSHA-256 checks of local files; no comparison against a remote cryptographic content digest. - Resilience — per-domain circuit breakers, exponential backoff, rate limiting.
- Destination ownership — cooperating processes reject overlapping local trees and shared cache/state paths; hard process termination releases ownership automatically.
- Crash recovery — the next owner reclaims recorded abandoned parallel chunks and staging under reserved state; live writers, unknown files and legitimate MOVE archives are preserved.
- Security — path-traversal and symlink-bomb defenses, private-IP/SSRF guards, URL-scope enforcement.
- Filename preflight — preserve distinct original names on a confirmed case-sensitive destination; reject case collisions on a case-insensitive destination before downloads. Rewritten, unsafe and Unicode-aliased paths remain blocked; see download behavior and preserving original names.
- Operability — metrics collection, multi-level progress, optional HTTP health-check server.
- Caching — directory listings and file metadata; discovery currently keeps the remote file list in memory.
- Known-URL downloads —
--mode download FILE_URLdownloads one file, or--mode download --url-list urls.txtstreams an exact list, without discovery or freshness probes. Both HTTPX and optional aiohttp use the same destination, scope, staging and receipt checks. - Selected freshness checks — combine
--missing-fileswith--check-files path1 path2or--check-files @list.txtto refresh selected existing files while downloading all missing files in scope. Paths stay relative to--url, including any suffix. - Explicit pacing —
--requests-per-secondand--request-delaycontrol the request budget independently of security checks.
Installation
Python 3.10 or newer is required. Install the published package from PyPI in a virtual environment:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install mirror-url
To include all optional acceleration, progress, and monitoring packages:
python -m pip install "mirror-url[all]"
Upgrade an existing installation:
python -m pip install --upgrade mirror-url
Check the installed package and CLI:
python -m pip check
mirror-url --version
mirror-url --help
Core dependencies: httpx[http2], pydantic (v2), PyYAML, portalocker
(including pywin32 on Windows). Optional extras:
fast (stringzilla, lxml), progress (tqdm), monitor (psutil),
aiohttp (the whole-file backend), and all
(all optional runtime packages). Contributor setup is described below.
Usage
For an exact, pre-built list of URLs:
# Install the optional backend in your virtual environment.
python -m pip install "mirror-url[aiohttp]"
python -m mirror_url --mode download --backend aiohttp \
--url https://example.org/files/ --url-list urls.txt \
--dest-path ./downloads --log-path ./logs --concurrency 20 \
--requests-per-second 0 --request-delay 0 --verify-content
For a single file URL:
mirror-url --mode download https://example.org/files/a.fits
# Optional destination and log paths:
mirror-url --mode download https://example.org/files/a.fits \
--dest-path ./downloads --log-path ./logs
The shortcut defaults to the current directory, with logs in a destination-specific
system temporary folder. It infers the file URL's parent as the allowed remote
scope; an explicit --url overrides that scope. Choose either a positional file
URL or --url-list, and keep --mode download for both.
urls.txt contains one absolute URL per line below the selected base URL.
The two zero pacing values explicitly remove the default 20 requests/second
ceiling and 50 ms spacing. TLS verification, public-IP DNS validation, redirect
scope, safe filename mapping and atomic publication stay enabled. Existing
files need a matching ownership receipt or explicit --overwrite. See
known-URL downloads for limits
and the HTTPX equivalent.
Run via the console entry point or the module:
mirror-url --url https://example.com/files/ --dest-path ./mirror --log-path ./mirror-log
# or
python -m mirror_url --url https://example.com/files/ --dest-path ./mirror --log-path ./mirror-log
Run mirror-url --help for the full option list.
Configuration can also be supplied via a YAML file (see MirrorConfig /
load_config_from_args).
📖 Full documentation: see the User Guide (HTML version) for detailed installation, CLI reference, config-file format, download modes, security notes, the Python API, and troubleshooting.
🛠 Contributing to the code? The Developer Guide
(HTML version) is an architecture deep-dive:
dependency layers, the MirrorURL mixin design, runtime data flow, and
step-by-step extension recipes.
Development
From a repository checkout, create and activate a virtual environment, then install the editable package with its runtime extras and development tools:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[all,dev]"
pre-commit install
ruff check . # lint
ruff format --check . # format check (Ruff 0.16.10)
mypy # type-check the package
pytest -m "not integration" # fast test lane
pytest # full suite (includes integration)
Continuous integration runs lint + tests across Python 3.10–3.14 (see
.github/workflows/ci.yml).
Project layout
src/mirror_url/ # the package (dependency-layered)
tests/ # pytest suite
REFACTORING_PLAN.md # module breakdown + migration roadmap
CHANGELOG.md # notable changes (Keep a Changelog format)
CONTRIBUTING.md # dev setup, checks, conventions
pyproject.toml # packaging, deps, tool config
Contributing
Contributions are welcome — see CONTRIBUTING.md for the dev setup, the checks CI runs, and the project conventions (dependency layering, lint policy, behavior-preserving refactors). Notable changes are tracked in CHANGELOG.md.
Authors
Borut Podlipnik, Max-Planck-Institute for Solar System Research, podlipnik@mps.mpg.de
License
MIT © BP
1 maintainer
Metadata
1 maintainer
Release files for mirror-url 3.3.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mirror_url-3.3.2.tar.gz | 567.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| mirror_url-3.3.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 806.6 kB
Release files / mirror_url-3.3.2.tar.gz
| Download URL | mirror_url-3.3.2.tar.gz |
|---|---|
| Size | 567.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e671038a9d494c7caab7ad694bc9e0a89503c3e778e33da532c3d07f979ade52
|
|
BLAKE2b-256 checksum How to use checksums |
7485adaadd571dae0a554f65dbeebd714677f81828b872425bbd344c0a9efc65
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 10, 2026.
Transparency logRelease files / mirror_url-3.3.2-py3-none-any.whl
| Download URL | mirror_url-3.3.2-py3-none-any.whl |
|---|---|
| Size | 238.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
50935d53c70174dcfcb9e0d5a842ee7411b878fbdda385af1dee29dd1c15f824
|
|
BLAKE2b-256 checksum How to use checksums |
87b300c4fa682ba3a842ed28d33dc2d4dfb42809b71cca1dbf8d9d34a25b01b2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 10, 2026.
Transparency log
