MirrorURL

MirrorURL is a Python command-line tool and library for mirroring and incrementally syncing public HTTP(S) directory trees to local disk. It also downloads individual file URLs or exact URL lists, with resumable transfers, parallel downloads, caching and integrity checks. URL-scope, private-network and filesystem protections guard discovery and transfers.

See the changelog for release notes.

Who it is for

MirrorURL is for people and organizations that regularly need to maintain local copies of files published over public HTTP(S):

Researchers maintaining local copies of scientific data archives.
Organizations downloading bulk files from public directory indexes.
Maintainers of software, package, artifact and documentation mirrors.
Sysadmins and data engineers maintaining local datasets.
Public-data preservation communities.
Users moving from wget --mirror, lftp mirror or custom scripts.

Scientific mission archives used in the examples illustrate workflows that also apply to these other sources. Recursive mirroring requires an HTML directory index; direct file and URL-list downloads do not require a listing.

Features

Recursive discovery of remote directory trees (BFS, depth/exclude limits, cycle-safe).
True parallel downloads — multiple files and multiple chunks per file concurrently.
Adaptive async concurrency that tunes itself to server RTT, throughput, and error rate.
Resumable & partial downloads with HTTP range requests and chunk assembly.
Integrity checks — size/timestamp comparison, ETag handling, verified byte ranges, and opt-in --verify-content SHA-256 checks of local files; no comparison against a remote cryptographic content digest.
Resilience — per-domain circuit breakers, exponential backoff, rate limiting.
Destination ownership — cooperating processes reject overlapping local trees and shared cache/state paths; hard process termination releases ownership automatically.
Crash recovery — the next owner reclaims recorded abandoned parallel chunks and staging under reserved state; live writers, unknown files and legitimate MOVE archives are preserved.
Security — path-traversal and symlink-bomb defenses, private-IP/SSRF guards, URL-scope enforcement.
Filename preflight — preserve distinct original names on a confirmed case-sensitive destination; reject case collisions on a case-insensitive destination before downloads. Rewritten, unsafe and Unicode-aliased paths remain blocked; see download behavior and preserving original names.
Operability — metrics collection, multi-level progress, optional HTTP health-check server.
Caching — directory listings and file metadata; discovery currently keeps the remote file list in memory.
Known-URL downloads — --mode download FILE_URL downloads one file, or --mode download --url-list urls.txt streams an exact list, without discovery or freshness probes. Both HTTPX and aiohttp use the same destination, scope, staging and receipt checks.
Selected freshness checks — combine --missing-files with --check-files path1 path2 or --check-files @list.txt to refresh selected existing files while downloading all missing files in scope. Paths stay relative to --url, including any suffix.
Parallel chunks with either backend — --parallel-downloads and --streaming-parallel use the same verified range engine with HTTPX or aiohttp, in mirror and download modes.
Explicit pacing — --requests-per-second and --request-delay control the request budget independently of security checks.

Installation

Python 3.10 or newer is required. Install the published package from PyPI in a virtual environment:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install mirror-url

To include all optional acceleration and monitoring packages:

python -m pip install "mirror-url[all]"

Upgrade an existing installation:

python -m pip install --upgrade mirror-url

Check the installed package and CLI:

python -m pip check
mirror-url --version
mirror-url --help

Core dependencies: httpx[http2], aiohttp, pydantic (v2), PyYAML, portalocker (including pywin32 on Windows), and tqdm for progress bars. Optional extras: fast (stringzilla, lxml), monitor (psutil), and all (all optional runtime packages). The progress and aiohttp extras remain compatibility aliases; normal installations include both backends and progress bars. Contributor setup is described below.

Usage

For an exact, pre-built list of URLs:

# Both backends are included in the standard installation.
python -m mirror_url --mode download --backend aiohttp \
  --url https://example.org/files/ --url-list urls.txt \
  --dest-path ./downloads --log-path ./logs --concurrency 20 \
  --requests-per-second 0 --request-delay 0 --verify-content

For a single file URL:

mirror-url --mode download https://example.org/files/a.fits
# Optional destination and log paths:
mirror-url --mode download https://example.org/files/a.fits \
  --dest-path ./downloads --log-path ./logs

The shortcut defaults to the current directory, with logs in a destination-specific system temporary folder. It infers the file URL's parent as the allowed remote scope; an explicit --url overrides that scope. Choose either a positional file URL or --url-list, and keep --mode download for both.

urls.txt contains one absolute URL per line below the selected base URL. The two zero pacing values explicitly remove the default 20 requests/second ceiling and 50 ms spacing. TLS verification, public-IP DNS validation, redirect scope, safe filename mapping and atomic publication stay enabled. Existing files need a matching ownership receipt or explicit --overwrite. See known-URL downloads for limits and the HTTPX equivalent.

Run via the console entry point or the module:

mirror-url --url https://example.com/files/ --dest-path ./mirror --log-path ./mirror-log
# or
python -m mirror_url --url https://example.com/files/ --dest-path ./mirror --log-path ./mirror-log

Run mirror-url --help for the full option list.

The public documentation page is maintained in index.html. After CLI or release changes, run python scripts/render_site.py to refresh its release references and generated argument table. CI checks it for drift, and each GitHub Release includes the page as an index.html asset for deployment.

Configuration can also be supplied via a YAML file (see MirrorConfig / load_config_from_args).

📖 Full documentation: see the User Guide (HTML version) for detailed installation, CLI reference, config-file format, download modes, security notes, the Python API, and troubleshooting.

🛠 Contributing to the code? The Developer Guide (HTML version) is an architecture deep-dive: dependency layers, the MirrorURL mixin design, runtime data flow, and step-by-step extension recipes.

Development

From a repository checkout, create and activate a virtual environment, then install the editable package with its runtime extras and development tools:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[all,dev]"
pre-commit install

ruff check .                  # lint
ruff format --check .         # format check (Ruff 0.16.10)
mypy                          # type-check the package
pytest -m "not integration"   # fast test lane
pytest                        # full suite (includes integration)

Continuous integration runs lint + tests across Python 3.10–3.14 (see .github/workflows/ci.yml).

Project layout

src/mirror_url/          # the package (dependency-layered)
tests/                   # pytest suite
REFACTORING_PLAN.md      # module breakdown + migration roadmap
CHANGELOG.md             # notable changes (Keep a Changelog format)
CONTRIBUTING.md          # dev setup, checks, conventions
pyproject.toml           # packaging, deps, tool config

Contributing

Contributions are welcome — see CONTRIBUTING.md for the dev setup, the checks CI runs, and the project conventions (dependency layering, lint policy, behavior-preserving refactors). Notable changes are tracked in CHANGELOG.md.

Authors

Borut Podlipnik, Max-Planck-Institute for Solar System Research, podlipnik@mps.mpg.de

License

MIT © BP

Metadata

Release files for mirror-url 3.8.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mirror-url 3.8.4
File	Size	Uploaded
mirror_url-3.8.4.tar.gz	681.7 kB	Oct 17, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for mirror-url 3.8.4
File	Interpreter	ABI	Platform	Reset
mirror_url-3.8.4-py3-none-any.whl 257.7 kB Oct 17, 2026	Python 3	none	any	Details

Total release size: 939.4 kB

Release files / mirror_url-3.8.4.tar.gz

Download URL	mirror_url-3.8.4.tar.gz
Size	681.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`50d3c3b77a736eb71d1819d60600f2b6c8a7970c4ae95dfe8a6badf4a76ffd82`
BLAKE2b-256 checksum How to use checksums	`8c51cbcebe61bdb901765b4154384ea1b539b64ae371581cadf630065552dac7`
Upload date	Oct 17, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/bpodlipnik/mirror-url
Publishing commit: github.com/bpodlipnik/mirror-url/tree/cd9a00026172f0d0bb77ce2e48ca2b20cbfa672f

Workflow

Publishing configuration: github.com/bpodlipnik/mirror-url/blob/cd9a00026172f0d0bb77ce2e48ca2b20cbfa672f/.github/workflows/release.yml
Publishing logs: github.com/bpodlipnik/mirror-url/actions/runs/39013548382/attempts/1

Artifact

Subject: mirror_url-3.8.4.tar.gz
SHA-256 checksum: 50d3c3b77a736eb71d1819d60600f2b6c8a7970c4ae95dfe8a6badf4a76ffd82 Verifying attestations

Release files / mirror_url-3.8.4-py3-none-any.whl

Download URL	mirror_url-3.8.4-py3-none-any.whl
Size	257.7 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`157233e8ff4e42103afd16426a8c6d4b3b449eb8e14bee18f29bdb34331b400b`
BLAKE2b-256 checksum How to use checksums	`885d3a41af6762abf0357df5b077e0b96a1df563daff0f041bda44103ee36cc8`
Upload date	Oct 17, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/bpodlipnik/mirror-url
Publishing commit: github.com/bpodlipnik/mirror-url/tree/cd9a00026172f0d0bb77ce2e48ca2b20cbfa672f

Workflow

Publishing configuration: github.com/bpodlipnik/mirror-url/blob/cd9a00026172f0d0bb77ce2e48ca2b20cbfa672f/.github/workflows/release.yml
Publishing logs: github.com/bpodlipnik/mirror-url/actions/runs/39013548382/attempts/1

Artifact

Subject: mirror_url-3.8.4-py3-none-any.whl
SHA-256 checksum: 157233e8ff4e42103afd16426a8c6d4b3b449eb8e14bee18f29bdb34331b400b Verifying attestations