MirrorURL
Security-hardened remote directory mirroring tool. MirrorURL recursively discovers files behind an HTTP(S) directory listing and mirrors them locally with adaptive concurrency, resumable/partial downloads, integrity verification, and an SSRF-hardened transport layer.
Features
- Recursive discovery of remote directory trees (BFS, depth/exclude limits, cycle-safe).
- True parallel downloads — multiple files and multiple chunks per file concurrently.
- Adaptive async concurrency that tunes itself to server RTT, throughput, and error rate.
- Resumable & partial downloads with HTTP range requests and chunk assembly.
- Integrity checks — size/timestamp comparison, ETag handling, content hashing.
- Resilience — per-domain circuit breakers, exponential backoff, rate limiting.
- Security — path-traversal and symlink-bomb defenses, private-IP/SSRF guards, URL-scope enforcement.
- Operability — metrics collection, multi-level progress, optional HTTP health-check server.
- Caching — directory listings and file metadata; discovery currently keeps the remote file list in memory.
Installation
Python 3.9 or newer is required. Install the published package from PyPI in a virtual environment:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install mirror-url
To include all optional acceleration, progress, and monitoring packages:
python -m pip install "mirror-url[all]"
Upgrade an existing installation:
python -m pip install --upgrade mirror-url
Check the installed package and CLI:
python -m pip check
mirror-url --version
mirror-url --help
Core dependencies: httpx[http2], pydantic (v2), PyYAML. Optional extras:
fast (stringzilla, lxml), progress (tqdm), monitor (psutil), and all
(all optional runtime packages). Contributor setup is described below.
Usage
Run via the console entry point or the module:
mirror-url --url https://example.com/files/ --dest-path ./mirror --log-path ./mirror-log
# or
python -m mirror_url --url https://example.com/files/ --dest-path ./mirror --log-path ./mirror-log
Run mirror-url --help for the full option list.
Configuration can also be supplied via a YAML file (see MirrorConfig /
load_config_from_args).
📖 Full documentation: see the User Guide (HTML version) for detailed installation, CLI reference, config-file format, download modes, security notes, the Python API, and troubleshooting.
🛠 Contributing to the code? The Developer Guide
(HTML version) is an architecture deep-dive:
dependency layers, the MirrorURL mixin design, runtime data flow, and
step-by-step extension recipes.
Development
From a repository checkout, create and activate a virtual environment, then install the editable package with its runtime extras and development tools:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[all,dev]"
pre-commit install
ruff check . # lint
ruff format --check . # format check (Ruff 0.16.8)
mypy # type-check the package
pytest -m "not integration" # fast test lane
pytest # full suite (includes integration)
Continuous integration runs lint + tests across Python 3.9–3.12 (see
.github/workflows/ci.yml).
Project layout
src/mirror_url/ # the package (dependency-layered)
tests/ # pytest suite
REFACTORING_PLAN.md # module breakdown + migration roadmap
CHANGELOG.md # notable changes (Keep a Changelog format)
CONTRIBUTING.md # dev setup, checks, conventions
pyproject.toml # packaging, deps, tool config
Contributing
Contributions are welcome — see CONTRIBUTING.md for the dev setup, the checks CI runs, and the project conventions (dependency layering, lint policy, behavior-preserving refactors). Notable changes are tracked in CHANGELOG.md.
Authors
Borut Podlipnik, Max-Planck-Institute for Solar System Research, podlipnik@mps.mpg.de
License
MIT © BP
1 maintainer
Metadata
1 maintainer
Release files for mirror-url 3.1.75
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mirror_url-3.1.75.tar.gz | 315.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| mirror_url-3.1.75-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 526.9 kB
Release files / mirror_url-3.1.75.tar.gz
| Download URL | mirror_url-3.1.75.tar.gz |
|---|---|
| Size | 315.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c1f5f6a21e551346125b20fdd9ec01fa5cc02e6e93d0e9616cf53103d1b02734
|
|
BLAKE2b-256 checksum How to use checksums |
d952a8ddcda18f02abc5880c4ee8094b84cb637091834b9edc0fbe62a11fb443
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 2, 2026.
Transparency logRelease files / mirror_url-3.1.75-py3-none-any.whl
| Download URL | mirror_url-3.1.75-py3-none-any.whl |
|---|---|
| Size | 211.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d7b29d919696ce6d55d97d3a3ce1f6f124cb74b53cb8b3d9605604e0f7d6af2d
|
|
BLAKE2b-256 checksum How to use checksums |
ec1385b00e059ab3b8ccc997566472e8bb8d5db5dbea7540934a44692cc30591
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 2, 2026.
Transparency log
