MirrorURL
Security-hardened remote directory mirroring tool. MirrorURL recursively discovers files behind an HTTP(S) directory listing and mirrors them locally with adaptive concurrency, resumable/partial downloads, integrity verification, and an SSRF-hardened transport layer.
Status: Refactor complete. The full implementation lives in the
src/mirror_url/package, migrated verbatim from the original single-file script perREFACTORING_PLAN.md. The legacy single-file script was retained as a frozen reference until the test suite passed against the package with real runtime dependencies installed, then removed (v3.1.20).
Features
- Recursive discovery of remote directory trees (BFS, depth/exclude limits, cycle-safe).
- True parallel downloads — multiple files and multiple chunks per file concurrently.
- Adaptive async concurrency that tunes itself to server RTT, throughput, and error rate.
- Resumable & partial downloads with HTTP range requests and chunk assembly.
- Integrity checks — size/timestamp comparison, ETag handling, content hashing.
- Resilience — per-domain circuit breakers, exponential backoff, rate limiting.
- Security — path-traversal and symlink-bomb defenses, private-IP/SSRF guards, URL-scope enforcement.
- Operability — metrics collection, multi-level progress, optional HTTP health-check server.
- Caching — filesystem and disk-backed indexes to skip unchanged content.
Installation
# From source (editable, recommended during the refactor)
pip install -e ".[all,dev]"
Python 3.9+ is required. Core dependencies: httpx, pydantic (v2), PyYAML.
Optional extras: fast (stringzilla, lxml), progress (tqdm), monitor (psutil).
Usage
Run via the console entry point or the module:
mirror-url https://example.com/files/ --output ./mirror
# or
python -m mirror_url https://example.com/files/ --output ./mirror
Run mirror-url --help for the full option list.
Configuration can also be supplied via a YAML file (see MirrorConfig /
load_config_from_args).
📖 Full documentation: see the User Guide (HTML version) for detailed installation, CLI reference, config-file format, download modes, security notes, the Python API, and troubleshooting.
🛠 Contributing to the code? The Developer Guide
(HTML version) is an architecture deep-dive:
dependency layers, the MirrorURL mixin design, runtime data flow, and
step-by-step extension recipes.
Development
pip install -e ".[dev]"
pre-commit install
ruff check . # lint
black --check . # format check
mypy # type-check the new package
pytest -m "not integration" # fast test lane
pytest # full suite (includes integration)
Continuous integration runs lint + tests across Python 3.9–3.12 (see
.github/workflows/ci.yml).
Project layout
src/mirror_url/ # the package (dependency-layered)
tests/ # pytest suite
REFACTORING_PLAN.md # module breakdown + migration roadmap
CHANGELOG.md # notable changes (Keep a Changelog format)
CONTRIBUTING.md # dev setup, checks, conventions
pyproject.toml # packaging, deps, tool config
Contributing
Contributions are welcome — see CONTRIBUTING.md for the dev setup, the checks CI runs, and the project conventions (dependency layering, lint policy, behavior-preserving refactors). Notable changes are tracked in CHANGELOG.md.
Authors
Borut Podlipnik, Max-Planck-Institute for Solar System Research, podlipnik@mps.mpg.de
License
MIT © BP
1 maintainer
Release files for mirror-url 3.1.58
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mirror_url-3.1.58.tar.gz | 255.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| mirror_url-3.1.58-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 460.5 kB
Release files / mirror_url-3.1.58.tar.gz
| Download URL | mirror_url-3.1.58.tar.gz |
|---|---|
| Size | 255.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d74017dd5b68566ff1eb875874eefb5014001788b6c6f3c0ffbeafb6468ef3fb
|
|
BLAKE2b-256 checksum How to use checksums |
71348a10afbf0bb4cd36f14d441bba266fe8595f28c99c3668238ad542367e58
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 23, 2026.
Transparency logRelease files / mirror_url-3.1.58-py3-none-any.whl
| Download URL | mirror_url-3.1.58-py3-none-any.whl |
|---|---|
| Size | 205.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
08985b40e9522fc197d9833241eeb6db12b47490a60fbe743d9d1895a91cb1e8
|
|
BLAKE2b-256 checksum How to use checksums |
956da6335f74062302207c0e0315a8e6d622c5bcf98f213cc8cc92340079a56a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 23, 2026.
Transparency log
