MirrorURL
Enterprise-grade remote directory mirroring tool. MirrorURL recursively discovers files behind an HTTP(S) directory listing and mirrors them locally with adaptive concurrency, resumable/partial downloads, integrity verification, and an SSRF-hardened transport layer.
Status: Refactor complete. The full implementation lives in the
src/mirror_url/package, migrated verbatim from the original single-file script perREFACTORING_PLAN.md. The legacy single-file script was retained as a frozen reference until the test suite passed against the package with real runtime dependencies installed, then removed (v3.1.20).
Features
- Recursive discovery of remote directory trees (BFS, depth/exclude limits, cycle-safe).
- True parallel downloads — multiple files and multiple chunks per file concurrently.
- Adaptive async concurrency that tunes itself to server RTT, throughput, and error rate.
- Resumable & partial downloads with HTTP range requests and chunk assembly.
- Integrity checks — size/timestamp comparison, ETag handling, content hashing.
- Resilience — per-domain circuit breakers, exponential backoff, rate limiting.
- Security — path-traversal and symlink-bomb defenses, private-IP/SSRF guards, URL-scope enforcement.
- Operability — metrics collection, multi-level progress, optional HTTP health-check server.
- Caching — filesystem and disk-backed indexes to skip unchanged content.
Installation
# From source (editable, recommended during the refactor)
pip install -e ".[all,dev]"
Python 3.9+ is required. Core dependencies: httpx, pydantic (v2), PyYAML.
Optional extras: fast (stringzilla, lxml), progress (tqdm), monitor (psutil).
Usage
Run via the console entry point or the module:
mirror-url https://example.com/files/ --output ./mirror
# or
python -m mirror_url https://example.com/files/ --output ./mirror
Run mirror-url --help for the full option list.
Configuration can also be supplied via a YAML file (see MirrorConfig /
load_config_from_args).
📖 Full documentation: see the User Guide (HTML version) for detailed installation, CLI reference, config-file format, download modes, security notes, the Python API, and troubleshooting.
🛠 Contributing to the code? The Developer Guide
(HTML version) is an architecture deep-dive:
dependency layers, the MirrorURL mixin design, runtime data flow, and
step-by-step extension recipes.
Development
pip install -e ".[dev]"
pre-commit install
ruff check . # lint
black --check . # format check
mypy # type-check the new package
pytest -m "not integration" # fast test lane
pytest # full suite (includes integration)
Continuous integration runs lint + tests across Python 3.9–3.12 (see
.github/workflows/ci.yml).
Project layout
src/mirror_url/ # the package (dependency-layered)
tests/ # pytest suite
REFACTORING_PLAN.md # module breakdown + migration roadmap
CHANGELOG.md # notable changes (Keep a Changelog format)
CONTRIBUTING.md # dev setup, checks, conventions
pyproject.toml # packaging, deps, tool config
Contributing
Contributions are welcome — see CONTRIBUTING.md for the dev setup, the checks CI runs, and the project conventions (dependency layering, lint policy, behavior-preserving refactors). Notable changes are tracked in CHANGELOG.md.
Authors
Borut Podlipnik, Max-Planck-Institute for Solar System Research, podlipnik@mps.mpg.de
License
MIT © BP
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file mirror_url-3.1.44.tar.gz.
File metadata
- Download URL: mirror_url-3.1.44.tar.gz
- Upload date:
- Size: 226.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
33066c757235889771a4cc7c15eabe6721c4a34fff9d9f0b668ac30de2a87bba
|
|
| MD5 |
d1af3c1f358beefb6543bb35faa2f896
|
|
| BLAKE2b-256 |
8d8fa0d0307a085815fe431133d464bb9f079a1019b60a8837ffe6fe3acf50d2
|
Provenance
The following attestation bundles were made for mirror_url-3.1.44.tar.gz:
Publisher:
release.yml on bpodlipnik/mirror-url
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
mirror_url-3.1.44.tar.gz -
Subject digest:
33066c757235889771a4cc7c15eabe6721c4a34fff9d9f0b668ac30de2a87bba - Sigstore transparency entry: 2372726878
- Sigstore integration time:
-
Permalink:
bpodlipnik/mirror-url@352111544000ad37319f6e4755e4a9b01034e785 -
Branch / Tag:
refs/tags/v3.1.44 - Owner: https://github.com/bpodlipnik
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@352111544000ad37319f6e4755e4a9b01034e785 -
Trigger Event:
push
-
Statement type:
File details
Details for the file mirror_url-3.1.44-py3-none-any.whl.
File metadata
- Download URL: mirror_url-3.1.44-py3-none-any.whl
- Upload date:
- Size: 193.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4b91f638b7c174d5fe6bc5e747408b648172650c697fe5b37f8fb359cfee6d89
|
|
| MD5 |
dbad28404f5ce978bb599f65cccb6395
|
|
| BLAKE2b-256 |
a0c272467eaa9cac22faf6ef287c57e254710c58f9fc77884e1551ef4e3e50c9
|
Provenance
The following attestation bundles were made for mirror_url-3.1.44-py3-none-any.whl:
Publisher:
release.yml on bpodlipnik/mirror-url
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
mirror_url-3.1.44-py3-none-any.whl -
Subject digest:
4b91f638b7c174d5fe6bc5e747408b648172650c697fe5b37f8fb359cfee6d89 - Sigstore transparency entry: 2372726898
- Sigstore integration time:
-
Permalink:
bpodlipnik/mirror-url@352111544000ad37319f6e4755e4a9b01034e785 -
Branch / Tag:
refs/tags/v3.1.44 - Owner: https://github.com/bpodlipnik
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@352111544000ad37319f6e4755e4a9b01034e785 -
Trigger Event:
push
-
Statement type:
