MirrorURL

Security-hardened remote directory mirroring tool. MirrorURL recursively discovers files behind an HTTP(S) directory listing and mirrors them locally with adaptive concurrency, resumable/partial downloads, integrity verification, and an SSRF-hardened transport layer.

Version 3.2.1 adds direct file URLs with mirror-url --mode download FILE_URL, using the same guarded HTTPX or optional aiohttp transfer pipeline as URL lists. See the 3.2.1 changelog.

Features

Recursive discovery of remote directory trees (BFS, depth/exclude limits, cycle-safe).
True parallel downloads — multiple files and multiple chunks per file concurrently.
Adaptive async concurrency that tunes itself to server RTT, throughput, and error rate.
Resumable & partial downloads with HTTP range requests and chunk assembly.
Integrity checks — size/timestamp comparison, ETag handling, verified byte ranges, and opt-in --verify-content SHA-256 checks of local files; no comparison against a remote cryptographic content digest.
Resilience — per-domain circuit breakers, exponential backoff, rate limiting.
Destination ownership — cooperating processes reject overlapping local trees and shared cache/state paths; hard process termination releases ownership automatically.
Crash recovery — the next owner reclaims recorded abandoned parallel chunks and staging under reserved state; live writers, unknown files and legitimate MOVE archives are preserved.
Security — path-traversal and symlink-bomb defenses, private-IP/SSRF guards, URL-scope enforcement.
Filename preflight — preserve distinct original names on a confirmed case-sensitive destination; reject case collisions on a case-insensitive destination before downloads. Rewritten, unsafe and Unicode-aliased paths remain blocked; see download behavior and preserving original names.
Operability — metrics collection, multi-level progress, optional HTTP health-check server.
Caching — directory listings and file metadata; discovery currently keeps the remote file list in memory.
Known-URL downloads — --mode download FILE_URL downloads one file, or --mode download --url-list urls.txt streams an exact list, without discovery or freshness probes. Both HTTPX and optional aiohttp use the same destination, scope, staging and receipt checks.
Explicit pacing — --requests-per-second and --request-delay control the request budget independently of security checks.

Installation

Python 3.10 or newer is required. Install the published package from PyPI in a virtual environment:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install mirror-url

To include all optional acceleration, progress, and monitoring packages:

python -m pip install "mirror-url[all]"

Upgrade an existing installation:

python -m pip install --upgrade mirror-url

Check the installed package and CLI:

python -m pip check
mirror-url --version
mirror-url --help

Core dependencies: httpx[http2], pydantic (v2), PyYAML, portalocker (including pywin32 on Windows). Optional extras: fast (stringzilla, lxml), progress (tqdm), monitor (psutil), aiohttp (the whole-file backend), and all (all optional runtime packages). Contributor setup is described below.

Usage

For an exact, pre-built list of URLs:

# Install the optional backend in your virtual environment.
python -m pip install "mirror-url[aiohttp]"
python -m mirror_url --mode download --backend aiohttp \
  --url https://example.org/files/ --url-list urls.txt \
  --dest-path ./downloads --log-path ./logs --concurrency 20 \
  --requests-per-second 0 --request-delay 0 --verify-content

Version 3.2.1 also accepts a single file URL directly:

mirror-url --mode download https://example.org/files/a.fits
# Optional destination and log paths:
mirror-url --mode download https://example.org/files/a.fits \
  --dest-path ./downloads --log-path ./logs

The shortcut defaults to the current directory, with logs in a destination-specific system temporary folder. It infers the file URL's parent as the allowed remote scope; an explicit --url overrides that scope. Choose either a positional file URL or --url-list, and keep --mode download for both.

urls.txt contains one absolute URL per line below the selected base URL. The two zero pacing values explicitly remove the default 20 requests/second ceiling and 50 ms spacing. TLS verification, public-IP DNS validation, redirect scope, safe filename mapping and atomic publication stay enabled. Existing files need a matching ownership receipt or explicit --overwrite. See known-URL downloads for limits and the HTTPX equivalent.

Run via the console entry point or the module:

mirror-url --url https://example.com/files/ --dest-path ./mirror --log-path ./mirror-log
# or
python -m mirror_url --url https://example.com/files/ --dest-path ./mirror --log-path ./mirror-log

Run mirror-url --help for the full option list.

Configuration can also be supplied via a YAML file (see MirrorConfig / load_config_from_args).

📖 Full documentation: see the User Guide (HTML version) for detailed installation, CLI reference, config-file format, download modes, security notes, the Python API, and troubleshooting.

🛠 Contributing to the code? The Developer Guide (HTML version) is an architecture deep-dive: dependency layers, the MirrorURL mixin design, runtime data flow, and step-by-step extension recipes.

Development

From a repository checkout, create and activate a virtual environment, then install the editable package with its runtime extras and development tools:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[all,dev]"
pre-commit install

ruff check .                  # lint
ruff format --check .         # format check (Ruff 0.16.8)
mypy                          # type-check the package
pytest -m "not integration"   # fast test lane
pytest                        # full suite (includes integration)

Continuous integration runs lint + tests across Python 3.10–3.14 (see .github/workflows/ci.yml).

Project layout

src/mirror_url/          # the package (dependency-layered)
tests/                   # pytest suite
REFACTORING_PLAN.md      # module breakdown + migration roadmap
CHANGELOG.md             # notable changes (Keep a Changelog format)
CONTRIBUTING.md          # dev setup, checks, conventions
pyproject.toml           # packaging, deps, tool config

Contributing

Contributions are welcome — see CONTRIBUTING.md for the dev setup, the checks CI runs, and the project conventions (dependency layering, lint policy, behavior-preserving refactors). Notable changes are tracked in CHANGELOG.md.

Authors

Borut Podlipnik, Max-Planck-Institute for Solar System Research, podlipnik@mps.mpg.de

License

MIT © BP

Metadata

Release files for mirror-url 3.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mirror-url 3.2.1
File	Size	Uploaded
mirror_url-3.2.1.tar.gz	549.3 kB	Oct 8, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for mirror-url 3.2.1
File	Interpreter	ABI	Platform	Reset
mirror_url-3.2.1-py3-none-any.whl 233.6 kB Oct 8, 2026	Python 3	none	any	Details

Total release size: 782.9 kB

Release files / mirror_url-3.2.1.tar.gz

Download URL	mirror_url-3.2.1.tar.gz
Size	549.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`5ab0468f14aadfae52f240cf2c6d4c551b93b47350c6e20d5ba781b1a66608e2`
BLAKE2b-256 checksum How to use checksums	`93ec537d070d2e3619bcb7c70ad08f389dd444089c07b97e1422ebf015ac2dce`
Upload date	Oct 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/bpodlipnik/mirror-url
Publishing commit: github.com/bpodlipnik/mirror-url/tree/e62d2508a1f027359811a364b77e42aa0817de26

Workflow

Publishing configuration: github.com/bpodlipnik/mirror-url/blob/e62d2508a1f027359811a364b77e42aa0817de26/.github/workflows/release.yml
Publishing logs: github.com/bpodlipnik/mirror-url/actions/runs/37805982227/attempts/1

Artifact

Subject: mirror_url-3.2.1.tar.gz
SHA-256 checksum: 5ab0468f14aadfae52f240cf2c6d4c551b93b47350c6e20d5ba781b1a66608e2 Verifying attestations

Release files / mirror_url-3.2.1-py3-none-any.whl

Download URL	mirror_url-3.2.1-py3-none-any.whl
Size	233.6 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`30e257116e5b6a3b7cca7aa214733663a689cf406578d0c1d5506852b8876f53`
BLAKE2b-256 checksum How to use checksums	`b0c79ca355e8ad63670058784537034caf5f7ebc6203049e2d8c442860409d61`
Upload date	Oct 8, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/bpodlipnik/mirror-url
Publishing commit: github.com/bpodlipnik/mirror-url/tree/e62d2508a1f027359811a364b77e42aa0817de26

Workflow

Publishing configuration: github.com/bpodlipnik/mirror-url/blob/e62d2508a1f027359811a364b77e42aa0817de26/.github/workflows/release.yml
Publishing logs: github.com/bpodlipnik/mirror-url/actions/runs/37805982227/attempts/1

Artifact

Subject: mirror_url-3.2.1-py3-none-any.whl
SHA-256 checksum: 30e257116e5b6a3b7cca7aa214733663a689cf406578d0c1d5506852b8876f53 Verifying attestations