MirrorURL

CI Python License: MIT

Enterprise-grade remote directory mirroring tool. MirrorURL recursively discovers files behind an HTTP(S) directory listing and mirrors them locally with adaptive concurrency, resumable/partial downloads, integrity verification, and an SSRF-hardened transport layer.

Status: Refactor complete. The full implementation lives in the src/mirror_url/ package, migrated verbatim from the original single-file script per REFACTORING_PLAN.md. The legacy single-file script was retained as a frozen reference until the test suite passed against the package with real runtime dependencies installed, then removed (v3.1.20).

Features

  • Recursive discovery of remote directory trees (BFS, depth/exclude limits, cycle-safe).
  • True parallel downloads — multiple files and multiple chunks per file concurrently.
  • Adaptive async concurrency that tunes itself to server RTT, throughput, and error rate.
  • Resumable & partial downloads with HTTP range requests and chunk assembly.
  • Integrity checks — size/timestamp comparison, ETag handling, content hashing.
  • Resilience — per-domain circuit breakers, exponential backoff, rate limiting.
  • Security — path-traversal and symlink-bomb defenses, private-IP/SSRF guards, URL-scope enforcement.
  • Operability — metrics collection, multi-level progress, optional HTTP health-check server.
  • Caching — filesystem and disk-backed indexes to skip unchanged content.

Installation

# From source (editable, recommended during the refactor)
pip install -e ".[all,dev]"

Python 3.9+ is required. Core dependencies: httpx, pydantic (v2), PyYAML. Optional extras: fast (stringzilla, lxml), progress (tqdm), monitor (psutil).

Usage

Run via the console entry point or the module:

mirror-url https://example.com/files/ --output ./mirror
# or
python -m mirror_url https://example.com/files/ --output ./mirror

Run mirror-url --help for the full option list.

Configuration can also be supplied via a YAML file (see MirrorConfig / load_config_from_args).

📖 Full documentation: see the User Guide (HTML version) for detailed installation, CLI reference, config-file format, download modes, security notes, the Python API, and troubleshooting.

🛠 Contributing to the code? The Developer Guide (HTML version) is an architecture deep-dive: dependency layers, the MirrorURL mixin design, runtime data flow, and step-by-step extension recipes.

Development

pip install -e ".[dev]"
pre-commit install

ruff check .                  # lint
black --check .               # format check
mypy                          # type-check the new package
pytest -m "not integration"   # fast test lane
pytest                        # full suite (includes integration)

Continuous integration runs lint + tests across Python 3.9–3.12 (see .github/workflows/ci.yml).

Project layout

src/mirror_url/          # the package (dependency-layered)
tests/                   # pytest suite
REFACTORING_PLAN.md      # module breakdown + migration roadmap
CHANGELOG.md             # notable changes (Keep a Changelog format)
CONTRIBUTING.md          # dev setup, checks, conventions
pyproject.toml           # packaging, deps, tool config

Contributing

Contributions are welcome — see CONTRIBUTING.md for the dev setup, the checks CI runs, and the project conventions (dependency layering, lint policy, behavior-preserving refactors). Notable changes are tracked in CHANGELOG.md.

Authors

Borut Podlipnik, Max-Planck-Institute for Solar System Research, podlipnik@mps.mpg.de

License

MIT © BP

Project links

Data verified by PyPI on Aug 4, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (3.1.44)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for podlipnik from gravatar.com podlipnik

Credits

Author: BP

GitHub Statistics

Data verified by PyPI on Aug 4, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (MIT License)

Requires

Python >=3.9

Provides Extra

fast progress monitor all dev

Tags

mirror download http scraper directory async httpx

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mirror_url-3.1.42.tar.gz (223.3 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

mirror_url-3.1.42-py3-none-any.whl (192.9 kB view details)

Uploaded Python 3

File details

Details for the file mirror_url-3.1.42.tar.gz.

File metadata

  • Download URL: mirror_url-3.1.42.tar.gz
  • Upload date:
  • Size: 223.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for mirror_url-3.1.42.tar.gz
Algorithm Hash digest
SHA256 fb6c0b829c1e92ee7b6f79f5cdc7238967a9908bf279716bd4b229507632719b
MD5 cd2765d53a5fb3fda5ddecb7086fd8ce
BLAKE2b-256 4d4e5ce005c03a5eafb28f3e7e6a7d92fb3b6e878e9f7b4943e9f4ea531a1a44

See more details on using hashes here.

Provenance

The following attestation bundles were made for mirror_url-3.1.42.tar.gz:

Publisher: release.yml on bpodlipnik/mirror-url

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file mirror_url-3.1.42-py3-none-any.whl.

File metadata

  • Download URL: mirror_url-3.1.42-py3-none-any.whl
  • Upload date:
  • Size: 192.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for mirror_url-3.1.42-py3-none-any.whl
Algorithm Hash digest
SHA256 58a1b42a81a650aeb036065a056d0e7159f46d246a9d74f13d406ab0e038a3c8
MD5 107cd337f0ba54a77a73ee390452b299
BLAKE2b-256 7bd72001531df8e46759092c46f6b17ae0a0fa04c20e2849dfd07139d718821e

See more details on using hashes here.

Provenance

The following attestation bundles were made for mirror_url-3.1.42-py3-none-any.whl:

Publisher: release.yml on bpodlipnik/mirror-url

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

3.1.44

2 files

3.1.43

2 files

This release

3.1.42 This release

2 files

3.1.41

2 files

3.1.40

2 files

3.1.39

2 files

3.1.38

2 files

3.1.37

2 files

3.1.36

2 files

3.1.35

2 files

3.1.34

2 files

3.1.33

2 files

3.1.32

2 files

3.1.31

2 files

3.1.30

2 files

3.1.29

2 files

3.1.28

2 files

3.1.27

2 files

3.1.26

2 files

3.1.25

2 files

3.1.24

2 files

3.1.23

2 files

3.1.22

2 files

3.1.21

2 files

3.1.20

2 files

3.1.19

2 files

3.1.18

2 files

3.1.17

2 files

3.1.16

2 files

3.1.15

2 files

3.1.14

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page