pypi-bulk-download

Download PyPI packages in bulk, filtered by real-world popularity, in parallel, with safe archive unpacking and a JSON manifest. Built for assembling research corpora (security scanning, ecosystem analysis, mirrors) without hand-rolling pip download loops.

Downloads go through pip download, so wheels, sdists, platform tags, and custom indexes behave exactly as pip does. Popularity comes from the public top-pypi-packages 30-day dataset.

Install

pip install pypi-bulk-download

CLI

# Top 50 packages with >= 100k downloads/month, source dists, no deps
pypi-bulk-download --min-downloads 100000 --limit 50 --format sdist --no-deps -o corpus/

# Specific packages, 4 workers, write a manifest
pypi-bulk-download --packages requests flask httpx --workers 4 --save-manifest corpus/manifest.json

# From a requirements file, then safely unpack every archive
pypi-bulk-download -r requirements.txt -o corpus/ --unpack

# See the plan without downloading
pypi-bulk-download --min-downloads 50000 --limit 20 --dry-run

Library

from pypi_bulk_download import fetch_top_packages, download_many, save_manifest

targets = fetch_top_packages(min_downloads=100_000, limit=25)
results = download_many(targets, "corpus/", workers=4, prefer_format="sdist", no_deps=True)
save_manifest(results, "corpus/manifest.json")

print(sum(r.success for r in results), "of", len(results), "downloaded")

Safe unpacking

unpack_archive / unpack_all validate every tar/zip member against path traversal (tar-slip / zip-slip) before extracting — untrusted archives can't write outside their destination folder.

from pypi_bulk_download import unpack_all
unpacked, failed = unpack_all("corpus/", remove_archives=False)

Notes

• Depends only on requests (plus pip, which you already have).
• --format sdist falls back to the best wheel when no source distribution exists, rather than failing hard.
• This tool downloads third-party code. Do not commit the download directory to your own repository; the included .gitignore excludes pkgs/, downloads/, and archive files by default.

License

MIT (this tool). Downloaded packages retain their own licenses.