hibp-downloader 0.4.8

Efficiently download HIBP new pwned password data by hash-prefix for a local-copy

hibp-downloader

This is a CLI tool to efficiently download a local copy of the pwned password hash data from the very awesome HIBP pwned passwords api-endpoint using all the good bits; multiprocessing, async-processes, local-caching, content-etags and http2-connection pooling to probably make things as fast as is Pythonly possible.

Features

• Direct password lookups via the query command — check passwords against the compressed data store with no database or decompression step needed. Fast enough to use behind a web service.
• Download and store acquired data in gzip compressed format to save on storage and speed up queries.
• Download the full dataset in under 45 mins (generally CPU bound).
• Easily resume interrupted download operations into a --data-path without re-clobbering api-source.
• Only download hash-prefix content blocks when the source content has changed (via content ETAG values); making it easy to periodically sync-up when needed.
• Ability to generate a single text file with in-order pwned password hash values, similar to PwnedPasswordsDownloader from the awesome HIBP team.
• Per prefix file metadata in JSON format for easy data reuse by other tooling if required.
• Standalone validation command to verify the local copy dataset, clean up corrupted or incomplete files, and remove orphaned metadata files.

Install

pipx install hibp-downloader

Usage (download)

Performance

Sample download activity log; host with 32 cores on 500Mbit/s connection.

...
2024-05-16T10:18:01-0400 | INFO | hibp-downloader | prefix=f80c7 source=[lc:13616 et:3 rc:1002358 ro:25 xx:1] processed=[17836.6MB ~414462H/s] api=[918req/s 17597.4MB] runtime=36.4min
2024-05-16T10:18:02-0400 | INFO | hibp-downloader | prefix=f81af source=[lc:13616 et:3 rc:1002558 ro:25 xx:1] processed=[17840.1MB ~414454H/s] api=[918req/s 17600.9MB] runtime=36.4min
2024-05-16T10:18:02-0400 | INFO | hibp-downloader | prefix=f826f source=[lc:13616 et:3 rc:1002758 ro:25 xx:1] processed=[17843.6MB ~414454H/s] api=[918req/s 17604.4MB] runtime=36.4min
2024-05-16T10:18:03-0400 | INFO | hibp-downloader | prefix=f833f source=[lc:13616 et:3 rc:1002958 ro:25 xx:1] processed=[17847.1MB ~414450H/s] api=[918req/s 17607.9MB] runtime=36.4min

• 918x requests per second to api.pwnedpasswords.com
• Log sources are shorthand:
  • lc: local-cache - request-responses handled locally without hitting the network.
  • et: ETag match - request-responses that confirmed our local data was up-to-date and did not require a new download.
  • rc: remote-cache - request-responses that were downloaded to local, but came from the remote-server cache.
  • ro: remote-origin - request-responses that were downloaded to local, and the download needed to be fetched from remote origin source.
  • xx: unknown/failed - request-responses that failed (and successfully retried).
• ~17GB downloaded in ~36 minutes (full dataset)
• Approx ~414k hash values received per second
• Processing in this example appears to be CPU bound, measured traffic around ~160 Mbit/s.

Usage (query)

Query passwords directly against the compressed data store — no decompression, no database import required. This is the recommended approach for any password-checking lookup.

Usage (generate)

Generate a single decompressed text file from the data store. If you are generating this to import into a database for lookups, consider using the query command directly instead — it is faster to set up, far easier to maintain, and uses a fraction of the storage.

hibp-downloader --data-path /path/to/data generate --filename pwned-hashes.txt --hash-type sha1

Usage (validate)

Validate local pwned password files and automatically clean up corrupted data or orphaned metadata files:

hibp-downloader --data-path /path/to/data validate --hash-type sha1

Project

• Docs - threatpatrols.github.io/hibp-downloader
• PyPI - pypi.org/project/hibp-downloader/
• Github - github.com/threatpatrols/hibp-downloader