Efficiently download HIBP new pwned password data by hash-prefix for a local-copy
Project description
hibp-downloader
This is a Python implementation of the original PwnedPasswordsDownloader that provides some additional useful functionality
- Automatically only download prefix-chunks that have changed since the last download
- Ability to start, stop and re-start without loss of data already collected
- Ability to name the
--first-hash
and--last-hash
positions - Metadata file per prefix file in JSON format for easy data reuse
Install
pip install --upgrade hibp-downloader
Usage
Runtime Logs
Sample download activity logs
2023-07-30T21:42:06+1000 | INFO | hibp-downloader | prefix=65747 source=[lc:207328 et:0 rc:56672 ro:0 xx:0] runtime_rate=[10.3MBit/s 79req/s ~65602H/s] runtime=0.2hr download=922.5MB
2023-07-30T21:42:07+1000 | INFO | hibp-downloader | prefix=29da7 source=[lc:207328 et:0 rc:56792 ro:0 xx:0] runtime_rate=[10.4MBit/s 79req/s ~65646H/s] runtime=0.2hr download=924.5MB
2023-07-30T21:42:09+1000 | INFO | hibp-downloader | prefix=43c7f source=[lc:207328 et:0 rc:56912 ro:0 xx:0] runtime_rate=[10.3MBit/s 79req/s ~65617H/s] runtime=0.2hr download=926.5MB
- 79 requests per second to api.pwnedpasswords.com
- 207,328 prefix files from (
lc
) local-cache - 56,912 prefix files from (
rc
) remote-cache - 0 files from (
ro
) remote-origin, 0 files failed (xx
) download - estimated 65,617 hash values downloaded per second
- 926MB downloaded in ~12 minutes (0.20 hour)
Issues
Source
Copyright
- Copyright © 2023 Threat Patrols Pty Ltd <contact@threatpatrols.com>
- Copyright © 2023 Nicholas de Jong <contact@nicholasdejong.com>
All rights reserved.
License
- BSD-3-Clause - see LICENSE file for details.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
hibp_downloader-0.1.2.tar.gz
(15.0 kB
view hashes)
Built Distribution
Close
Hashes for hibp_downloader-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 049172882cdba3ad2289f40da760ba15f12503354b9384961b20681ed9335b08 |
|
MD5 | ea26b948636dae26e82984074f05956e |
|
BLAKE2b-256 | 1c6032ece0977af5cb5d7f7c0697dc394ca33e952682a92bafaa2531902cd8fa |