Skip to main content

Each Downloader maintains an sqlite3-based disk cache that utilizes zlib compression. Network requests are only made if the cached version of the resource has an age larger or equal to the stale_after value provided by the programmer.

Between network requests a throttling interval needs to elapse. This throttling interval is randomly chosen, but lies within the throttle_bounds defined by the programmer.

HTML resources can be parsed using lxml and in this case an lxml ElementTree is returned instead of a file object, with the links rewritten to be absolute in order to facilitate following them. The parsing is done leniently in order to not fail when invalid HTML is encountered.

The programmer can also supply a function that decides whether the server has banned the client (possibly by examining the returned resource). In this case an exception will be raised.

Downloader’s features make it ideal for writing scrapers, as it can keep its network footprint small (due to the cache) and irregular (due to the random throttling interval).

To install, simply run:

python setup.py install

For documentation, after installing, run:

python -m pydoc downloader

Metadata

Release files for downloader-py3 1.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for downloader-py3 1.0.2
File Size Uploaded Provenance Actions
downloader-py3-1.0.2.tar.gz 7.8 kB Without provenance Details for downloader-py3-1.0.2.tar.gz

Release files / downloader-py3-1.0.2.tar.gz

Download URL downloader-py3-1.0.2.tar.gz
Size 7.8 kB
Tags Source
SHA-256 checksum
How to use checksums
2bd80f0c9d991c80b9ded268e8262b55846c2e669dd6c7569ddd9e0786ef34e0
BLAKE2b-256 checksum
How to use checksums
c313fff56c22ff633967fcbbef2cd49ae0c73d27cc6958726ac19be0440b0b4d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for downloader-py3 1.0.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.2 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page