scrapelib is a library for making requests to less-than-reliable websites.
This repository has moved to Codeberg, GitHub will remain as a read-only mirror.
Source: https://codeberg.org/jpt/scrapelib
Documentation: https://jamesturk.github.io/scrapelib/
Issues: https://codeberg.org/jpt/scrapelib/issues
Features
scrapelib originated as part of the Open States project to scrape the websites of all 50 state legislatures and as a result was therefore designed with features desirable when dealing with sites that have intermittent errors or require rate-limiting.
Advantages of using scrapelib over using requests as-is:
- HTTP(S) and FTP requests via an identical API
- support for simple caching with pluggable cache backends
- highly-configurable request throtting
- configurable retries for non-permanent site failures
- All of the power of the suberb requests library.
Installation
scrapelib is on PyPI, and can be installed via any standard package management tool.
Example Usage
import scrapelib
s = scrapelib.Scraper(requests_per_minute=10)
# Grab Google front page
s.get('http://google.com')
# Will be throttled to 10 HTTP requests per minute
while True:
s.get('http://example.com')
Release files for scrapelib 2.4.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapelib-2.4.1.tar.gz | 19.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapelib-2.4.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 37.2 kB
Release files / scrapelib-2.4.1.tar.gz
| Download URL | scrapelib-2.4.1.tar.gz |
|---|---|
| Size | 19.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
48340199e92e860a423aeae09ef03cb9c99b78213eddedc9a31c1545b9bf2b6a
|
|
BLAKE2b-256 checksum How to use checksums |
9e5b4207c24a2daf172cddaa9d994e2ac5397313390be72f1ffac293ac9ad624
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.13
|
Release files / scrapelib-2.4.1-py3-none-any.whl
| Download URL | scrapelib-2.4.1-py3-none-any.whl |
|---|---|
| Size | 17.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1332f8ab05ab3e23b3db3f62e33ffad68ac3d1f1ac9be9fdc9410d0a30a5bf59
|
|
BLAKE2b-256 checksum How to use checksums |
3d481f9e4e877f79d84b19c31b1753b0140d0ece723aa4b0506ec27e15845f8b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.13
|
