ScrapyRTA
ScrapyRTA is an asynchronous HTTP API for running Scrapy spiders, built with FastAPI. It's a modern rewrite of the legacy ScrapyRT project, focusing on asynchronous operation and scalability.
Features
- Run Scrapy spiders via an Async HTTP API
- Configurable request parameters
API Endpoints
POST /crawl.json
Run a spider with specified parameters.
Example request:
curl --data '{"request": {"url": "http://quotes.toscrape.com/page/2/"}, "spider_name": "toscrape-css", "crawl_args": {"zipcode":"14000"}}' http://localhost:9080/crawl -v
Have a look at
http://127.0.0.1:9080/docsfor more details and examples.
You can also create an .env file with the following content to alter ScrapyRTA behavior:
SCRAPYRTA_DEBUG=False
SCRAPYRTA_LOG_LEVEL=INFO
SCRAPYRTA_ENABLE_OPEN_API=False
SCRAPYRTA_TIMEOUT_LIMIT=30 # seconds
Notes
- Requires
scrapy.cfgin project directory, raises error if missing.
1 maintainer
Metadata
1 maintainer
Release files for scrapyrta 1.0.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapyrta-1.0.5.tar.gz | 101.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapyrta-1.0.5-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 115.6 kB
Release files / scrapyrta-1.0.5.tar.gz
| Download URL | scrapyrta-1.0.5.tar.gz |
|---|---|
| Size | 101.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c65e4161cf4a706c4b62dc67e238497656923435d0570d3e5a9c71a4d61e17b5
|
|
BLAKE2b-256 checksum How to use checksums |
73688655c7b135f1f82ecf709e27d0a7233ca75ec8213264263b82633ef9b55a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.1
|
Release files / scrapyrta-1.0.5-py3-none-any.whl
| Download URL | scrapyrta-1.0.5-py3-none-any.whl |
|---|---|
| Size | 13.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a0543447b73b3bce3a3fcd698bd600ea7af3fd93879f5ec946273d94829c754e
|
|
BLAKE2b-256 checksum How to use checksums |
e9f5e54e88df05a65e817891dc3b3f4b858354594b70490cc14c4e2d01c1314c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.1
|
