ScrapyRT (Scrapy realtime)
Add HTTP API for your Scrapy project in minutes.
You send a request to ScrapyRT with spider name and URL, and in response, you get items collected by a spider visiting this URL.
All Scrapy project components (e.g. middleware, pipelines, extensions) are supported
You run Scrapyrt in Scrapy project directory. It starts HTTP server allowing you to schedule spiders and get spider output in JSON.
Quickstart
1. install
> pip install scrapyrt2. switch to Scrapy project (e.g. quotesbot project)
> cd my/project_path/is/quotesbot3. launch ScrapyRT
> scrapyrt4. run your spiders
> curl "localhost:9080/crawl.json?spider_name=toscrape-css&url=http://quotes.toscrape.com/"5. run more complex query, e.g. specify callback for Scrapy request and zipcode argument for spider
> curl --data '{"request": {"url": "http://quotes.toscrape.com/page/2/", "callback":"some_callback"}, "spider_name": "toscrape-css", "crawl_args": {"zipcode":"14000"}}' http://localhost:9080/crawl.json -vScrapyrt will look for scrapy.cfg file to determine your project settings, and will raise error if it won’t find one. Note that you need to have all your project requirements installed.
Note
Project is not a replacement for Scrapyd or Scrapy Cloud or other infrastructure to run long running crawls
Not suitable for long running spiders, good for spiders that will fetch one response from some website and return items quickly
Documentation
Support
Open source support is provided here in Github. Please create a question issue (ie. issue with “question” label).
Commercial support is also available by Zyte.
License
ScrapyRT is offered under BSD 3-Clause license.
Development
Development taking place on Github.
2 maintainers
Release files for scrapyrt 0.18.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapyrt-0.18.1.tar.gz | 69.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapyrt-0.18.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 85.5 kB
Release files / scrapyrt-0.18.1.tar.gz
| Download URL | scrapyrt-0.18.1.tar.gz |
|---|---|
| Size | 69.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
58b39f502b979dca2031b6be6c6b6f740bf83731f208501ed69ec43cfc584710
|
|
BLAKE2b-256 checksum How to use checksums |
65425efb10c172be4277cc0e686c56ce6c0d05af350a397d97f58a62f9d1aac0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / scrapyrt-0.18.1-py3-none-any.whl
| Download URL | scrapyrt-0.18.1-py3-none-any.whl |
|---|---|
| Size | 16.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2f82ba7d7754a11f00ab968bf8bb699daaf8f0f2ea4637065cd1b9c48d2529ae
|
|
BLAKE2b-256 checksum How to use checksums |
5a8dc5a2b2ac04e6d82b5669e7ebb89898fccd7f9aa4bed5a06dd122fc7d2396
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
