Scrapyd launcher module that schedules scrapy spiders by time
Project description
Scrapyd launcher module that schedules scrapy spiders by time.
Install
$ pip install scrapyduler
Config
To start using this library you just need to override the launcher option in your scrapyd.conf file:
[scrapyd]
launcher = scrapyduler.launcher.Launcher
and then add the schedulers configuration, e.g.:
[scheduler.1]
cron = * * * * *
project = quotesbot
spider = toscrape-xpath
settings = HTTPPROXY_ENABLED=True
args = key1=value1 key2=value2 start_url=http://quotes.toscrape.com/
[scheduler.2]
interval = weeks=0 days=0 hours=0 minutes=0 seconds=30
project = quotesbot
spider = toscrape-css
settings = HTTPPROXY_ENABLED=True
args = key1=value1 key2=value2 start_url=http://quotes.toscrape.com/
In the examples above, we set up two schedulers. The first scheduler uses cron syntax to run spiders. The second scheduler triggers on specified intervals, starting on start_date if specified, datetime.now() + interval otherwise. See https://github.com/agronholm/apscheduler for more information.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scrapyduler-1.0.1.tar.gz
(3.0 kB
view hashes)
Built Distribution
Close
Hashes for scrapyduler-1.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 49a5b8c71405db032edff38d14f3123ef510c4dc534cd2f1aa08532de316c83f |
|
MD5 | 34bad7cb566b5196e2488ff077092783 |
|
BLAKE2b-256 | 9b79a7f0cbb9c3818bd7ef1459bafc539248298473cd2cb6f1631ccf458c3437 |