Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

Deploy to SpiderKeeper

Project description

CLI tool to deploy Scrapy projects to SpiderKeeper



pip install spiderkeeper-deploy


Config values can be supplied at runtime as arguments. If arguments are not supplied spiderkeeper-deploy will try to find scrapy.cfg in the current project. Config values in scrapy.cfg will be loaded from [skdeploy] section. Any value not supplied as argument or in scrapy.cfg can be entered in the interactive prompt.

The one caveat to using scrapy.cfg is that passwords will not be read. They must be supplied at run time or set in SK_PASSWORD environment variable. Don’t save passwords in config files!

spiderkeeper-deploy --help

Usage: spiderkeeper-deploy [OPTIONS]

    Deploy Scrapy projects to SpiderKeeper.

    Hint: you can define CLI args in scrapy.cfg file in your project.

    CLI args override scrapy.cfg

    -u, --url       Server name or ip. Default: http://localhost:8080
    -p, --project   Project name.
    -j, --jobs      Jobs in json format
    --user          Default: admin
    --password      Will use ENV SK_PASSWORD if present. Default: admin
    --help          Show this message and exit.

Example scrapy.cfg

Note: jobs format is exactly the same as the api provided by SpiderKeeper to add and update jobs. This array must be indented at least 1 level. Proper JSON formatting is required so double quotes and no trailing comma.

default = project.settings

url = http://localhost:6800/
project = project

url = http://localhost:5000/
project = project
user = me
jobs = [
            "spider_name": "spider_name",
            "spider_arguments": "arg1,arg2",
            "run_type": "periodic",
            "desc": "description",
            "tags": "tag1,tag2",
            "priority": 1,
            "cron_minutes": "0",
            "cron_hour": "*",
            "cron_day_of_month": "*",
            "cron_day_of_week": "*",
            "cron_month": "*"


  • If the project does not already exist it will be created
  • Any jobs not yet in SpiderKeeper will be added
  • Jobs already in SpiderKeeper will be updated (i.e. tags, desc can be updated)
  • Jobs not in config will be deleted

A job is defined as already existing in SpiderKeeper if spider_name, spider_arguments, cron_minutes, cron_hour, cron_day_of_month, cron_day_of_week, and cron_month all match what is in config.

Note: spider_name is always required. spider_arguments can be omitted and will default to None. cron settings can be omitted and will default to “*”

SpiderKeeper uses numbers to identify projects. This means it is possible to have two projects with the same name. spiderkeeper-deploy will use the first project who’s name matches the project config value. If you have an existing deployment with duplicates you should keep this in mind. If not you should never get duplicates as long as you only use spiderkeeper-deploy. It is still possible to get duplicates with a manual deployment.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for spiderkeeper-deploy, version 0.1.3
Filename, size File type Python version Upload date Hashes
Filename, size spiderkeeper-deploy-0.1.3.tar.gz (7.2 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page