Skip to main content

Get the checksum of eggs in case of building distributed scrapy clusters

Project description

scrapyd-egg-checksum

Extension of scrapyd to get egg's md5 checksum for distributed scrapyd

Installation

pip install scrapyd-egg-checksum

Configuration

Locate the scrapyd.conf file and adding line like

listversions_advanced.json = scrapyd_egg_checksum.webservice.ListVersions

under service section, the json resource name depends on you,

Usage

curl http://127.0.0.1:6800/listversions_advanced.json?project=ABC

you will get like

{"node_name": "Christians-Another-MacBook-Pro.local", "status": "ok", "versions": [{"version": "1_0_0", "checksum": "793afec3676f9749d1616f48dd57fe07"}, {"version": "1_0_1", "checksum": "ccb6b8841e2b5443db4bff56924527fa"}]}

Use case

This is for distributed scrapyd clusters, while in single server mode, this is useless.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for scrapyd-egg-checksum, version 0.1.2
Filename, size File type Python version Upload date Hashes
Filename, size scrapyd_egg_checksum-0.1.2-py3-none-any.whl (3.7 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size scrapyd_egg_checksum-0.1.2.tar.gz (2.1 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page