Skip to main content

Scrapy state between job runs

Project description

scrapy-state

Scrapy state between job runs. Recreating spider's state from settings See https://medium.com/@alex_ber/scrapy-state-between-job-runs-b880c7b34a9d

See CHANGELOG.md for detail description.

Getting Help

QuickStart

python3 -m pip install -U scrapy-state

Installing from Github

python3 -m pip install -U https://github.com/alex-ber/scrapy-state/archive/master.zip

Optionally installing tests requirements.

python3 -m pip install -U https://github.com/alex-ber/scrapy-state/archive/master.zip#egg=alex-ber-utils[tests]

Or explicitly:

wget https://github.com/alex-ber/scrapy-state/archive/master.zip -O master.zip; unzip master.zip; rm master.zip

And then installing from source (see below).

Installing from source

python3 -m pip install . # only installs "required"
python3 -m pip install .[tests] # installs dependencies for tests

Alternatively you install install from requirements file:

python3 -m pip install -r req.txt # only installs "required" (relaxed)
python3 -m pip install -r requirements.txt # only installs "required"
python3 -m pip install -r requirements-tests.txt # installs dependencies for tests

From the directory with setup.py

python3 setup.py test #run all tests

or

pytest

Installing new version

See https://docs.python.org/3.1/distutils/uploading.html

python3 setup.py sdist upload

Requirements

scrapy-state requires the following modules.

  • Python 3.6+

Changelog

Scrapy state between job runs. Recreating spider's state from settings. See https://medium.com/@alex_ber/scrapy-state-between-job-runs-b880c7b34a9d

All notable changes to this project will be documented in this file.

#https://pypi.org/manage/project/scrapy-state/releases/

[Unrelased]

[0.0.1] - 02/07/2020

Added

  • SpiderSettingsState class in alexber.spiderstate.state module. Inspired by scrapy.extensions.spiderstate
  • See [https://medium.com/@alex_ber/scrapy-item-with-general-fields-7552bd6e4622] for description.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrapy-state-0.0.1.tar.gz (6.2 kB view hashes)

Uploaded Source

Built Distribution

scrapy_state-0.0.1-py3-none-any.whl (5.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page