scrapy-state 0.0.1

Scrapy state between job runs

scrapy-state

Scrapy state between job runs. Recreating spider's state from settings See https://medium.com/@alex_ber/scrapy-state-between-job-runs-b880c7b34a9d

See CHANGELOG.md for detail description.

Getting Help

QuickStart

python3 -m pip install -U scrapy-state

Installing from Github

python3 -m pip install -U https://github.com/alex-ber/scrapy-state/archive/master.zip

Optionally installing tests requirements.

python3 -m pip install -U https://github.com/alex-ber/scrapy-state/archive/master.zip#egg=alex-ber-utils[tests]

Or explicitly:

wget https://github.com/alex-ber/scrapy-state/archive/master.zip -O master.zip; unzip master.zip; rm master.zip

And then installing from source (see below).

Installing from source

python3 -m pip install . # only installs "required"

python3 -m pip install .[tests] # installs dependencies for tests

Alternatively you install install from requirements file:

python3 -m pip install -r req.txt # only installs "required" (relaxed)

python3 -m pip install -r requirements.txt # only installs "required"

python3 -m pip install -r requirements-tests.txt # installs dependencies for tests

From the directory with setup.py

python3 setup.py test #run all tests

or

pytest

Installing new version

See https://docs.python.org/3.1/distutils/uploading.html

python3 setup.py sdist upload

Requirements

scrapy-state requires the following modules.

• Python 3.6+

Changelog

Scrapy state between job runs. Recreating spider's state from settings. See https://medium.com/@alex_ber/scrapy-state-between-job-runs-b880c7b34a9d

All notable changes to this project will be documented in this file.

#https://pypi.org/manage/project/scrapy-state/releases/

[Unrelased]

[0.0.1] - 02/07/2020

Added

• SpiderSettingsState class in alexber.spiderstate.state module. Inspired by scrapy.extensions.spiderstate
• See [https://medium.com/@alex_ber/scrapy-item-with-general-fields-7552bd6e4622] for description.