scrapy-dotpersistence

Scrapy extension to sync .scrapy folder to an S3 bucket.

Installation

You can install scrapy-dotpersistence using pip:

pip install scrapy-dotpersistence

You can then enable the extension in your settings.py:

EXTENSIONS = {
    ...
    'scrapy_dotpersistence.DotScrapyPersistence': 0
}

Enable extension through settings.py:

DOTSCRAPY_ENABLED = True

Configure the exension through settings.py:

ADDONS_AWS_ACCESS_KEY_ID = "ABC"
ADDONS_AWS_SECRET_ACCESS_KEY = "DEF"
ADDONS_AWS_USERNAME = "username"
ADDONS_S3_BUCKET = "test-bucket-name"

You can change a dotpersistence folder path with environ:

export DOTSCRAPY_DIR='/tmp/.scrapy'

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution for scrapy-dotpersistence 0.3.0
File	Size	Uploaded
scrapy-dotpersistence-0.3.0.tar.gz	2.5 kB	Aug 4, 2016	Details

Table of built distributions (wheels) for scrapy-dotpersistence 0.3.0
File	Interpreter	ABI	Platform	Reset
scrapy_dotpersistence-0.3.0-py2-none-any.whl 4.1 kB Aug 4, 2016	Python 2	none	any	Details

Total release size: 6.6 kB

Download URL	scrapy-dotpersistence-0.3.0.tar.gz
Size	2.5 kB
Tags	Source
SHA-256 checksum How to use checksums	`a13d0c275b5a1272ecd659060f8e4edc645b15c77cb5505acfeefbf5ceb4b205`
BLAKE2b-256 checksum How to use checksums	`fc111301ff0f945ac183ff6735a4cbca351f7b8cee31923fc3e79eeb1dda4e7a`
Upload date	Aug 4, 2016
Uploaded using Trusted Publishing? What is trusted publishing?	No

Download URL	scrapy_dotpersistence-0.3.0-py2-none-any.whl
Size	4.1 kB
Tags	Python 2
SHA-256 checksum How to use checksums	`1750f1f79eda55f68db2e0ca8ee46283e5e607f911770b0a784276ed0b690f52`
BLAKE2b-256 checksum How to use checksums	`206205c62dc744e624a625b12536c3f192896dec5ef9cb849017956edf9ab826`
Upload date	Aug 4, 2016
Uploaded using Trusted Publishing? What is trusted publishing?	No