Skip to main content
Help us improve PyPI by participating in user testing. All experience levels needed!

Scrapy extension to sync `.scrapy` folder to an S3 bucket

Project description

Scrapy extension to sync .scrapy folder to an S3 bucket.

Installation

You can install scrapy-dotpersistence using pip:

pip install scrapy-dotpersistence

You can then enable the extension in your settings.py:

EXTENSIONS = {
    ...
    'scrapy_dotpersistence.DotScrapyPersistence': 0
}

How to use it

Enable extension through settings.py:

DOTSCRAPY_ENABLED = True

Configure the exension through settings.py:

ADDONS_AWS_ACCESS_KEY_ID = "ABC"
ADDONS_AWS_SECRET_ACCESS_KEY = "DEF"
ADDONS_AWS_USERNAME = "username"
ADDONS_S3_BUCKET = "test-bucket-name"

You can change a dotpersistence folder path with environ:

export DOTSCRAPY_DIR='/tmp/.scrapy'

Project details


Release history Release notifications

This version
History Node

0.3.0

History Node

0.2.3

History Node

0.2.2

History Node

0.2.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
scrapy_dotpersistence-0.3.0-py2-none-any.whl (4.1 kB) Copy SHA256 hash SHA256 Wheel py2 Aug 4, 2016
scrapy-dotpersistence-0.3.0.tar.gz (2.5 kB) Copy SHA256 hash SHA256 Source None Aug 4, 2016

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page