Scrapy extension to sync `.scrapy` folder to an S3 bucket
Project description
Scrapy extension to sync .scrapy folder to an S3 bucket.
Installation
You can install scrapy-dotpersistence using pip:
pip install scrapy-dotpersistence
You can then enable the extension in your settings.py:
EXTENSIONS = { ... 'scrapy_dotpersistence.DotScrapyPersistence': 0 }
How to use it
Enable extension through settings.py:
DOTSCRAPY_ENABLED = True
Configure the exension through settings.py:
ADDONS_AWS_ACCESS_KEY_ID = "ABC" ADDONS_AWS_SECRET_ACCESS_KEY = "DEF" ADDONS_AWS_USERNAME = "username" ADDONS_S3_BUCKET = "test-bucket-name"
You can change a dotpersistence folder path with environ:
export DOTSCRAPY_DIR='/tmp/.scrapy'
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for scrapy-dotpersistence-0.3.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a13d0c275b5a1272ecd659060f8e4edc645b15c77cb5505acfeefbf5ceb4b205 |
|
MD5 | 6303205f93b65963dbc944be77da05da |
|
BLAKE2b-256 | fc111301ff0f945ac183ff6735a4cbca351f7b8cee31923fc3e79eeb1dda4e7a |
Close
Hashes for scrapy_dotpersistence-0.3.0-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1750f1f79eda55f68db2e0ca8ee46283e5e607f911770b0a784276ed0b690f52 |
|
MD5 | e937440375aa0e9794c6d952d6d71ad8 |
|
BLAKE2b-256 | 206205c62dc744e624a625b12536c3f192896dec5ef9cb849017956edf9ab826 |