AWS DynamoDB pipeline for Scrapy
Project description
scrapy-dynamodb allows you to export your items to a DynamoDB database.
Installation
You can install scrapy-dynamodb either via the Python Package Index (PyPI) or from source.
To install using pip,:
$ pip install scrapy-dynamodb
To install using easy_install,:
$ easy_install scrapy-dynamodb
Downloading and installing from source
Download the latest version of scrapy-dynamodb from http://pypi.python.org/pypi/scrapy-dynamodb/
You can install it by doing the following,:
$ tar xvfz scrapy-dynamodb-0.0.0.tar.gz $ cd scrapy-dynamodb-0.0.0 # python setup.py install # as root
Using the development version
You can clone the git repository by doing the following:
$ git clone git://github.com/acordiner/scrapy-dynamodb.git
Setup
In pipelines.py add import statement:
from scrapy_dynamodb import DynamoDbPipeline
Using scrapy-dynamodb
To use scrapy-dynamodb in your Scrapy project, add the following line to your settings.py:
ITEM_PIPELINES = { 'scraper.pipelines.DynamoDbPipeline': 1, # replace scraper with your scrapy project name } AWS_ACCESS_KEY_ID = '<aws access key id>' AWS_SECRET_ACCESS_KEY = '<aws secret access key>' DYNAMODB_PIPELINE_REGION_NAME = 'us-east-1' DYNAMODB_PIPELINE_TABLE_NAME = 'my_table' DYNAMODB_ENDPOINT_URL = '<dynamodb endpoint url>' # optional (e.g.: http://localhost:8000)
That’s it!
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scrapy-dynamodb-0.2.3.tar.gz
(2.6 kB
view hashes)