scrapy-dynamodb 0.2.3

Installation

You can install scrapy-dynamodb either via the Python Package Index (PyPI) or from source.

To install using pip,:

$ pip install scrapy-dynamodb

To install using easy_install,:

$ easy_install scrapy-dynamodb

$ tar xvfz scrapy-dynamodb-0.0.0.tar.gz
$ cd scrapy-dynamodb-0.0.0
# python setup.py install # as root

$ git clone git://github.com/acordiner/scrapy-dynamodb.git

Setup

In pipelines.py add import statement:

from scrapy_dynamodb import DynamoDbPipeline

Using scrapy-dynamodb

To use scrapy-dynamodb in your Scrapy project, add the following line to your settings.py:

ITEM_PIPELINES = {
    'scraper.pipelines.DynamoDbPipeline': 1,
    # replace scraper with your scrapy project name
}

AWS_ACCESS_KEY_ID = '<aws access key id>'
AWS_SECRET_ACCESS_KEY = '<aws secret access key>'
DYNAMODB_PIPELINE_REGION_NAME = 'us-east-1'
DYNAMODB_PIPELINE_TABLE_NAME = 'my_table'
DYNAMODB_ENDPOINT_URL = '<dynamodb endpoint url>' # optional  (e.g.: http://localhost:8000)

That’s it!