scrapy-feedexporter-azure-blob 0.2

Scrapy extension Feed Exporter Storage Backend to export items to a Azure blob container

scrapy-feedexporter-azure is a Scrapy Feed Exporter Storage Backend that allows you to export Scrapy items to an Azure Blob Container.

Using scrapy-feedexporter-azure-blob

Add a the following to your Scrapy settings:

FEED_STORAGES = {"azure": "scrapy_feedexporter_azure_blob.AzureBlobFeedStorage"}

FEED_URI = "azure://accountname/container"

AZURE_ACCOUNT_NAME = "mycrawldata"  # this is the subdomain to https://*.blob.core.windows.net/

AZURE_ACCOUNT_KEY = "your_account_key"

AZURE_CONTAINER = 'sites'   # the name of the container (you should have already created it)

AZURE_FILENAME = 'bob.json'   # the name of the file as it will be in blob storage

Note, you can define all of these settings when you run your crawler with the -s command line switch:

scrapy crawl <crawler_name> -s AZURE_ACCOUNT_KEY='<account_key>' -s AZURE_ACCOUNT_NAME='<account_name>'

TODO

Have the feed exporter properly parse the FEED_URI string: azure://account_name:api_key@container/filename.json

DEVELOPER NOTES

When deploying to pypi, just use twine