Skip to main content

Scrapy pipeline which allow you to store multiple scrapy items in Elastic Search.

Project description


Scrapy pipeline which allows you to store scrapy items in Elastic Search.


pip install ScrapyElasticSearch

If you need support for ntlm:
pip install "ScrapyElasticSearch[extras]"

Usage (Configure

    'scrapyelasticsearch.scrapyelasticsearch.ElasticSearchPipeline': 500

ELASTICSEARCH_UNIQ_KEY = 'url'  # Custom unique key

# can also accept a list of fields if need a composite key

ELASTICSEARCH_SERVERS - list of hosts or string (single host). Host format: protocol://username:password@host:port.


Available parameters (in

 ELASTICSEARCH_INDEX - elastic search index
 ELASTICSEARCH_INDEX_DATE_FORMAT - the format for date suffix for the index, see python datetime.strftime for format. Default is no date suffix.
 ELASTICSEARCH_TYPE - elastic search type
 ELASTICSEARCH_UNIQ_KEY - optional field, unique key in string (must be a field or a list declared in model, see
 ELASTICSEARCH_BUFFER_LENGTH - optional field, number of items to be processed during each bulk insertion to Elasticsearch. Default size is 500.
 ELASTICSEARCH_AUTH  - optional field, set to 'NTLM' to use NTLM authentification
 ELASTICSEARCH_USERNAME - optional field, set to 'DOMAIN\username', only used with NLTM authentification
 ELASTICSEARCH_PASSWORD - optional field, set to your 'password', only used with NLTM authentification

 ELASTICSEARCH_CA - optional settings to if es servers require custom CA files.
      'CA_CERT': '/path/to/cacert.pem',
      'CLIENT_CERT': '/path/to/client_cert.pem',
      'CLIENT_KEY': '/path/to/client_key.pem'

Here is an example app (dirbot in case you are still confused.


See requirements.txt


  • 0.9: Accept custom CA cert to connect to es clusters
  • 0.8: Added support for NTLM authentification
  • 0.7.1: Added date format to the index name and a small bug fix
    • ELASTICSEARCH_BUFFER_LENGTH default was 9999, this has been changed to reflect documentation.
  • 0.7: A number of backwards incompatibility changes are introduced:
    • ELASTICSEARCH_SERVERS accepts string or list
    • ELASTICSEARCH_PORT is removed, you can specify it in the url
    • ELASTICSEARCH_USERNAME and ELASTICSEARCH_PASSWORD are removed. You can use this format ELASTICSEARCH_SERVERS=[’http://username:password@host:port’]
    • Changed scrapy.log to logging as scrapy now uses the logging module
  • 0.6.1: Able to pull configs from spiders (in addition to reading from config file)
  • 0.6: Bug fix
  • 0.5: Abilit to persist object; Option to specify logging level
  • 0.4: Remove debug
  • 0.3: Auth support
  • 0.2: Scrapy 0.18 support
  • 0.1: Initial release


If you find any bugs or have any questions, please report them to “issues” (



Copyright 2014 Michael Malocha

Expanded on the work by Julien Duponchelle

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ScrapyElasticSearch-0.9.2.tar.gz (10.7 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page