Skip to main content

Scrapy Save Statistics: Save statistics extension for Scrapy

Project description

Save statistics to mongo for analytics.


The quick way:

pip install scrapy-save-statistics

Or install from GitHub:

pip install git+git://

Or checkout the source and run:

python install

Mongodb settings for save statistics, need a statistics database.

MONGO_PORT = 27017
MONGO_DB = "myspider"
MONGO_STATISTICS = "statistics"

    'scrapy_save_statistics.SaveStatistics': 100,


Spider must have statistics attributes and contains spider_url. We’ll save that info to mongodb.

class TestSpider(scrapy.Spider):
    name = "test"

    def __init__(self, name=None, **kwargs):
        super(TestSpider, self).__init__(name=name, **kwargs)
        self.statistics = []

    def parse(self, response):
        crawl_info = {'spider_url': spider_url,
                      'expected_crawl_num': expected_crawl_num,
                      'pages': total_page}

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release. See tutorial on generating distribution archives.

Built Distribution

scrapy_save_statistics-0.2-py2.py3-none-any.whl (4.4 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page