Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (
Help us improve Python packaging - Donate today!

Simple website crawler that asynchronously crawls a website and all subpages that it can find, along with static content that they rely on.

Project Description

Simple website crawler that asynchronously crawls a website and all subpages that it can find, along with static content that they rely on. You can either use it as a library, inside your Python project or check out the provided CLI that can currently show you the crawled data (links, images, CSS and Javascript files) for each found site and create a sitemap.xml file.

Created primarily to play with asyncio, aiohttp and the new async/await syntax, so:

  • it requires Python 3.5 or higher
  • new features are not planned at the moment; feel free to suggest them though, as I’m happy to implement them if someone will actually use them ; -)

Full disclosure - halfway through the project I found this article (and code) which does pretty much exactly what I wanted and is co-written by the BDFL himself. Oh well. I still finished the project and didn’t copy anything explicitly but it did influence some of my choices. After all, if it’s good enough for the creator of the language I’m using, it’s probably good enough for me.


>From PyPI:

$ pip3 install simple-site-crawler

With git clone:

$ git clone
$ pip3 install -r simple-site-crawler/requirements.txt
$ cd simple-site-crawler/bin


$ simple-site-crawler --help
Usage: simple-site-crawler [OPTIONS] URL

  Simple website crawler that generates its sitemap and can either print it
  (and its static content) or export it to standard XML format.

  See for more info.

  -t, --max-tasks INTEGER  Maximum allowed number of async tasks.
  -e, --export-to-xml      Export sitemap to XML file.
  -s, --suppress           Suppress printing output to stdout.
  --help                   Show this message and exit.


There’s no proper documentation as of now, but the code is commented and should be pretty straightforward to use.

That said - feel free to ask me either via email or GitHub issues if anything is unclear.


Package was tested with the help of py.test and tox on Python 3.5 and 3.6 (see tox.ini).

Code coverage is available at Coveralls.

To run tests yourself you need to run tox inside the repository:

$ pip install -r requirements/dev.txt
$ tox


Package source code is available at GitHub.

Feel free to use, ask, fork, star, report bugs, fix them, suggest enhancements, add functionality and point out any mistakes. Thanks!


Developed and maintained by Paweł Adamczak.

Released under MIT License.

Release History

This version
History Node


History Node


Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, Size & Hash SHA256 Hash Help File Type Python Version Upload Date
(14.9 kB) Copy SHA256 Hash SHA256
Wheel py2.py3 Jan 30, 2017
(12.2 kB) Copy SHA256 Hash SHA256
Source None Jan 30, 2017

Supported By

Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Google Google Cloud Servers DreamHost DreamHost Log Hosting