Simple website crawler that asynchronously crawls a website and all subpages that it can find, along with static content that they rely on.
Created primarily to play with asyncio, aiohttp and the new async/await syntax, so:
it requires Python 3.5 or higher
new features are not planned at the moment; feel free to suggest them though, as I’m happy to implement them if someone will actually use them ; -)
Full disclosure - halfway through the project I found this article (and code) which does pretty much exactly what I wanted and is co-written by the BDFL himself. Oh well. I still finished the project and didn’t copy anything explicitly but it did influence some of my choices. After all, if it’s good enough for the creator of the language I’m using, it’s probably good enough for me.
$ pip3 install simple-site-crawler
With git clone:
$ git clone https://github.com/pawelad/simple-site-crawler $ pip3 install -r simple-site-crawler/requirements.txt $ cd simple-site-crawler/bin
$ simple-site-crawler --help Usage: simple-site-crawler [OPTIONS] URL Simple website crawler that generates its sitemap and can either print it (and its static content) or export it to standard XML format. See https://github.com/pawelad/simple-site-crawler for more info. Options: -t, --max-tasks INTEGER Maximum allowed number of async tasks. -e, --export-to-xml Export sitemap to XML file. -s, --suppress Suppress printing output to stdout. --help Show this message and exit.
There’s no proper documentation as of now, but the code is commented and should be pretty straightforward to use.
Package was tested with the help of py.test and tox on Python 3.5 and 3.6 (see tox.ini).
Code coverage is available at Coveralls.
To run tests yourself you need to run tox inside the repository:
$ pip install -r requirements/dev.txt $ tox
Package source code is available at GitHub.
Feel free to use, ask, fork, star, report bugs, fix them, suggest enhancements, add functionality and point out any mistakes. Thanks!
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for simple-site-crawler-0.1.1.tar.gz
Hashes for simple_site_crawler-0.1.1-py2.py3-none-any.whl