Skip to main content

Simple website crawler that asynchronously crawls a website and all subpages that it can find, along with static content that they rely on.

Project description

Build status Test coverage PyPI version Python versions License

Simple website crawler that asynchronously crawls a website and all subpages that it can find, along with static content that they rely on. You can either use it as a library, inside your Python project or check out the provided CLI that can currently show you the crawled data (links, images, CSS and Javascript files) for each found site amd create a sitemap.xml file.

Created primarily to play with asyncio, aiohttp and the new async/await syntax, so:

  • it requires Python 3.5 or higher

  • new features are not planned at the moment; feel free to suggest them though, as I’m happy to implement them if someone will actually use them ; -)

Full disclosure - halfway through the project I found this article (and code) which does pretty much exactly what I wanted and is co-written by the BDFL himself. Oh well. I still finished the project and didn’t copy anything explicitly but it did influence some of my choices. After all, if it’s good enough for the creator of the language I’m using, it’s probably good enough for me.

Installation

>From PyPI:

$ pip3 install simple-site-crawler

With git clone:

$ git clone https://github.com/pawelad/simple-site-crawler
$ pip3 install -r simple-site-crawler/requirements.txt
$ cd simple-site-crawler/bin

Usage

$ simple-site-crawler --help
Usage: simple-site-crawler [OPTIONS] URL

  Simple website crawler that generates its sitemap and can either print it
  (and its static content) or export it to standard XML format.

  See https://github.com/pawelad/simple-site-crawler for more info.

Options:
  -t, --max-tasks INTEGER  Maximum allowed number of async tasks.
  -e, --export-to-xml      Export sitemap to XML file.
  -s, --suppress           Suppress printing output to stdout.
  --help                   Show this message and exit.

API

There’s no proper documentation as of now, but the code is commented and should be pretty straightforward to use.

That said - feel free to ask me either via email or GitHub issues if anything is unclear.

Tests

Package was tested with the help of py.test and tox on Python 3.5 and 3.6 (see tox.ini).

Code coverage is available at Coveralls.

To run tests yourself you need to run tox inside the repository:

$ pip install -r requirements/dev.txt
$ tox

Contributions

Package source code is available at GitHub.

Feel free to use, ask, fork, star, report bugs, fix them, suggest enhancements, add functionality and point out any mistakes. Thanks!

Authors

Developed and maintained by Paweł Adamczak.

Released under MIT License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

simple-site-crawler-0.1.0.tar.gz (12.1 kB view details)

Uploaded Source

Built Distribution

simple_site_crawler-0.1.0-py3-none-any.whl (14.5 kB view details)

Uploaded Python 3

File details

Details for the file simple-site-crawler-0.1.0.tar.gz.

File metadata

File hashes

Hashes for simple-site-crawler-0.1.0.tar.gz
Algorithm Hash digest
SHA256 3009f6de271528996f34b9421c2ef41fae2c1c584967bd8665cbee52425005f4
MD5 96c39c1f7e9892df66607cbb4b946098
BLAKE2b-256 af638a33754eb08318edceccd2d0d03f39fb70f38b942ca3e917bf3603e8f889

See more details on using hashes here.

File details

Details for the file simple_site_crawler-0.1.0-py3-none-any.whl.

File metadata

File hashes

Hashes for simple_site_crawler-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 7e4317fa3d85675560d56c05083904b361a0c92b00e3b7ca3c2ad036a5943b44
MD5 37152544d3a07b9279db4e63ab2548bb
BLAKE2b-256 e1620eafac1c281d00feeb3cca3c48263571694da2f5bf11ca622b03c59651e5

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page