sitemapcrawler 0.0.1

A simple sitemap crawler that acts as the backbone for other operations

Sitemap Crawler

sitemapcrawler is a simple, blocking Python Crawler that is the backbone of a few other projects.

You're welcome to use it, but it's only as modular as we've needed it to be, which is to say, probably not fit for projects that aren't built with this in mind.

It works pretty simply.

Installation

pip install sitemapcrawler

Usage

from sitemapcrawler import Crawler
crawler = Crawler(domain="https://yourdomain.com", sitemap="https://yourdomain.com/sitemap.xml", fetch=True)
crawler.run()

If you just want to fetch a given page, create an instance of the crawler and call it like this:

crawler.fetch_page(url="https://yourdomain.com/blog/title")

The init will create a nanoid crawl_id so that when results are persisted, they'll be associated to a given crawl, to make it easy for reports to be built against crawls and such.

Building / Distributing

python3 -m build
python3 -m twine upload dist/* --skip-existing