Skip to main content

Simple sitemap builder

Project description

A simple sitemap builder

The sitemap builder traverses links from a website and constrains itself to the given domain name. The final result will be a simple sitemap deduced from the links visited. The crawler will accept & process only URLs with http or https schemes.

Installation and usage

To run the following command to install the tool:

pip install -U sitemapbuilder

To run the sitemap builder:

sitemapbuilder -u '' -o

Some websites have strong protection and the tool will not work for them:

sitemapbuilder -u '' -o


  1. Generate Graphviz .dot file showing directed links between pages. One can generate PNG/PDF and other image/document formats.
  2. Have configurable decay (maximum depth) to avoid abuse.
  3. Visit web link within the same hostname by default.
  4. Use 5 threads by default and times out after 10 seconds.
  5. Timeout after 5 seconds when fetching a URL.
  6. Handle timeout exceptions when querying a website.
  7. Send a HTTP HEAD request and verify that Content-Type is text/html and charset is either UTF-8 or US-ASCII.
  8. Have a map of visited URLs to avoid revisiting them.
  9. Follow HTTP redirects.

Upcoming features

  • Configure the number of threads and timeout via cmd args.
  • Allow web links from all subdomains.
  • Allow web links from a list of domains.
  • Allow web links matching a pattern.
  • Add an option for hierarchical sitemap instead of directed graph.
  • Use PriorityQueue instead of Queue to process links with higher decay first.
  • Fine-graned info, warn and error logging.
  • Pass seed links from a file.
  • Save to and resume from a DB/persistent data source.
  • Faster concurrency and better performance with asyncio.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for sitemapbuilder, version 0.0.7
Filename, size File type Python version Upload date Hashes
Filename, size sitemapbuilder-0.0.7.tar.gz (6.1 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page