fetch-sitemap

Retrieves all URL's of a given sitemap.xml URL and fetches each page one by one. Useful for (load) testing the entire site for error responses.

Sample Output

Note: The default concurrency limit is 10, so 10 URLs are fetched at once. Depending on your server's worker count, this might already be enough to DoS it. Try --concurrency-limit=2 and increase if you feel comfortable.

usage: fetch-sitemap 
    [-h] 
    [--basic-auth BASIC_AUTH] 
    [-l LIMIT] 
    [-c CONCURRENCY_LIMIT] 
    [-t REQUEST_TIMEOUT] 
    [--report-path REPORT_PATH] 
    sitemap_url

Fetch a given sitemap and retrieve all URLs in it.

positional arguments:
  sitemap_url           URL of the sitemap to fetch

options:
  -h, --help            show this help message and exit
  --basic-auth BASIC_AUTH
                        Basic auth information. Use: 'username:password'.
  -l LIMIT, --limit LIMIT
                        Maximum number of URLs to fetch from the given sitemap.xml. Default: All
  -c CONCURRENCY_LIMIT, --concurrency-limit CONCURRENCY_LIMIT
                        Max number of concurrent requests. Default: 10
  -t REQUEST_TIMEOUT, --request-timeout REQUEST_TIMEOUT
                        Timeout for fetching a URL in seconds. Default: 30
  --random              Append a random string like ?12334232343 to each URL to bypass frontend cache. Default: False
  --report-path REPORT_PATH
                        Store results in a CSV file. Example: ./report.csv
  -o OUTPUT, --output-dir OUTPUT
                        Store all fetched sitemap documents in this folder.

License
- OSI Approved :: MIT License
Operating System
- OS Independent
Programming Language
- Python :: 3

Release files for fetch-sitemap 6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for fetch-sitemap 6
File	Size	Uploaded
fetch-sitemap-6.tar.gz	498.6 kB	Oct 25, 2023	Details

Built distribution (wheel)

Table of built distributions (wheels) for fetch-sitemap 6
File	Interpreter	ABI	Platform	Reset
fetch_sitemap-6-py3-none-any.whl 6.9 kB Oct 25, 2023	Python 3	none	any	Details

Total release size: 505.5 kB

Release files / fetch-sitemap-6.tar.gz

Download URL	fetch-sitemap-6.tar.gz
Size	498.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`7f326c638aa7f1a63f0eb3bf62514618aa3a21c8888545aed40ca134bd4c6816`
BLAKE2b-256 checksum How to use checksums	`fccb4ff5ecde20760046ed4ceb2daac1a4ab534f0125e885a450ac0739b957ad`
Upload date	Oct 25, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.12.0`

Release files / fetch_sitemap-6-py3-none-any.whl

Download URL	fetch_sitemap-6-py3-none-any.whl
Size	6.9 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`9505510d55eac2990f5bb70abd267c7131eee6c2787c2d0997793f41c7b73abf`
BLAKE2b-256 checksum How to use checksums	`2f2615ae8ad3f5567f0b0d72fb3b0f35f018b154a2ebf2a09a5c7ac6d15045ff`
Upload date	Oct 25, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.12.0`