fetch-sitemap

Retrieves all URLs of a given sitemap.xml URL and fetches each page one by one. Useful for (load) testing the entire site for error responses.

Sample Output

Installation

$ pip install fetch-sitemap

Usage

$ fetch-sitemap --help

 Usage: fetch-sitemap [OPTIONS] SITEMAP_URL

 Fetch a given sitemap and retrieve all URLs in it.

╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --basic-auth         -a  TEXT       Basic auth information. Format: 'username:password'                            │
│ --limit              -l  INTEGER    Maximum number of URLs to fetch from the given sitemap.xml.                    │
│ --concurrency-limit  -c  INTEGER    Max number of concurrent requests. [default: 5]                                │
│ --request-timeout    -t  INTEGER    Timeout for fetching a URL in seconds. [default: 30]                           │
│ --random             -r             Append a random string like ?12334232343 to each URL to bypass frontend cache. │
│ --random-length          INTEGER    Length of the --random hash. [default: 15]                                     │
│ --report-path        -p  FILE       Store results in a CSV file. Example: ./report.csv                             │
│ --output-dir         -o  DIRECTORY  Store all fetched sitemap documents in this folder. Example:                   │
│                                     /tmp/my.domain.com/                                                            │
│ --slow-threshold         FLOAT      Responses slower than this (in seconds) are considered 'slow'. [default: 5.0]  │
│ --slow-num               INTEGER    How many 'slow' responses to show. [default: 10]                               │
│ --version            -v             Show the version and exit.                                                     │
│ --help                              Show this message and exit.                                                    │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

🤺 Local Development

poetry install
poetry run fetch-sitemap -h
poetry run ./tests.sh

Release files for fetch-sitemap 17

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for fetch-sitemap 17
File	Size	Uploaded
fetch_sitemap-17.tar.gz	6.3 kB	Apr 25, 2024	Details

Built distribution (wheel)

Table of built distributions (wheels) for fetch-sitemap 17
File	Interpreter	ABI	Platform	Reset
fetch_sitemap-17-py3-none-any.whl 7.2 kB Apr 25, 2024	Python 3	none	any	Details

Total release size: 13.5 kB

Release files / fetch_sitemap-17.tar.gz

Download URL	fetch_sitemap-17.tar.gz
Size	6.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`3bbdea6dbe949a7bbd760f67b086dde8271b374e620030c3f8986c16afd09d8c`
BLAKE2b-256 checksum How to use checksums	`09bdb4f8f24bbc9c70b5a4e99e78c4883826abbc98a5de6bd500af74791e3a32`
Upload date	Apr 25, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.8.2 CPython/3.12.3 Darwin/23.4.0`

Release files / fetch_sitemap-17-py3-none-any.whl

Download URL	fetch_sitemap-17-py3-none-any.whl
Size	7.2 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`64c7ad75c7f6f13ffbcb4c377849e70bf667cde543e4833a4a3caa29dfa2ae33`
BLAKE2b-256 checksum How to use checksums	`62921c7d344114ba18f9d81a3e9ed2e35d1933a165bfeadc2fc1acc26b815303`
Upload date	Apr 25, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.8.2 CPython/3.12.3 Darwin/23.4.0`