Fetch a given sitemap and retrieve all URLs in it.
Project description
fetch-sitemap
Retrieves all URLs of a given sitemap.xml URL and fetches each page one by one. Useful for (load) testing the entire site for error responses.
Installation
$ pip install fetch-sitemap
Usage
$ fetch-sitemap --help
Usage: fetch-sitemap [OPTIONS] SITEMAP_URL
Fetch a given sitemap and retrieve all URLs in it.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --basic-auth -a TEXT Basic auth information. Format: 'username:password' │
│ --limit -l INTEGER Maximum number of URLs to fetch from the given sitemap.xml. │
│ --concurrency-limit -c INTEGER Max number of concurrent requests. [default: 5] │
│ --request-timeout -t INTEGER Timeout for fetching a URL in seconds. [default: 30] │
│ --random -r Append a random string like ?12334232343 to each URL to bypass frontend cache. │
│ --random-length INTEGER Length of the --random hash. [default: 15] │
│ --report-path -p FILE Store results in a CSV file. Example: ./report.csv │
│ --output-dir -o DIRECTORY Store all fetched sitemap documents in this folder. Example: │
│ /tmp/my.domain.com/ │
│ --slow-threshold FLOAT Responses slower than this (in seconds) are considered 'slow'. [default: 5.0] │
│ --slow-num INTEGER How many 'slow' responses to show. [default: 10] │
│ --version -v Show the version and exit. │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
🤺 Local Development
poetry install
poetry run fetch-sitemap -h
poetry run ./tests.sh
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
fetch_sitemap-18.tar.gz
(6.8 kB
view hashes)
Built Distribution
Close
Hashes for fetch_sitemap-18-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b88a3d47491f34fe391f204e36cb0a3cc8c76c8ee8723bfbd781186e1126bac |
|
MD5 | ffc154fc14c4b0ee76d03c83f902957c |
|
BLAKE2b-256 | a465d9cb8587f6a71f9459e7f36a43d10df63bf4a995ec283c0a57252dbd7b8f |