fetch-sitemap
Retrieves all URL's of a given sitemap.xml URL and fetches each page one by one. Useful for (load) testing the entire site for error responses.
Note: The default concurrency limit is 10, so 10 URLs are fetched at once.
Depending on your server's worker count, this might already be enough to DoS it.
Try --concurrency-limit=2 and increase if you feel comfortable.
usage: fetch-sitemap
[-h]
[--basic-auth BASIC_AUTH]
[-l LIMIT]
[-c CONCURRENCY_LIMIT]
[-t REQUEST_TIMEOUT]
[--report-path REPORT_PATH]
sitemap_url
Fetch a given sitemap and retrieve all URLs in it.
positional arguments:
sitemap_url URL of the sitemap to fetch
options:
-h, --help show this help message and exit
--basic-auth BASIC_AUTH
Basic auth information. Use: 'username:password'.
-l LIMIT, --limit LIMIT
Maximum number of URLs to fetch from the given sitemap.xml. Default: All
-c CONCURRENCY_LIMIT, --concurrency-limit CONCURRENCY_LIMIT
Max number of concurrent requests. Default: 10
-t REQUEST_TIMEOUT, --request-timeout REQUEST_TIMEOUT
Timeout for fetching a URL in seconds. Default: 30
--random Append a random string like ?12334232343 to each URL to bypass frontend cache. Default: False
--report-path REPORT_PATH
Store results in a CSV file. Example: ./report.csv
-o OUTPUT, --output-dir OUTPUT
Store all fetched sitemap documents in this folder.
1 maintainer
Release files for fetch-sitemap 4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| fetch-sitemap-4.tar.gz | 498.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| fetch_sitemap-4-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 504.7 kB
Release files / fetch-sitemap-4.tar.gz
| Download URL | fetch-sitemap-4.tar.gz |
|---|---|
| Size | 498.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c3043f2edc9f3ec255834f5033957162aa6826ca4c37c9d5b9e427add2fa5623
|
|
BLAKE2b-256 checksum How to use checksums |
4ff39e9b7ae8c74767e055091af7439d2ca984ea7639e61486ef4c657fda25af
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.2
|
Release files / fetch_sitemap-4-py3-none-any.whl
| Download URL | fetch_sitemap-4-py3-none-any.whl |
|---|---|
| Size | 6.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
390a65238f61d8ddb50f5ad32a93f9eb3c0e5e4c0057561d94606badd9041a52
|
|
BLAKE2b-256 checksum How to use checksums |
053d31d3165cb1269a50dfdc91e481653920e938b043d8d80dda01a7c2440a12
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.2
|
