passageway-connector-sitemap
A passageway connector that indexes a website from its sitemap: a product's public documentation, a help centre, a company site. It reads sitemap.xml, fetches every page and, if you let it, follows the links on those pages to the ones the sitemap forgot.
What it does
- Each page is one item. Its id is the page address and its version is a hash of the extracted text, so an edit re-indexes only that page. The main content of the page is kept (
<main>, else<article>, else the body); navigation, headers, footers, scripts and forms are left out, and headings become sections. - A site address means its sitemap.
https://docs.example.orgis read ashttps://docs.example.org/sitemap.xml. A sitemap index (a sitemap that lists sitemaps) is followed. - It follows links, on the same host only. A link on an indexed page to another page of the site, one the sitemap did not list, is indexed too. Turn that off with
FOLLOW_LINKS. Links to other hosts are never followed. - The folders of the address become the breadcrumb, and the links between indexed pages become related items.
- It is polite. It reads
robots.txtand skips what it forbids, waits between requests, identifies itself, and stops at a page limit. - A page that is gone is skipped, a server that fails stops the sync. A
404or a file that is not HTML is skipped with a warning. A5xx, a429or a network error fails the whole listing, so a site that is down for a minute never deletes what was indexed before.
Install
Add the connector to the environment that runs passageway (see Install):
uv tool install \
--with 'passageway-connector-sitemap @ git+https://github.com/getpassageway/passageway#subdirectory=connectors/sitemap' \
git+https://github.com/getpassageway/passageway
An example
A page https://docs.example.org/guide/setup:
<html><head><title>Setup | Example</title></head>
<body><nav>Home Docs</nav>
<main><h1>Setup</h1><p>Install it.</p><h2>Linux</h2><p>Use the package manager.</p></main>
<footer>Copyright</footer></body></html>
becomes the item:
{
"title": "Setup",
"path": ["guide"],
"sections": [
{
"heading": null,
"text": "Install it."
},
{
"heading": "Linux",
"text": "Use the package manager."
}
]
}
and is configured as:
[[connector]]
name = "docs"
command = ["/path/to/python", "-m", "passageway_connector_sitemap"]
env = { SITEMAP = "https://docs.example.org", EXCLUDE = "https://docs.example.org/blog/" }
Settings
All settings are strings in the connector's env.
| Setting | Meaning | Default |
|---|---|---|
SITEMAP |
Required. Site addresses or sitemap addresses, separated by |. Their hosts are the only ones crawled. |
|
FOLLOW_LINKS |
true or false: also index pages that the sitemap does not list but an indexed page links to. |
true |
INCLUDE |
Address prefixes, separated by |: only addresses that start with one are indexed. |
all |
EXCLUDE |
Address prefixes, separated by |, that are never indexed. |
none |
MAX_PAGES |
The most pages indexed. The rest are skipped with a warning. | 500 |
PAUSE |
Seconds to wait between requests. | 0.2 |
RESPECT_ROBOTS |
true or false: skip what robots.txt forbids for this crawler. |
true |
RENDER |
true or false: open every page in a headless browser first, for sites whose pages a script builds. Needs passageway-connector-sitemap[render], and Chrome on the machine or playwright install chromium. |
false |
CONTACT |
An address or URL added to the user agent, so a site's owner can reach you. | none |
A page that a script builds in the browser has little text in what the server sends: Teamscale's documentation pages, for one, hand back a one-line notice. Set RENDER = "true" and the connector opens each page in a headless Chromium and reads what the script built (slower, and it needs the render extra and a browser: the Google Chrome on the machine is used when there is one, else run playwright install chromium once, about 100 MB). If the site publishes an llms-full.txt or llms.txt, the llms.txt connector is faster and needs no browser.
Requests carry the user agent passageway-connector-sitemap, follow redirects and verify HTTPS against the operating system's trust store. Only crawl sites you may index.
Metadata
Release files for passageway-connector-sitemap 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| passageway_connector_sitemap-0.3.0.tar.gz | 10.4 kB | Without provenance | Details for passageway_connector_sitemap-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| passageway_connector_sitemap-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for passageway_connector_sitemap-0.3.0-py3-none-any.whl |
Total release size: 17.9 kB
Release files / passageway_connector_sitemap-0.3.0.tar.gz
| Download URL | passageway_connector_sitemap-0.3.0.tar.gz |
|---|---|
| Size | 10.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ad5cb3dd2747181a55a4b0b1bf5cfffc145a82c37546292382127e51397ed65a
|
|
BLAKE2b-256 checksum How to use checksums |
17026ae3884935fe06b6672ef39fe2222c994e5b4243d9dfbf51f4ce415b205d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / passageway_connector_sitemap-0.3.0-py3-none-any.whl
| Download URL | passageway_connector_sitemap-0.3.0-py3-none-any.whl |
|---|---|
| Size | 7.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ccef6fd0c77ff2c12042e3f1f5ce9d48f5b1b0b26b6a185bb8fdce6901001b7a
|
|
BLAKE2b-256 checksum How to use checksums |
88d406f3353d5f4c54d611399bf751cdd0956460753e47d52cf52104e8d2e65e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for passageway-connector-sitemap 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.