Skip to main content

passageway-connector-sitemap

A passageway connector that indexes a website from its sitemap: a product's public documentation, a help centre, a company site. It reads sitemap.xml, fetches every page and, if you let it, follows the links on those pages to the ones the sitemap forgot.

What it does

  • Each page is one item. Its id is the page address and its version is a hash of the extracted text, so an edit re-indexes only that page. The main content of the page is kept (<main>, else <article>, else the body); navigation, headers, footers, scripts and forms are left out, and headings become sections.
  • A site address means its sitemap. https://docs.example.org is read as https://docs.example.org/sitemap.xml. A sitemap index (a sitemap that lists sitemaps) is followed.
  • It follows links, on the same host only. A link on an indexed page to another page of the site, one the sitemap did not list, is indexed too. Turn that off with FOLLOW_LINKS. Links to other hosts are never followed.
  • The folders of the address become the breadcrumb, and the links between indexed pages become related items.
  • It is polite. It reads robots.txt and skips what it forbids, waits between requests, identifies itself, and stops at a page limit.
  • A page that is gone is skipped, a server that fails stops the sync. A 404 or a file that is not HTML is skipped with a warning. A 5xx, a 429 or a network error fails the whole listing, so a site that is down for a minute never deletes what was indexed before.

Install

Add the connector to the environment that runs passageway (see Install):

uv tool install --python 3.14 "passageway[sitemap]"

An example

A page https://docs.example.org/guide/setup:

<html><head><title>Setup | Example</title></head>
<body><nav>Home Docs</nav>
<main><h1>Setup</h1><p>Install it.</p><h2>Linux</h2><p>Use the package manager.</p></main>
<footer>Copyright</footer></body></html>

becomes the item:

{
  "title": "Setup",
  "path": ["guide"],
  "sections": [
    {
      "heading": null,
      "text": "Install it."
    },
    {
      "heading": "Linux",
      "text": "Use the package manager."
    }
  ]
}

and is configured as:

[[connector]]
name = "docs"
command = ["/path/to/python", "-m", "passageway_connector_sitemap"]
env = { SITEMAP = "https://docs.example.org", EXCLUDE = "https://docs.example.org/blog/" }

Settings

All settings are strings in the connector's env.

Setting Meaning Default
SITEMAP Required. Site addresses or sitemap addresses, separated by |. Their hosts are the only ones crawled.
FOLLOW_LINKS true or false: also index pages that the sitemap does not list but an indexed page links to. true
INCLUDE Address prefixes, separated by |: only addresses that start with one are indexed. all
EXCLUDE Address prefixes, separated by |, that are never indexed. none
MAX_PAGES The most pages indexed. The rest are skipped with a warning. 500
PAUSE Seconds to wait between requests. 0.2
RESPECT_ROBOTS true or false: skip what robots.txt forbids for this crawler. true
RENDER true or false: open every page in a headless browser first, for sites whose pages a script builds. Needs passageway-connector-sitemap[render], and Chrome on the machine or playwright install chromium. false
CONTACT An address or URL added to the user agent, so a site's owner can reach you. none

A page that a script builds in the browser has little text in what the server sends: Teamscale's documentation pages, for one, hand back a one-line notice. Set RENDER = "true" and the connector opens each page in a headless Chromium and reads what the script built (slower, and it needs the render extra and a browser: the Google Chrome on the machine is used when there is one, else run playwright install chromium once, about 100 MB). If the site publishes an llms-full.txt or llms.txt, the llms.txt connector is faster and needs no browser.

Requests carry the user agent passageway-connector-sitemap, follow redirects and verify HTTPS against the operating system's trust store. Only crawl sites you may index.

Metadata

Release files for passageway-connector-sitemap 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for passageway-connector-sitemap 0.4.0
File Size Uploaded Provenance Actions
passageway_connector_sitemap-0.4.0.tar.gz 9.7 kB Without provenance Details for passageway_connector_sitemap-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for passageway-connector-sitemap 0.4.0
File Interpreter ABI Platform Actions
passageway_connector_sitemap-0.4.0-py3-none-any.whl Python 3 none any Details for passageway_connector_sitemap-0.4.0-py3-none-any.whl

Total release size: 16.5 kB

Release files / passageway_connector_sitemap-0.4.0.tar.gz

Download URL passageway_connector_sitemap-0.4.0.tar.gz
Size 9.7 kB
Tags Source
SHA-256 checksum
How to use checksums
d3d48b9c20b8326c1dc868005ce2d2d1fe1ee44ebb8ccf74029735502c8a85e4
BLAKE2b-256 checksum
How to use checksums
a38c5b476b3749797b45072a8e05625acb54b3926340085fde62fb6cb26ffd04
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / passageway_connector_sitemap-0.4.0-py3-none-any.whl

Download URL passageway_connector_sitemap-0.4.0-py3-none-any.whl
Size 6.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a186b64707a7840292d24d5b38b3c6fba3cac484f29e6e82f032449ebaa212b9
BLAKE2b-256 checksum
How to use checksums
8b730deccc7719c65cea62ce8d2b665531172ca54cef3e2accc3e6d2feea6c2f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for passageway-connector-sitemap 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page