Skip to main content

passageway-connector-sitemap

A passageway connector that indexes a website from its sitemap: a product's public documentation, a help centre, a company site. It reads sitemap.xml, fetches every page and, if you let it, follows the links on those pages to the ones the sitemap forgot.

What it does

  • Each page is one item. Its id is the page address and its version is a hash of the extracted text, so an edit re-indexes only that page. The main content of the page is kept (<main>, else <article>, else the body); navigation, headers, footers, scripts and forms are left out, and headings become sections.
  • A site address means its sitemap. https://docs.example.org is read as https://docs.example.org/sitemap.xml. A sitemap index (a sitemap that lists sitemaps) is followed.
  • It follows links, on the same host only. A link on an indexed page to another page of the site, one the sitemap did not list, is indexed too. Turn that off with FOLLOW_LINKS. Links to other hosts are never followed.
  • The folders of the address become the breadcrumb, and the links between indexed pages become related items.
  • It is polite. It reads robots.txt and skips what it forbids, waits between requests, identifies itself, and stops at a page limit.
  • A page that is gone is skipped, a server that fails stops the sync. A 404 or a file that is not HTML is skipped with a warning. A 5xx, a 429 or a network error fails the whole listing, so a site that is down for a minute never deletes what was indexed before.

Install

Add the connector to the environment that runs passageway (see Install):

uv tool install --python 3.14 "passageway[sitemap]"

An example

A page https://docs.example.org/guide/setup:

<html><head><title>Setup | Example</title></head>
<body><nav>Home Docs</nav>
<main><h1>Setup</h1><p>Install it.</p><h2>Linux</h2><p>Use the package manager.</p></main>
<footer>Copyright</footer></body></html>

becomes the item:

{
  "title": "Setup",
  "path": ["guide"],
  "sections": [
    {
      "heading": null,
      "text": "Install it."
    },
    {
      "heading": "Linux",
      "text": "Use the package manager."
    }
  ]
}

and is configured as:

[[connector]]
name = "docs"
command = ["/path/to/python", "-m", "passageway_connector_sitemap"]
env = { SITEMAP = "https://docs.example.org", EXCLUDE = "https://docs.example.org/blog/" }

Settings

All settings are strings in the connector's env.

Setting Meaning Default
SITEMAP Required. Site addresses or sitemap addresses, separated by |. Their hosts are the only ones crawled.
FOLLOW_LINKS true or false: also index pages that the sitemap does not list but an indexed page links to. true
INCLUDE Address prefixes, separated by |: only addresses that start with one are indexed. all
EXCLUDE Address prefixes, separated by |, that are never indexed. none
MAX_PAGES The most pages indexed. The rest are skipped with a warning. 500
PAUSE Seconds to wait between requests. 0.2
RESPECT_ROBOTS true or false: skip what robots.txt forbids for this crawler. true
RENDER true or false: open every page in a headless browser first, for sites whose pages a script builds. Needs passageway-connector-sitemap[render], and Chrome on the machine or playwright install chromium. false
CONTACT An address or URL added to the user agent, so a site's owner can reach you. none

A page that a script builds in the browser has little text in what the server sends: Teamscale's documentation pages, for one, hand back a one-line notice. Set RENDER = "true" and the connector opens each page in a headless Chromium and reads what the script built (slower, and it needs the render extra and a browser: the Google Chrome on the machine is used when there is one, else run playwright install chromium once, about 100 MB). If the site publishes an llms-full.txt or llms.txt, the llms.txt connector is faster and needs no browser.

Requests carry the user agent passageway-connector-sitemap, follow redirects and verify HTTPS against the operating system's trust store. Only crawl sites you may index.

Metadata

Release files for passageway-connector-sitemap 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for passageway-connector-sitemap 0.5.0
File Size Uploaded Provenance Actions
passageway_connector_sitemap-0.5.0.tar.gz 9.7 kB Without provenance Details for passageway_connector_sitemap-0.5.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for passageway-connector-sitemap 0.5.0
File Interpreter ABI Platform Actions
passageway_connector_sitemap-0.5.0-py3-none-any.whl Python 3 none any Details for passageway_connector_sitemap-0.5.0-py3-none-any.whl

Total release size: 16.5 kB

Release files / passageway_connector_sitemap-0.5.0.tar.gz

Download URL passageway_connector_sitemap-0.5.0.tar.gz
Size 9.7 kB
Tags Source
SHA-256 checksum
How to use checksums
e82d7fd429da1c265d5e458b73f259e5da6f32239362e03ff0562c127c4b299a
BLAKE2b-256 checksum
How to use checksums
a45ba8b72c8985a791f193dc7c35b5204d1ddff0aed7a52b47dee1ae741477f0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / passageway_connector_sitemap-0.5.0-py3-none-any.whl

Download URL passageway_connector_sitemap-0.5.0-py3-none-any.whl
Size 6.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
fdeceab10bd782ffac03594227887f7d4cda725851593fa3d0c4e0c00fb1420e
BLAKE2b-256 checksum
How to use checksums
6ef03ce675f233203c7bb5ee9bc900cd8dd9ad094908ad80418c64055ab3dba4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for passageway-connector-sitemap 0.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page