Skip to main content

passageway-connector-sitemap

A passageway connector that indexes a website from its sitemap: a product's public documentation, a help centre, a company site. It reads sitemap.xml, fetches every page and, if you let it, follows the links on those pages to the ones the sitemap forgot.

What it does

  • Each page is one item. Its id is the page address and its version is a hash of the extracted text, so an edit re-indexes only that page. The main content of the page is kept (<main>, else <article>, else the body); navigation, headers, footers, scripts and forms are left out, and headings become sections.
  • A site address means its sitemap. https://docs.example.org is read as https://docs.example.org/sitemap.xml. A sitemap index (a sitemap that lists sitemaps) is followed.
  • It follows links, on the same host only. A link on an indexed page to another page of the site, one the sitemap did not list, is indexed too. Turn that off with FOLLOW_LINKS. Links to other hosts are never followed.
  • The folders of the address become the breadcrumb, and the links between indexed pages become related items.
  • It is polite. It reads robots.txt and skips what it forbids, waits between requests, identifies itself, and stops at a page limit.
  • A page that is gone is skipped, a server that fails stops the sync. A 404 or a file that is not HTML is skipped with a warning. A 5xx, a 429 or a network error fails the whole listing, so a site that is down for a minute never deletes what was indexed before.

Install

Add the connector to the environment that runs passageway (see Install):

uv tool install \
  --with 'passageway-connector-sitemap @ git+https://github.com/getpassageway/passageway#subdirectory=connectors/sitemap' \
  git+https://github.com/getpassageway/passageway

An example

A page https://docs.example.org/guide/setup:

<html><head><title>Setup | Example</title></head>
<body><nav>Home Docs</nav>
<main><h1>Setup</h1><p>Install it.</p><h2>Linux</h2><p>Use the package manager.</p></main>
<footer>Copyright</footer></body></html>

becomes the item:

{
  "title": "Setup",
  "path": ["guide"],
  "sections": [
    {
      "heading": null,
      "text": "Install it."
    },
    {
      "heading": "Linux",
      "text": "Use the package manager."
    }
  ]
}

and is configured as:

[[connector]]
name = "docs"
command = ["/path/to/python", "-m", "passageway_connector_sitemap"]
env = { SITEMAP = "https://docs.example.org", EXCLUDE = "https://docs.example.org/blog/" }

Settings

All settings are strings in the connector's env.

Setting Meaning Default
SITEMAP Required. Site addresses or sitemap addresses, separated by |. Their hosts are the only ones crawled.
FOLLOW_LINKS true or false: also index pages that the sitemap does not list but an indexed page links to. true
INCLUDE Address prefixes, separated by |: only addresses that start with one are indexed. all
EXCLUDE Address prefixes, separated by |, that are never indexed. none
MAX_PAGES The most pages indexed. The rest are skipped with a warning. 500
PAUSE Seconds to wait between requests. 0.2
RESPECT_ROBOTS true or false: skip what robots.txt forbids for this crawler. true
RENDER true or false: open every page in a headless browser first, for sites whose pages a script builds. Needs passageway-connector-sitemap[render], and Chrome on the machine or playwright install chromium. false
CONTACT An address or URL added to the user agent, so a site's owner can reach you. none

A page that a script builds in the browser has little text in what the server sends: Teamscale's documentation pages, for one, hand back a one-line notice. Set RENDER = "true" and the connector opens each page in a headless Chromium and reads what the script built (slower, and it needs the render extra and a browser: the Google Chrome on the machine is used when there is one, else run playwright install chromium once, about 100 MB). If the site publishes an llms-full.txt or llms.txt, the llms.txt connector is faster and needs no browser.

Requests carry the user agent passageway-connector-sitemap, follow redirects and verify HTTPS against the operating system's trust store. Only crawl sites you may index.

Metadata

Release files for passageway-connector-sitemap 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for passageway-connector-sitemap 0.3.0
File Size Uploaded Provenance Actions
passageway_connector_sitemap-0.3.0.tar.gz 10.4 kB Without provenance Details for passageway_connector_sitemap-0.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for passageway-connector-sitemap 0.3.0
File Interpreter ABI Platform Actions
passageway_connector_sitemap-0.3.0-py3-none-any.whl Python 3 none any Details for passageway_connector_sitemap-0.3.0-py3-none-any.whl

Total release size: 17.9 kB

Release files / passageway_connector_sitemap-0.3.0.tar.gz

Download URL passageway_connector_sitemap-0.3.0.tar.gz
Size 10.4 kB
Tags Source
SHA-256 checksum
How to use checksums
ad5cb3dd2747181a55a4b0b1bf5cfffc145a82c37546292382127e51397ed65a
BLAKE2b-256 checksum
How to use checksums
17026ae3884935fe06b6672ef39fe2222c994e5b4243d9dfbf51f4ce415b205d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / passageway_connector_sitemap-0.3.0-py3-none-any.whl

Download URL passageway_connector_sitemap-0.3.0-py3-none-any.whl
Size 7.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ccef6fd0c77ff2c12042e3f1f5ce9d48f5b1b0b26b6a185bb8fdce6901001b7a
BLAKE2b-256 checksum
How to use checksums
88d406f3353d5f4c54d611399bf751cdd0956460753e47d52cf52104e8d2e65e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for passageway-connector-sitemap 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page