Skip to main content

pagespring

PyPI

Acquire and normalize online documentation into clean, convertible source files — the acquisition front-end to pagespeak.

Point it at a manual's URL. A pattern recognizes the source type, acquires the raw pages (stdlib urllib), and normalizes them into ONE clean HTML/markdown file with absolute asset URLs under incoming/<slug>/. That clean file is the deliverable; converting it into the finished RAG corpus is a separate step (pagespeak) that consumes incoming/ on its own — pagespring never runs it.

Lean by design: pw-foundation (PyPI) and pw-core[cli] (PyPI) + beautifulsoup4, pyyaml, and pypdfium2 for PDF page counts and for cutting PDFs printed as 2-up spreads into single pages. Stdlib fetch, no ML stack.

Intended use

pagespring is for publicly available documentation — vendor manuals, help centers, open textbooks, API specs. It fetches only what the source serves to any reader: there is no login/session handling, no paywall traversal, and no bot-detection evasion. Where a help site's pages load their articles with a form POST to the site's own API, it sends the same guest request a browser does. It is a polite client: it identifies itself with a pagespring/<version> User-Agent (PAGESPRING_UA overrides it), honors 429 Retry-After, backs off on server errors, paces crawl requests, and caps crawl sizes.

It is a user-invoked archiver — closer to "Save Page As" than to an autonomous crawler. Every source is a URL you supply (one per ingest, or one per line of an ingest --batch file), and it never discovers sources on its own, so it does not consult robots.txt (which governs bots that find URLs themselves), except to warn when a Fluid Topics portal's robots.txt disallows the API an ingest reads. Before mirroring a site, check its terms of use. What you may do with the acquired copy (personal RAG corpus, internal search, redistribution) is governed by the source's license — the deliverable under incoming/ stays on your machine, and nothing is re-published by this tool.

Install

pip install pagespring

Quick start

pagespring ingest https://docs.tableplus.com   # acquire + normalize → incoming/tableplus/
pagespring ingest --batch manuals.txt           # one ingest per URL line, with a summary
pagespring renormalize <slug>                   # replay normalize from kept raw/ — no re-crawl
pagespring refresh --all                        # re-check every manual against its source
pagespring audit --all                          # $0 sanity checks on everything staged
pagespring localize <slug>                      # pull a deliverable's images later (resumable; --all)
pagespring patterns                             # list the source patterns
pagespring classify <url>                       # which pattern handles a URL (no fetch; --probe names docs_probe's route)
pagespring status                               # what's been acquired

Deliverables land in ./incoming/<slug>/ under the directory you run from.

Dev

bin/setup   # clone → venv + editable install with dev extras
bin/test    # pytest
bin/lint    # ruff check + ruff format --check + mypy (strict) + structural gate + framework-first

See docs/usage.md for the full command set and docs/architecture.md for the acquire → normalize flow and how to add a new source pattern.

License

Apache-2.0 — see LICENSE. Releases through 0.11.0 were published under the MIT license and stay MIT.

Metadata

Release files for pagespring 0.21.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pagespring 0.21.0
File Size Uploaded Provenance Actions
pagespring-0.21.0.tar.gz 424.1 kB With provenance Details for pagespring-0.21.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pagespring 0.21.0
File Interpreter ABI Platform Actions
pagespring-0.21.0-py3-none-any.whl Python 3 none any Details for pagespring-0.21.0-py3-none-any.whl

Total release size: 653.4 kB

Release files / pagespring-0.21.0.tar.gz

Download URL pagespring-0.21.0.tar.gz
Size 424.1 kB
Tags Source
SHA-256 checksum
How to use checksums
c047525ce1c3f2a891de54213eef81b65abe62d2738a844262f7c97c5761feae
BLAKE2b-256 checksum
How to use checksums
7346d729bb0125d297d2ba64e1a1c31e7929144dd02cba5f0d9fc1f2e481a53f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Release files / pagespring-0.21.0-py3-none-any.whl

Download URL pagespring-0.21.0-py3-none-any.whl
Size 229.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c46ba072d1b1d294c30f95746060e3862174df3b359fbbca11295810e0b7207e
BLAKE2b-256 checksum
How to use checksums
65e440524bbf131aefbbf811518705695ea9348fb8800d0cf563eafcc9447378
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Security for pagespring 0.21.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for pagespring-0.21.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page