Skip to main content

pagespring

PyPI

Acquire and normalize online documentation into clean, convertible source files — the acquisition front-end to pagespeak.

Point it at a manual's URL. A pattern recognizes the source type, acquires the raw pages (stdlib urllib), and normalizes them into ONE clean HTML/markdown file with absolute asset URLs under incoming/<slug>/. That clean file is the deliverable; converting it into the finished RAG corpus is a separate step (pagespeak) that consumes incoming/ on its own — pagespring never runs it.

Lean by design: pw-foundation (PyPI) and pw-core[cli] (PyPI) + beautifulsoup4, pyyaml, and pypdfium2 for PDF page counts and for cutting PDFs printed as 2-up spreads into single pages. Stdlib fetch, no ML stack.

Intended use

pagespring is for publicly available documentation — vendor manuals, help centers, open textbooks, API specs. It fetches only what the source serves to any reader: there is no login/session handling, no paywall traversal, and no bot-detection evasion. Where a help site's pages load their articles with a form POST to the site's own API, it sends the same guest request a browser does. It is a polite client: it identifies itself with a pagespring/<version> User-Agent (PAGESPRING_UA overrides it), honors 429 Retry-After, backs off on server errors, paces crawl requests, and caps crawl sizes.

It is a user-invoked archiver — closer to "Save Page As" than to an autonomous crawler. Every source is a URL you supply (one per ingest, or one per line of an ingest --batch file), and it never discovers sources on its own, so it does not consult robots.txt (which governs bots that find URLs themselves), except to warn when a Fluid Topics portal's robots.txt disallows the API an ingest reads. Before mirroring a site, check its terms of use. What you may do with the acquired copy (personal RAG corpus, internal search, redistribution) is governed by the source's license — the deliverable under incoming/ stays on your machine, and nothing is re-published by this tool.

Install

pip install pagespring

Quick start

pagespring ingest https://docs.tableplus.com   # acquire + normalize → incoming/tableplus/
pagespring ingest --batch manuals.txt           # one ingest per URL line, with a summary
pagespring renormalize <slug>                   # replay normalize from kept raw/ — no re-crawl
pagespring refresh --all                        # re-check every manual against its source
pagespring audit --all                          # $0 sanity checks on everything staged
pagespring localize <slug>                      # pull a deliverable's images later (resumable; --all)
pagespring patterns                             # list the source patterns
pagespring classify <url>                       # which pattern handles a URL (no fetch; --probe names docs_probe's route)
pagespring status                               # what's been acquired

Deliverables land in ./incoming/<slug>/ under the directory you run from.

Dev

bin/setup   # clone → venv + editable install with dev extras
bin/test    # pytest
bin/lint    # ruff check + ruff format --check + mypy (strict) + structural gate + framework-first

See docs/usage.md for the full command set and docs/architecture.md for the acquire → normalize flow and how to add a new source pattern.

License

Apache-2.0 — see LICENSE. Releases through 0.11.0 were published under the MIT license and stay MIT.

Metadata

Release files for pagespring 0.20.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pagespring 0.20.0
File Size Uploaded Provenance Actions
pagespring-0.20.0.tar.gz 413.9 kB With provenance Details for pagespring-0.20.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pagespring 0.20.0
File Interpreter ABI Platform Actions
pagespring-0.20.0-py3-none-any.whl Python 3 none any Details for pagespring-0.20.0-py3-none-any.whl

Total release size: 639.6 kB

Release files / pagespring-0.20.0.tar.gz

Download URL pagespring-0.20.0.tar.gz
Size 413.9 kB
Tags Source
SHA-256 checksum
How to use checksums
12047a82fd432bebbb7789dc586eec4e0deadc3ac84273357908cb505c6d1dd3
BLAKE2b-256 checksum
How to use checksums
a1063ef333e2d1535e3b8836c943154f9b34fa770718cad4ad9e65d942bb0c3a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.

Transparency log

Release files / pagespring-0.20.0-py3-none-any.whl

Download URL pagespring-0.20.0-py3-none-any.whl
Size 225.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
937e61c8ed503c63bf1bd599cf4ccf6d9b369b3f666c9de1e2c65e8fb1f676cd
BLAKE2b-256 checksum
How to use checksums
f0983e0352835bd76d921ca1f325c57379c1a6c50e762e2069347b08744e82d4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.

Transparency log

Security for pagespring 0.20.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for pagespring-0.20.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page