Skip to main content

pagespring

PyPI

Acquire and normalize online documentation into clean, convertible source files — the acquisition front-end to pagespeak.

Point it at a manual's URL. A pattern recognizes the source type, acquires the raw pages (stdlib urllib), and normalizes them into ONE clean HTML/markdown file with absolute asset URLs under incoming/<slug>/. That clean file is the deliverable; converting it into the finished RAG corpus is a separate step (pagespeak) that consumes incoming/ on its own — pagespring never runs it.

Lean by design: pw-foundation (PyPI) and pw-core[cli] (PyPI) + beautifulsoup4, pyyaml, and pypdfium2 for PDF page counts and for cutting PDFs printed as 2-up spreads into single pages. Stdlib fetch, no ML stack.

Intended use

pagespring is for publicly available documentation — vendor manuals, help centers, open textbooks, API specs. It fetches only what the source serves to any reader: there is no login/session handling, no paywall traversal, and no bot-detection evasion. Where a help site's pages load their articles with a form POST to the site's own API, it sends the same guest request a browser does. It is a polite client: it identifies itself with a pagespring/<version> User-Agent (PAGESPRING_UA overrides it), honors 429 Retry-After, backs off on server errors, paces crawl requests, and caps crawl sizes.

It is a user-invoked archiver — closer to "Save Page As" than to an autonomous crawler. Every source is a URL you supply (one per ingest, or one per line of an ingest --batch file), and it never discovers sources on its own, so it does not consult robots.txt (which governs bots that find URLs themselves), except to warn when a Fluid Topics portal's robots.txt disallows the API an ingest reads. Before mirroring a site, check its terms of use. What you may do with the acquired copy (personal RAG corpus, internal search, redistribution) is governed by the source's license — the deliverable under incoming/ stays on your machine, and nothing is re-published by this tool.

Install

pip install pagespring

Quick start

pagespring ingest https://docs.tableplus.com   # acquire + normalize → incoming/tableplus/
pagespring ingest --batch manuals.txt           # one ingest per URL line, with a summary
pagespring renormalize <slug>                   # replay normalize from kept raw/ — no re-crawl
pagespring refresh --all                        # re-check every manual against its source
pagespring audit --all                          # $0 sanity checks on everything staged
pagespring localize <slug>                      # pull a deliverable's images later (resumable; --all)
pagespring patterns                             # list the source patterns
pagespring classify <url>                       # which pattern handles a URL (no fetch; --probe names docs_probe's route)
pagespring status                               # what's been acquired

Deliverables land in ./incoming/<slug>/ under the directory you run from.

Dev

bin/setup   # clone → venv + editable install with dev extras
bin/test    # pytest
bin/lint    # ruff check + ruff format --check + mypy (strict) + structural gate + framework-first

See docs/usage.md for the full command set and docs/architecture.md for the acquire → normalize flow and how to add a new source pattern.

License

Apache-2.0 — see LICENSE. Releases through 0.11.0 were published under the MIT license and stay MIT.

Metadata

Release files for pagespring 0.19.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pagespring 0.19.0
File Size Uploaded Provenance Actions
pagespring-0.19.0.tar.gz 410.4 kB With provenance Details for pagespring-0.19.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pagespring 0.19.0
File Interpreter ABI Platform Actions
pagespring-0.19.0-py3-none-any.whl Python 3 none any Details for pagespring-0.19.0-py3-none-any.whl

Total release size: 634.4 kB

Release files / pagespring-0.19.0.tar.gz

Download URL pagespring-0.19.0.tar.gz
Size 410.4 kB
Tags Source
SHA-256 checksum
How to use checksums
da2459d1fe5bc9e691aef534ea995d01d6c08820ede762d0a76d0c588fa2165e
BLAKE2b-256 checksum
How to use checksums
acf86e9c491f3bd058028b6e0f5dc3d504dc347361366ef66eb2ed0dcd66c446
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

Release files / pagespring-0.19.0-py3-none-any.whl

Download URL pagespring-0.19.0-py3-none-any.whl
Size 224.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f8277e166ca344f871f00c6446e09f7614b5dff01eb9bf16ba8cfe69a40ce2f8
BLAKE2b-256 checksum
How to use checksums
3411dedd9afe1fc84f1fb494c178025262269cdbcab3580a2b26fa34b42da7e8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

Security for pagespring 0.19.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for pagespring-0.19.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page