Skip to main content

pagespring

PyPI

Acquire and normalize online documentation into clean, convertible source files — the acquisition front-end to pagespeak.

Point it at a manual's URL. A pattern recognizes the source type, acquires the raw pages (stdlib urllib), and normalizes them into ONE clean HTML/markdown file with absolute asset URLs under incoming/<slug>/. That clean file is the deliverable; converting it into the finished RAG corpus is a separate step (pagespeak) that consumes incoming/ on its own — pagespring never runs it.

Lean by design: pw-foundation (PyPI) and pw-core[cli] (PyPI) + beautifulsoup4, pyyaml, and pypdfium2 for PDF page counts and for cutting PDFs printed as 2-up spreads into single pages. Stdlib fetch, no ML stack.

Intended use

pagespring is for publicly available documentation — vendor manuals, help centers, open textbooks, API specs. It fetches only what the source serves to any reader: there is no login/session handling, no paywall traversal, and no bot-detection evasion. Where a help site's pages load their articles with a form POST to the site's own API, it sends the same guest request a browser does. It is a polite client: it identifies itself with a pagespring/<version> User-Agent (PAGESPRING_UA overrides it), honors 429 Retry-After, backs off on server errors, paces crawl requests, and caps crawl sizes.

It is a user-invoked archiver — closer to "Save Page As" than to an autonomous crawler. Every source is a URL you supply (one per ingest, or one per line of an ingest --batch file), and it never discovers sources on its own, so it does not consult robots.txt (which governs bots that find URLs themselves), except to warn when a Fluid Topics portal's robots.txt disallows the API an ingest reads. Before mirroring a site, check its terms of use. What you may do with the acquired copy (personal RAG corpus, internal search, redistribution) is governed by the source's license — the deliverable under incoming/ stays on your machine, and nothing is re-published by this tool.

Install

pip install pagespring

Quick start

pagespring ingest https://docs.tableplus.com   # acquire + normalize → incoming/tableplus/
pagespring ingest --batch manuals.txt           # one ingest per URL line, with a summary
pagespring renormalize <slug>                   # replay normalize from kept raw/ — no re-crawl
pagespring refresh --all                        # re-check every manual against its source
pagespring audit --all                          # $0 sanity checks on everything staged
pagespring localize <slug>                      # pull a deliverable's images later (resumable; --all)
pagespring patterns                             # list the source patterns
pagespring classify <url>                       # which pattern handles a URL (no fetch; --probe names docs_probe's route)
pagespring status                               # what's been acquired

Deliverables land in ./incoming/<slug>/ under the directory you run from.

Dev

bin/setup   # clone → venv + editable install with dev extras
bin/test    # pytest
bin/lint    # ruff check + ruff format --check + mypy (strict) + structural gate + framework-first

See docs/usage.md for the full command set and docs/architecture.md for the acquire → normalize flow and how to add a new source pattern.

License

Apache-2.0 — see LICENSE. Releases through 0.11.0 were published under the MIT license and stay MIT.

Metadata

Release files for pagespring 0.18.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pagespring 0.18.0
File Size Uploaded Provenance Actions
pagespring-0.18.0.tar.gz 399.7 kB With provenance Details for pagespring-0.18.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pagespring 0.18.0
File Interpreter ABI Platform Actions
pagespring-0.18.0-py3-none-any.whl Python 3 none any Details for pagespring-0.18.0-py3-none-any.whl

Total release size: 621.6 kB

Release files / pagespring-0.18.0.tar.gz

Download URL pagespring-0.18.0.tar.gz
Size 399.7 kB
Tags Source
SHA-256 checksum
How to use checksums
87458cd112630d6bb3ae08f18132844003990ce11dd77644d5914d9107c93992
BLAKE2b-256 checksum
How to use checksums
9937ff08a01d49201cbdca27b0adca75389ffb3196cabedfcb982ed6b6f12641
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

Release files / pagespring-0.18.0-py3-none-any.whl

Download URL pagespring-0.18.0-py3-none-any.whl
Size 221.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
47642845f3746f5f458d4494fef385e44b93098d54ee85960f5c1b8cc5651c3f
BLAKE2b-256 checksum
How to use checksums
467f2b7b6ceaf343b57ac69e68ea9bf45b79e2e361128b6e11f88a8ed515e938
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

Security for pagespring 0.18.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for pagespring-0.18.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page