DocPull
Active open-source project · MIT License
DocPull turns changing public web sources into cited, reproducible context for AI agents and retrieval pipelines. Use it when your application needs to know which sources it used, whether they changed, and how to rebuild the same context later.
Install and sync your first source
pip install docpull
docpull init stripe-docs
docpull add https://docs.stripe.com
docpull sync
docpull diff
docpull export context-pack --target cursor
The project stores declared sources in docpull.yaml and resolved inputs in
.docpull/context.lock.json. Later syncs produce a hash-based diff while preserving
source URLs, content hashes, run IDs, citations, and export metadata.
No account or paid API is required for this path. Direct fetching, discovery, extraction, indexing, pack analysis, and diffs run locally.
Why use DocPull
- Reproduce agent context. Stable IDs, hashes, manifests, and lockfiles show which source versions produced an answer or artifact.
- Detect source drift. Sync and diff documentation, product pages, policies, feeds, repositories, packages, standards, and local documents.
- Keep evidence inspectable. Markdown, NDJSON, SQLite, citations, and provenance sidecars remain readable without a hosted service.
- Choose the downstream surface. Export context for agent clients, vector import, data workflows, or a versioned context-pack release.
- Keep expensive routes explicit. Browser and cloud rendering require an explicit choice and can be blocked with a zero-dollar budget.
How it works
declared sources → local acquisition → versioned evidence → diff and validation → export
DocPull's v3 pack contract separates raw extraction, agent-ready context, and eval-grade evidence. Validate the level a downstream system requires:
docpull pack prepare packs/docs --eval-grade
docpull pack validate packs/docs --level eval
docpull ci --prepare
docpull ci checks freshness, citation coverage, pack quality, rights metadata, and
other configured gates. It writes context-ci.report.json and CONTEXT_CI.md, then
exits non-zero when a hard gate fails.
Supported surfaces
| Surface | Use it for | Start here |
|---|---|---|
| CLI | Fetch, sync, diff, validate, and export | docs/cli-recipes.md |
| Python SDK | Embed acquisition in Python applications | docs/surface-contract.md |
| MCP server | Give local agent clients source tools | MCP server |
| TypeScript SDK | Read local packs and invoke the CLI from Node or Bun | sdk/js/README.md |
| Agent plugin | Install the supported MCP workflow in an agent client | plugin/README.md |
Common source shapes include static and server-rendered websites, OpenAPI documents, feeds, papers, public GitHub repositories, npm and PyPI packages, standards, datasets, transcripts, Wikimedia pages, product and policy pages, and local PDF or office files. See context-pack workflows for the complete surface.
MCP server
pip install 'docpull[mcp]'
docpull mcp
Claude Code can register the same local server:
claude mcp add --transport stdio docpull -- docpull mcp
The Python stdio server is the supported release path. The TypeScript code formerly
documented under mcp/ is an internal semantic-search lab,
not part of the package contract.
Limits and security boundary
DocPull is an evidence-acquisition engine, not a hosted competitive-intelligence product. It owns fetching, explicit rendering adapters, versioning, citations, hashing, validation, replay, and export. Downstream products own scheduling, human review, approved claims, legal conclusions, accounts, and notifications.
The default path does not handle complex interactive browser workflows, CAPTCHAs, stealth scraping, or private dashboards. JavaScript rendering is explicit. Authenticated sources require environment-variable references; DocPull does not persist credential values in project artifacts.
Security defaults include HTTPS-only fetching, robots.txt compliance, SSRF and DNS rebinding protections, redirect guards, XXE protection, and path-traversal checks. Read the web-source boundary, security posture, and evidence-engine decision before extending acquisition behavior.
Documentation and evidence
- CLI recipes — Common commands and advanced workflows.
- Context dependencies — Project and lockfile model.
- Context Pack Contract v3 — Artifact levels and compatibility.
- Public contracts — Schemas and versioning rules.
- Alternatives — Browser automation and hosted extraction tradeoffs.
- Evaluation lab — Reproducible methods, results, and claim boundaries.
- Changelog — Release history.
Raintree open-source system
DocPull owns evidence acquisition and reproducible agent context. It can be used independently; the sibling projects do not imply a required integration or shared release cycle.
| Project | Responsibility |
|---|---|
| Raintree Standards | Defines governed requirements and evidence. |
| Trellis | Enforces shared JavaScript and TypeScript code policy. |
| HIG Doctor | Audits interface source and provides HIG guidance. |
| PolicyStrata | Tests cross-layer policy behavior. |
See the Raintree open-source portfolio for current lifecycle and distribution links.
Project policies
Contributing · Code of Conduct · Security · Metrics and evidence limits · Source repository · MIT License
1 maintainer
Release files for docpull 6.5.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| docpull-6.5.2.tar.gz | 860.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| docpull-6.5.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.6 MB
Release files / docpull-6.5.2.tar.gz
| Download URL | docpull-6.5.2.tar.gz |
|---|---|
| Size | 860.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2c8b9f960ce0744a68f6be1c688ebadc2626fdb1c25f3dc02e93e127e7dddace
|
|
BLAKE2b-256 checksum How to use checksums |
eb1a7adf5f9b61cb8a47d410a4289d5c0e8ad1dce6e57e8dc1381b09384a1372
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 4, 2026.
Transparency logRelease files / docpull-6.5.2-py3-none-any.whl
| Download URL | docpull-6.5.2-py3-none-any.whl |
|---|---|
| Size | 764.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
769b6d180f08d08806f628a298040ecbc57e4577a6d9d632a29fcc3e4df9cc4b
|
|
BLAKE2b-256 checksum How to use checksums |
df3d00488a7e7c26e0e5ed3fbc2ddac64979099019d6d54df67ba0ac1c2c22c6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.13
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 4, 2026.
Transparency log
