Skip to main content

capturekit-haystack

Haystack integration for CaptureKit, a web page API that loads pages in a real browser and returns them as clean Markdown.

CaptureKitFetcher fetches web pages and returns them as Haystack Documents: the content is the page as Markdown, and the metadata holds url, title and description. No HTML converter is needed.

Installation

pip install capturekit-haystack

Get an API key at app.capturekit.dev and set it in the environment:

export CAPTUREKIT_API_KEY="your-api-key"

Usage

from haystack_integrations.components.fetchers.capturekit import CaptureKitFetcher

fetcher = CaptureKitFetcher()
docs = fetcher.run(urls=["https://example.com"])["documents"]
print(docs[0].meta["title"], docs[0].content[:200])
Parameter Default Description
api_key Secret.from_env_var("CAPTUREKIT_API_KEY") CaptureKit API key
main_content_only True Keep the article or product and drop menus, sidebars and footers
raise_on_failure True Raise when a single URL fails; with several URLs, failures are logged and skipped
timeout 120 Seconds to wait for each page

A failed request raises CaptureKitError, a Haystack ComponentError with the HTTP status. The API key never appears in the message. run_async is available too.

Index pages for RAG

from haystack import Pipeline
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.fetchers.capturekit import CaptureKitFetcher

store = InMemoryDocumentStore()
pipeline = Pipeline()
pipeline.add_component("fetcher", CaptureKitFetcher())
pipeline.add_component("splitter", DocumentSplitter(split_by="word", split_length=200))
pipeline.add_component("writer", DocumentWriter(document_store=store))
pipeline.connect("fetcher.documents", "splitter.documents")
pipeline.connect("splitter.documents", "writer.documents")

pipeline.run({"fetcher": {"urls": ["https://docs.haystack.deepset.ai/docs/intro"]}})
print(store.count_documents())

Development

pip install -e ".[test]"
pytest

License

MIT

Metadata

Release files for capturekit-haystack 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for capturekit-haystack 0.1.0
File Size Uploaded Provenance Actions
capturekit_haystack-0.1.0.tar.gz 7.1 kB Without provenance Details for capturekit_haystack-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for capturekit-haystack 0.1.0
File Interpreter ABI Platform Actions
capturekit_haystack-0.1.0-py3-none-any.whl Python 3 none any Details for capturekit_haystack-0.1.0-py3-none-any.whl

Total release size: 13.9 kB

Release files / capturekit_haystack-0.1.0.tar.gz

Download URL capturekit_haystack-0.1.0.tar.gz
Size 7.1 kB
Tags Source
SHA-256 checksum
How to use checksums
b476acd55319f88f983fdd2e6e146d07ba048a3afc67563794accfd707bb1312
BLAKE2b-256 checksum
How to use checksums
f2e39ad2eafa7ce94677e2402488d1e8e2dd349164cec7225853427860398ca7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Release files / capturekit_haystack-0.1.0-py3-none-any.whl

Download URL capturekit_haystack-0.1.0-py3-none-any.whl
Size 6.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
603c2e677fc342035b6862ba0c2fb801ce42d8bc57b825191f1dd9aac5b2c1d5
BLAKE2b-256 checksum
How to use checksums
09a4f217618da142a9801a700e689549b5af33529c245e2411f740b2aeaf82a1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Security for capturekit-haystack 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page