Context.dev for Haystack
Haystack components for live web search, webpage and YouTube transcript retrieval, and bounded website crawling with Context.dev.
Installation
pip install context-dev-haystack
Create an API key in the Context.dev dashboard, then export it:
export CONTEXT_API_KEY="your-api-key"
Components
| Component | Purpose | Import |
|---|---|---|
ContextWebSearch |
Search the live web and return ranked Haystack Documents and source links | haystack_integrations.components.websearch.context |
ContextFetcher |
Fetch webpages or YouTube videos as clean Markdown Documents | haystack_integrations.components.fetchers.context |
ContextCrawler |
Crawl websites into Documents with explicit page and depth limits | haystack_integrations.components.fetchers.context |
All components support both run() and run_async(), Haystack serialization, custom timeouts, and retry configuration.
Search the live web
from haystack_integrations.components.websearch.context import ContextWebSearch
search = ContextWebSearch(top_k=5, include_markdown=True)
result = search.run(query="Recent advances in retrieval-augmented generation")
documents = result["documents"]
links = result["links"]
Use include_domains, exclude_domains, freshness, and country to constrain results. Extra Context.dev Search API fields can be supplied through search_params.
Fetch webpages or YouTube transcripts
from haystack_integrations.components.fetchers.context import ContextFetcher
fetcher = ContextFetcher()
result = fetcher.run(
urls=[
"https://haystack.deepset.ai",
"https://www.youtube.com/watch?v=UF8uR6Z6KLc",
]
)
documents = result["documents"]
Each URL becomes a Haystack Document. Webpages contain clean Markdown and page metadata; supported YouTube URLs return timestamped transcript Markdown.
Crawl a website
from haystack_integrations.components.fetchers.context import ContextCrawler
crawler = ContextCrawler(crawl_params={"maxPages": 25, "maxDepth": 2})
result = crawler.run(urls=["https://docs.haystack.deepset.ai"])
documents = result["documents"]
ContextCrawler defaults to one page to prevent accidental credit consumption. Set maxPages explicitly for larger crawls.
Async usage
result = await search.run_async(query="Haystack agents")
documents = result["documents"]
The fetcher and crawler process multiple input URLs concurrently in their async methods.
Development
See CONTRIBUTING.md for the Hatch-based development and release workflow.
License
Apache-2.0. See LICENSE.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file context_dev_haystack-0.1.0.tar.gz.
File metadata
- Download URL: context_dev_haystack-0.1.0.tar.gz
- Upload date:
- Size: 17.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
7eb3b1f86777c0a7838b159329ac1647099f92e543bd3b8b83f1c54532d339c9
|
|
| MD5 |
7eb7ba9f97d32abc80a29d6e92c4e6d1
|
|
| BLAKE2b-256 |
afa73e39298acb69f65ab377271c7bfcdc81ef567b9380aa3cfe29a7e9e42982
|
Provenance
The following attestation bundles were made for context_dev_haystack-0.1.0.tar.gz:
Publisher:
release.yml on context-dot-dev/context-haystack
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
context_dev_haystack-0.1.0.tar.gz -
Subject digest:
7eb3b1f86777c0a7838b159329ac1647099f92e543bd3b8b83f1c54532d339c9 - Sigstore transparency entry: 2683587068
- Sigstore integration time:
-
Permalink:
context-dot-dev/context-haystack@cd21b9744e0bd6858b75263eb9d3473a7b839827 -
Branch / Tag:
refs/tags/v0.1.0 - Owner: https://github.com/context-dot-dev
-
Access:
internal
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@cd21b9744e0bd6858b75263eb9d3473a7b839827 -
Trigger Event:
push
-
Statement type:
File details
Details for the file context_dev_haystack-0.1.0-py3-none-any.whl.
File metadata
- Download URL: context_dev_haystack-0.1.0-py3-none-any.whl
- Upload date:
- Size: 15.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3b7c44457a4b39d7414ba8f3e7c48f7c383969742bb95b8378a254964a31ec18
|
|
| MD5 |
8b24804939962b0494bc3976baaf9430
|
|
| BLAKE2b-256 |
19234ff24fe954fa6df2b0b9d3a6935b645124c50374bff9946ce683c0249aae
|
Provenance
The following attestation bundles were made for context_dev_haystack-0.1.0-py3-none-any.whl:
Publisher:
release.yml on context-dot-dev/context-haystack
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
context_dev_haystack-0.1.0-py3-none-any.whl -
Subject digest:
3b7c44457a4b39d7414ba8f3e7c48f7c383969742bb95b8378a254964a31ec18 - Sigstore transparency entry: 2683587078
- Sigstore integration time:
-
Permalink:
context-dot-dev/context-haystack@cd21b9744e0bd6858b75263eb9d3473a7b839827 -
Branch / Tag:
refs/tags/v0.1.0 - Owner: https://github.com/context-dot-dev
-
Access:
internal
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@cd21b9744e0bd6858b75263eb9d3473a7b839827 -
Trigger Event:
push
-
Statement type:
