Twitter search and user timeline components for Haystack

TwexAPI is an independent third-party service. Not affiliated with X Corp. "Twitter" and "X" are trademarks of X Corp.

Search Twitter and fetch public user timelines in Haystack RAG pipelines. Each TwexAPI result becomes a Haystack Document.

Components

from haystack_integrations.components.websearch.x_api_scraper import (
    TwexApiTweetSearch,
    TwexApiUserTweetsFetcher,
)

Both components read X_API_SCRAPER_KEY by default and accept a Haystack Secret. They send Authorization: Bearer. Set base_url for another TwexAPI-compatible endpoint. Results include documents, links, has_more, and next_cursor.

Search returns up to 20 tweets per page. Timeline top_k maps to the page count (1-100). Use REST or an SDK for follower pagination or approved posting.

Install

pip install x-api-scraper-haystack

Build a Haystack RAG pipeline with Twitter search

Search posts

from haystack import Pipeline
from haystack.utils import Secret
from haystack_integrations.components.websearch.x_api_scraper import TwexApiTweetSearch

search = TwexApiTweetSearch(api_key=Secret.from_env_var("X_API_SCRAPER_KEY"), top_k=10)

pipeline = Pipeline()
pipeline.add_component("x_search", search)

result = pipeline.run({"x_search": {"query": "haystack ai"}})
documents = result["x_search"]["documents"]

Fetch user posts

from haystack.utils import Secret
from haystack_integrations.components.websearch.x_api_scraper import (
    TwexApiUserTweetsFetcher,
)

fetcher = TwexApiUserTweetsFetcher(api_key=Secret.from_env_var("X_API_SCRAPER_KEY"))

result = fetcher.run(screen_name="elonmusk")
documents = result["documents"]

Document mapping

Each tweet becomes a Haystack Document with this mapping.

• Document.content: full_text or text, or an empty string when both are missing
• Document.meta["endpoint"]: search or timeline
• Document.meta: available id, url, and created_at source values
• Document.meta["author"]: available author identity and verification data
• Document.meta: available like, retweet, reply, quote, view, and bookmark counts

License

MIT. TwexAPI is an independent third-party service. Not affiliated with X Corp. "Twitter" and "X" are trademarks of X Corp.