langchain-trawl 0.1.0

LangChain integration for the Trawl unified content API — transcripts from 10+ sources

langchain-trawl

LangChain integration for the Trawl unified content API. Extract transcripts and text from 10+ sources as LangChain Document objects, ready for RAG pipelines, summarization chains, and more.

Installation

pip install langchain-trawl

This installs langchain-core and trawl-sdk as dependencies.

Quick Start

from langchain_trawl import TrawlLoader

loader = TrawlLoader(
    urls=["https://youtube.com/watch?v=dQw4w9WgXcQ"],
    api_key="trawl_...",  # or set TRAWL_API_KEY env var
)
docs = loader.load()

print(docs[0].page_content[:200])
print(docs[0].metadata)
# {'video_id': 'dQw4w9WgXcQ', 'title': '...', 'channel': '...', ...}

Supported Sources

Source	Example URL
YouTube	https://youtube.com/watch?v=...
Podcasts	https://podcasts.apple.com/... or RSS feed URL
TikTok	https://tiktok.com/@user/video/...
Instagram Reels	https://instagram.com/reel/...
Twitter/X Spaces	https://twitter.com/i/spaces/...
Reddit	https://reddit.com/r/sub/comments/...
News Articles	Any news article URL
Earnings Calls	Ticker symbol via Trawl API
SEC Filings	EDGAR filing URLs
Academic Papers	ArXiv or DOI URLs

Parameters

Parameter	Type	Default	Description
urls	list[str]	required	URLs to extract content from
api_key	str | None	None	Trawl API key (falls back to TRAWL_API_KEY env var)
language	str | None	None	Language code filter (e.g. "en", "es")
base_url	str	"https://api.gettrawl.com"	API base URL

Document Metadata

Each loaded document includes these metadata fields:

• video_id -- Source content identifier
• title -- Content title
• channel -- Channel or author name
• language -- Detected language code
• is_auto_generated -- Whether the transcript was auto-generated
• segment_count -- Number of transcript segments
• source -- Original URL

If extraction fails, the document will have empty page_content and an error key in metadata.

Use in RAG Pipelines

from langchain_trawl import TrawlLoader
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load transcripts
loader = TrawlLoader(
    urls=[
        "https://youtube.com/watch?v=video1",
        "https://youtube.com/watch?v=video2",
    ],
)
docs = loader.load()

# Split and embed
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split_documents(docs)
vectorstore = FAISS.from_documents(chunks, OpenAIEmbeddings())

# Query
results = vectorstore.similarity_search("your question here")

Development

# Install dependencies
poetry install

# Run tests
make test

# Lint and format
make lint
make format

License

MIT