langchain-trawl

LangChain integration for the Trawl unified content API. Extract transcripts and text from 10+ sources as LangChain Document objects, ready for RAG pipelines, summarization chains, and more.

Installation

pip install langchain-trawl

This installs langchain-core and trawl-sdk as dependencies.

Quick Start

from langchain_trawl import TrawlLoader

loader = TrawlLoader(
    urls=["https://youtube.com/watch?v=dQw4w9WgXcQ"],
    api_key="trawl_...",  # or set TRAWL_API_KEY env var
)
docs = loader.load()

print(docs[0].page_content[:200])
print(docs[0].metadata)
# {'video_id': 'dQw4w9WgXcQ', 'title': '...', 'channel': '...', ...}

Supported Sources

Parameters

Document Metadata

Each loaded document includes these metadata fields:

• video_id -- Source content identifier
• title -- Content title
• channel -- Channel or author name
• language -- Detected language code
• is_auto_generated -- Whether the transcript was auto-generated
• segment_count -- Number of transcript segments
• source -- Original URL

If extraction fails, the document will have empty page_content and an error key in metadata.

Use in RAG Pipelines

from langchain_trawl import TrawlLoader
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load transcripts
loader = TrawlLoader(
    urls=[
        "https://youtube.com/watch?v=video1",
        "https://youtube.com/watch?v=video2",
    ],
)
docs = loader.load()

# Split and embed
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split_documents(docs)
vectorstore = FAISS.from_documents(chunks, OpenAIEmbeddings())

# Query
results = vectorstore.similarity_search("your question here")

Development

# Install dependencies
poetry install

# Run tests
make test

# Lint and format
make lint
make format

License

MIT