langchain-sourcey 0.1.3

Native LangChain retriever for Sourcey-generated documentation sites.

langchain-sourcey

Your docs retriever should not depend on somebody else's SaaS either.

langchain-sourcey reads a published Sourcey docs site directly.

Sourcey already ships the files a retriever needs. This package uses them:

• search-index.json for candidate discovery
• llms-full.txt for full-page hydration
• canonical page URLs for citations

If llms-full.txt is missing, it falls back to the matched page HTML.

Install

pip install -U langchain-sourcey

Point site_url at the root of a published Sourcey build:

• https://sourcey.com/docs
• https://sourcey.com/cheesestore
• https://cheesestore.github.io

Quickstart

from langchain_sourcey import SourceyRetriever

retriever = SourceyRetriever(
    site_url="https://sourcey.com/docs",
    top_k=3,
)

docs = retriever.invoke("mcp integration")

for doc in docs:
    print(doc.metadata["title"])
    print(doc.metadata["source"])
    print(doc.page_content[:160])
    print()

For a runnable script, see examples/live_quickstart.py.

Sourcey guide: https://sourcey.com/docs/guides/guide-langchain-retriever.html

Use In A LangChain Chain

Install a chat model integration of your choice. This example uses OpenAI:

pip install -U langchain-openai

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI

from langchain_sourcey import SourceyRetriever

retriever = SourceyRetriever(site_url="https://sourcey.com/docs", top_k=3)

prompt = ChatPromptTemplate.from_template(
    """Answer the question using the documentation context below.

{context}

Question: {question}"""
)

chain = (
    RunnablePassthrough.assign(context=(lambda x: x["question"]) | retriever)
    | prompt
    | ChatOpenAI(model="gpt-4.1-mini")
    | StrOutputParser()
)

answer = chain.invoke({"question": "How does Sourcey document MCP servers?"})
print(answer)

For a fuller example, see examples/rag_chain.py.

Sourcey Contract

This package assumes the published Sourcey site exposes:

• publish search-index.json
• publish llms-full.txt
• set siteUrl in sourcey.config.ts so citations are canonical

search-index.json is required. llms-full.txt is strongly recommended because it gives the retriever full page content instead of HTML-derived fallback text.

Returned Metadata

Each returned Document includes:

• source: canonical page URL used for citations
• matched_url: original matched URL, including anchors when relevant
• matched_title: matched search entry title
• title: hydrated page title
• path: Sourcey output path such as guides/search.html
• anchor: matched fragment, if any
• tab: Sourcey tab label
• category: Sourcey search category
• site_url: docs root used for retrieval
• score: retriever ranking score

Development

python -m pip install -e .[dev] build twine
PYTHONPATH=src pytest -q
SOURCEY_TEST_SITE_URL=https://sourcey.com/docs PYTHONPATH=src pytest tests/integration_tests/test_live_retriever.py -q
python -m build
python -m twine check dist/*

See CONTRIBUTING.md for the release and verification flow.

LangChain Submission Assets

This repo includes draft docs ready to turn into a LangChain docs PR:

• docs/langchain/provider-sourcey.mdx
• docs/langchain/retriever-sourcey.mdx

Scope

This package ships SourceyRetriever only. No loader yet.