langchain-wtmapi

LangChain Document Loader for WTM API — convert any webpage to clean, structured Markdown.

Installation

pip install langchain-wtmapi

Quick Start

from langchain_wtmapi import WTMApiLoader

# Get your free API key at https://wtmapi.com
loader = WTMApiLoader(
    urls=[
        "https://en.wikipedia.org/wiki/Artificial_intelligence",
        "https://developer.mozilla.org/en-US/docs/Web/JavaScript",
    ],
    api_key="wtm_your_api_key",
)

# Load all documents
docs = loader.load()

for doc in docs:
    print(f"Source: {doc.metadata['source']}")
    print(f"Length: {doc.metadata['length']} chars")
    print(f"Response time: {doc.metadata['response_time_ms']}ms")
    print(doc.page_content[:500])
    print("---")

Lazy Loading

For large batches, use lazy loading to process documents one at a time:

for doc in loader.lazy_load():
    # Process each document as it's loaded
    process_document(doc)

Use with RAG Pipeline

from langchain_wtmapi import WTMApiLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS

# Load web pages as Markdown
loader = WTMApiLoader(
    urls=["https://docs.python.org/3/tutorial/index.html"],
    api_key="wtm_your_api_key",
)
docs = loader.load()

# Split into chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter.split_documents(docs)

# Create vector store
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(chunks, embeddings)

# Query
results = vectorstore.similarity_search("How do I use lists in Python?")

Options

Document Metadata

Each loaded document includes metadata:

{
    "source": "https://example.com",        # Original URL
    "length": 15234,                         # Markdown length in chars
    "response_time_ms": 523,                 # API response time
    "usage_used": 5,                         # Monthly calls used
    "usage_limit": 50,                       # Monthly call limit
    "plan": "free",                          # Current plan
}

Get Your API Key

Sign up at wtmapi.com — free tier includes 50 calls/month, no credit card required.

License

MIT