web2md-client (Python)

Turn any web page into clean Markdown and heading-aware RAG chunks with the Web2MD API. The free BASIC plan allows 500 requests/month; PRO is $9/month for 10,000.

Install

pip install web2md-client
pip install "web2md-client[langchain]"    # LangChain document loader
pip install "web2md-client[llamaindex]"   # LlamaIndex reader

Requires Python 3.10+. Get a key from the RapidAPI listing above and export it as RAPIDAPI_KEY.

Quickstart

from web2md_client import Web2MD

with Web2MD() as w:  # reads RAPIDAPI_KEY
    doc = w.read("https://example.com")
    print(doc.metadata.title, doc.tokens)
    print(doc.markdown)
    for chunk in w.chunks("https://example.com", max_tokens=512).chunks:
        print(chunk.heading_path, chunk.tokens)

Other methods: convert(html, url=None) for HTML you already have, and metadata(url) for page metadata only.

Async:

import asyncio
from web2md_client import AsyncWeb2MD


async def main():
    async with AsyncWeb2MD() as w:
        doc = await w.read("https://example.com")
        print(doc.markdown)


asyncio.run(main())

RAG over any website in 10 lines

No LLM key needed; embeddings run locally with FastEmbed.

# pip install "web2md-client[langchain]" langchain-community fastembed
# export RAPIDAPI_KEY=...  (free key: https://rapidapi.com/kanavsingla28/api/web2md-webpage-to-markdown-rag-chunks)
from langchain_community.embeddings import FastEmbedEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore

from web2md_client.langchain import Web2MDLoader

docs = Web2MDLoader(["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"], max_tokens=400).load()
store = InMemoryVectorStore.from_documents(docs, FastEmbedEmbeddings())
for hit in store.similarity_search("What problem does RAG solve?", k=3):
    print(f"[{hit.metadata['heading_path'] or hit.metadata.get('title', '')}]", hit.page_content[:300], "\n")

LlamaIndex

No LLM key needed: this uses a local FastEmbed embedding and a retriever (no query engine).

# pip install "web2md-client[llamaindex]" llama-index-embeddings-fastembed
from llama_index.core import Settings, VectorStoreIndex
from llama_index.embeddings.fastembed import FastEmbedEmbedding
from web2md_client.llamaindex import Web2MDReader

Settings.embed_model = FastEmbedEmbedding(model_name="BAAI/bge-small-en-v1.5")
docs = Web2MDReader(max_tokens=400).load_data(["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"])
index = VectorStoreIndex.from_documents(docs)
for hit in index.as_retriever(similarity_top_k=3).retrieve("What problem does RAG solve?"):
    print(hit.node.metadata.get("heading_path"), hit.node.get_content()[:200])

VectorStoreIndex.from_documents may re-split our chunks with its default SentenceSplitter; that is harmless.

Loader options

Both loaders accept keyword-only include_links, include_images and include_tables (each True, False or None). They are forwarded to the API only when set; None leaves the API default. include_links=False gives cleaner embeddings.

Web2MDLoader(urls, include_links=False, include_tables=True)

A missing API key is reported when loading starts, not when the loader is constructed.

Loader metadata

In chunks mode (the default), every LangChain Document / LlamaIndex node carries:

In read mode the keys are source, title, tokens, word_count, author, published, description, language and site_name. Keys with no value are omitted.

Errors

API, gateway and network failures raise Web2MDError; invalid arguments, such as passing both url and html, raise ValueError. Web2MDError has .code, .message and .status (HTTP status, or None for client-side failures).

from web2md_client import Web2MD, Web2MDError

try:
    Web2MD().read("https://example.com")
except Web2MDError as e:
    print(e.code, e.status, e.message)

Loaders (Web2MDLoader, Web2MDReader) raise on the first failure by default. Pass continue_on_failure=True to skip failing URLs and keep loading the rest.

Licence

MIT