langchain-mesharc

The LangChain integration for MeshArc, the web crawler that turns sites into clean markdown and records what changed between crawls.

• MeshArcLoader loads web pages as LangChain Documents, in four modes: scrape, crawl, map and changes.
• MeshArcSearch and MeshArcScrape are tools an agent can call to search the web and read pages.

pip install langchain-mesharc

You need a MeshArc API key. Make one under Settings → API keys at mesharc.dev; the free plan includes 1,000 credits with no card. Set it as MESHARC_API_KEY, or pass api_key=. Reading pages and searching need a key with write access.

Document loader

from langchain_mesharc import MeshArcLoader

# One page, or many read as one batch
docs = MeshArcLoader("https://example.com/pricing").load()
docs = MeshArcLoader(["https://example.com/", "https://example.com/about"]).load()

# A whole site, or a section of it; pages arrive as they are read
loader = MeshArcLoader("https://docs.example.com", mode="crawl",
                       params={"limit": 100, "includePaths": ["/guides/*"]})
for doc in loader.lazy_load():
    print(doc.metadata["url"], len(doc.page_content))

# The URLs a site declares in its sitemaps, without fetching them
urls = MeshArcLoader("https://docs.example.com", mode="map").load()

# What changed on a watched site since the last crawl
changed = MeshArcLoader(mode="changes", project_id="prj_...").load()

Metadata on every page: source and url, title, page_status (one word: ok, missing, blocked, captcha, …), crawled_at and mode.

• Scraped and changed pages also have status, the HTTP status. A crawl's pages don't, because the API leaves it out of crawl results.
• Scraped pages also have verdict and credits.
• A crawl adds crawl_id.
• Changes add project_id, run_id, previous_run_id, changed_at, change (added, modified, removed or fields), words_before and words_after.

Values are always strings, numbers or booleans, so any vector store accepts them.

Pages that came back with no content (blocked, or a 404) are left out. Pass include_empty=True to keep them; their page_status and error_code say why. In changes mode, removed pages are left out unless include_removed=True.

changes mode keeps a vector store current: load it after each run of a MeshArc project, and re-embed only the pages that changed. Each Document's id is run_id:url.

Tools

from langchain_mesharc import MeshArcScrape, MeshArcSearch

tools = [MeshArcSearch(), MeshArcScrape()]

Both tools return content_and_artifact:

• the content is the text sent to the model, with each page cut at max_chars_per_page (8,000 by default; 0 for no limit);
• the artifact is MeshArc's full response.

A refused request, such as one with too few credits left, comes back to the agent as an error message rather than stopping it.

Cost

Requests are billed in MeshArc credits, the same as any other API call:

• a plain page read costs 1 credit, and blocked pages and 404s cost nothing;
• a web search costs what reading its results page costs;
• a sitemap read costs 1 credit per sitemap file.

See pricing.

Links

• MeshArc docs and API reference
• The Python SDK, which this package uses
• Questions · security reports to security@mesharc.dev

License

MIT