Skip to main content

langchain-adscrawl

A small, standalone LangChain tool for reading rendered public webpages through AdsCrawl's /html API.

Maintained by AdsCrawl. This is not a LangChain-maintained package, an endorsement, or a claim of inclusion in LangChain's official integration directory.

What the tool does

AdsCrawlReadTool implements LangChain's BaseTool interface with both invoke() and native-async ainvoke(). It sends one request to https://api.adscrawl.net/html and returns the response in one of three modes:

content_mode Returned value Intended use
html (default) Rendered HTML string General page content, including non-article pages
markdown Readability Markdown string Readable article content
json Readability article dictionary Article title, byline, excerpt and body, when available

Markdown and JSON are article extraction modes, not a promise to convert any entire site into Markdown or arbitrary structured data. They can fail when Readability does not find an article. An optional selector maps to the API's contentSelector field; with article modes, Readability runs against that element.

This package does not expose screenshots, forms, cookies, proxy credentials, CDP, MCP sessions or persistent browser profiles. Those are different AdsCrawl capabilities, not features of this tool.

Install

Python 3.11 or later is required (matching langchain-core>=1.8.0). Use a published PyPI release when one is listed:

python -m pip install langchain-adscrawl

Alternatively, install from source (requires Git):

python -m venv .venv
source .venv/bin/activate
python -m pip install "git+https://github.com/AdsCrawl/langchain-adscrawl.git@main"

For reproducible deployments, pin a registry version or replace main with a reviewed commit SHA. If no registry release is listed yet, use the source installation above.

Read a page without an LLM

Create your own AdsCrawl key following the official quickstart, then set it in your server environment:

export ADSCRAWL_API_KEY="YOUR_ADSCRAWL_API_KEY"

The value above is only a placeholder and must be replaced locally. Never commit a real key. The package does not load a .env file automatically.

from langchain_adscrawl import AdsCrawlReadTool

reader = AdsCrawlReadTool()
html = reader.invoke({
    "url": "https://example.com",
    "content_mode": "html",
})
print(html)

No OpenAI or other model key is needed for a direct tool invocation. A real AdsCrawl request may consume credits; this repository and its source installation are free, not necessarily the underlying service usage.

Async and article output

import asyncio
from langchain_adscrawl import AdsCrawlReadTool

async def main():
    reader = AdsCrawlReadTool(timeout=120_000, wait_until="domcontentloaded")
    article = await reader.ainvoke({
        "url": "https://example.com/article",
        "content_mode": "json",
        # "selector": "article",  # Only use a selector present on your target page.
    })
    print(article.get("title"))

asyncio.run(main())

https://example.com/article is an illustrative URL, not a claim that an article exists there. Use a public article you are authorized to access. timeout is in milliseconds (1–3,600,000); wait_until is domcontentloaded, load, or networkidle. A later navigation event can help deferred page content, but can also take longer or time out.

You can also pass api_key as a SecretStr constructor value obtained from your secret manager. The model-facing input schema contains only url, content_mode and selector, never the key or transport configuration.

Use in a LangChain agent

Pass the tool instance in the tools list of LangChain's create_agent just as you would another BaseTool. Configure your model separately; this package deliberately does not depend on a model provider or choose a model for you.

Before giving any agent the tool, enforce your own URL/domain allowlist and usage budget. The tool validates HTTP(S) URL syntax, rejects embedded credentials and non-80/443 ports, but it is not a complete SSRF filter, authorization layer or prompt-injection defense. Page content is untrusted data.

Errors, clients and usage limits

  • Non-success HTTP responses and transport failures raise a sanitized LangChain ToolException. Raw error bodies, headers and transport exception chains are not forwarded to the model.
  • The default implementation does not retry requests and explicitly refuses HTTP redirects at the API endpoint. Decide whether another attempt is appropriate before making a second potentially billable request.
  • A local timeout or cancellation does not establish whether a remote task completed. This is not a durable task manager.
  • The tool creates and closes a local HTTP client for each call by default. You may inject client=httpx.Client(...) and/or async_client=httpx.AsyncClient(...) for pooling or tests; their lifecycle belongs to you. The configured timeout and redirect refusal still apply to the request.
  • Authentication, credits, concurrency limits, site restrictions and Readability failures can prevent a result. No bypass or success guarantee is made.

The target URL and configuration are sent to AdsCrawl. If you feed the result to a model, that content is also sent to your selected model provider. Your application's callbacks or LangSmith tracing may record URLs and tool outputs; review those settings before using sensitive pages. This library does not configure tracing for your application.

Development and validation

git clone https://github.com/AdsCrawl/langchain-adscrawl.git
cd langchain-adscrawl
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[test,lint]" build
python -m pytest tests/unit_tests
python -m ruff check .
python -m build
python -m pip check

Tests use placeholder credentials and mocked transports. They include the applicable LangChain standard unit suite and request-contract, async, validation and error-safety checks. They do not invoke the real AdsCrawl service or a model, and are not production end-to-end verification. See CONTRIBUTING.md for the maintenance and release checklist.

The architecture follows LangChain's standalone integration implementation guide and standard test guide. Official directory listing is a separate process that requires a published package; see LangChain's current publishing rules.

License

MIT. See LICENSE.

Metadata

Release files for langchain-adscrawl 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for langchain-adscrawl 0.1.0
File Size Uploaded Provenance Actions
langchain_adscrawl-0.1.0.tar.gz 13.2 kB With provenance Details for langchain_adscrawl-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for langchain-adscrawl 0.1.0
File Interpreter ABI Platform Actions
langchain_adscrawl-0.1.0-py3-none-any.whl Python 3 none any Details for langchain_adscrawl-0.1.0-py3-none-any.whl

Total release size: 21.2 kB

Release files / langchain_adscrawl-0.1.0.tar.gz

Download URL langchain_adscrawl-0.1.0.tar.gz
Size 13.2 kB
Tags Source
SHA-256 checksum
How to use checksums
1831e623d0218f15d9dfd967fcf2d2d43fa1edb39fc1702505eb9e2d4f1e3d59
BLAKE2b-256 checksum
How to use checksums
4aa737b32a78d89838b5740c52123c01d04f192f1c7b8886c309b14ec16a0190
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / langchain_adscrawl-0.1.0-py3-none-any.whl

Download URL langchain_adscrawl-0.1.0-py3-none-any.whl
Size 8.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
862889001add6c2d6e767a623318ac24a5adf1d6e08fe68514c94cff73ec9e18
BLAKE2b-256 checksum
How to use checksums
d142dbdbf7453965723899098fe46556a72e4f635a57d41c7595d90500746d0c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Security for langchain-adscrawl 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for langchain_adscrawl-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page