pydantic-ai-crawling
Seamlessly integrate Pydantic AI with Crawl4AI to empower your AI agents with advanced web crawling and scraping capabilities.
Features
- 🔎 Crawling & Scraping: High-performance web content extraction tailored for AI agents.
- 🖼️ Media Support: Extract images, audio, videos, and responsive formats (srcset, picture).
- 🚀 Dynamic Crawling: Execute JavaScript and handle async/sync content extraction.
- 📸 Screenshots: Capture page screenshots for debugging or visual analysis.
- 📂 Raw Data Crawling: Process raw HTML (
raw:) or local files (file://) directly. - 🔗 Link Extraction: Comprehensive extraction of internal, external, and iframe links.
- 🛠️ Customizable Hooks: Define hooks at every step to customize crawling behavior.
- 💾 Caching: Built-in caching for improved speed and efficiency.
- 📄 Metadata Extraction: Retrieve structured metadata from any web page.
- 📡 IFrame Support: Seamless extraction from embedded iframe content.
- 🕵️ Lazy Load Handling: Automatically waits for images and content to load.
- 🔄 Full-Page Scanning: Simulates scrolling for infinite-scroll and dynamic pages.
Installation
pip install pydantic-ai-crawling
Usage
import pydantic_ai_crawling
# Example usage
pydantic_ai_crawling.greet()
CLI
After installation, you can use the built-in CLI:
crawler
Development
To set up the development environment:
uv sync
To run tests:
uv run pytest
License
MIT
1 maintainer
Metadata
1 maintainer
Release files for pydantic-ai-crawlers 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pydantic_ai_crawlers-0.1.1.tar.gz | 346.4 kB | Without provenance | Details for pydantic_ai_crawlers-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pydantic_ai_crawlers-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for pydantic_ai_crawlers-0.1.1-py3-none-any.whl |
Total release size: 376.4 kB
Release files / pydantic_ai_crawlers-0.1.1.tar.gz
| Download URL | pydantic_ai_crawlers-0.1.1.tar.gz |
|---|---|
| Size | 346.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
10895ce4d0ebba51ecf277bb980cf1177d99b26510984f362cfd9b296709eef0
|
|
BLAKE2b-256 checksum How to use checksums |
7275df69bf79ee932e026dbd1da0c13de5564f22e6eb353af8f95b5a31a6a98f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.17 {"installer":{"name":"uv","version":"0.11.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / pydantic_ai_crawlers-0.1.1-py3-none-any.whl
| Download URL | pydantic_ai_crawlers-0.1.1-py3-none-any.whl |
|---|---|
| Size | 30.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c8cf4320da9e7c37538821875d2a734b698ac52f53f35211dab1748c9546f8ed
|
|
BLAKE2b-256 checksum How to use checksums |
276664331d05f405d31c34f5e29cf152a349e6bf78ec4ebb47999184eafb95c6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.17 {"installer":{"name":"uv","version":"0.11.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for pydantic-ai-crawlers 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
