🌾 Harvest v0.8.0
🇷🇺 Русский
Бесплатный open-source веб-скрейпер с обходом Cloudflare и LLM-экстракцией. Аналог Firecrawl и Crawl4AI, но без подписок и API-ключей. Работает на твоём компьютере, 100% локально.
Быстрый старт
pip install harvest-agent
# Скейпинг — одна команда
harvest scrape https://example.com
# LLM-экстракция — описывай что нужно
harvest llm-extract https://shop.com \
--prompt "Найди все цены и названия товаров"
# Обнаружение скрытых API
harvest detect-api https://shop.com
# P2P кэш — автоматически
harvest p2p-stats
# Проверка установки
harvest doctor
Ключевые возможности
| Возможность | Описание |
|---|---|
| 🧠 Semantic Cache | Экономия 50–70% токенов LLM. Кэш по смыслу, а не по тексту |
| 🔧 Self-Healing Parsers | Авто-починка CSS-селекторов через LLM |
| 🤖 Script Generator | Один раз анализируешь → скрейпишь без LLM (0 токенов) |
| 📊 Structural Diff | git diff для веб-страниц |
| 🔍 API Detector | Обнаружение скрытых REST/GraphQL API из браузерного трафика |
| 🌐 P2P Cache Network | Децентрализованный обмен кэшем между пользователями |
| 🔌 MCP Server | Интеграция с Hermes, Claude, Cursor |
| 🛡️ Cloudflare bypass | Обход JS-челленджей и Interstitial-страниц |
| ✨ 4 режима предобработки | full / economy / hybrid / auto |
Все команды
| Команда | Описание |
|---|---|
harvest scrape <url> |
Контент страницы (Markdown/text/HTML) |
harvest extract <url> --schema |
CSS-экстракция |
harvest llm-extract <url> --prompt |
AI-экстракция (описание → JSON) |
harvest detect-api <url> |
Обнаружение скрытых API |
harvest generate <url> |
Генерация скрипта (0 токенов) |
harvest crawl <url> |
Обход всего сайта |
harvest contacts <url> |
Сбор email/телефонов |
harvest monitor <url> |
Мониторинг изменений |
harvest snapshot <url> |
Снимок DOM-структуры |
harvest diff <url> v1 v2 |
Сравнение снимков |
harvest map <url> |
Обнаружение всех URL |
harvest batch <file> |
Пакетная обработка |
harvest cache-stats |
Статистика кэша |
harvest p2p-stats |
Статистика P2P сети |
harvest p2p-peers |
Список подключённых peers |
harvest doctor |
Проверка здоровья |
Сравнение с аналогами
| Возможность | Harvest | Crawl4AI (72k★) | Firecrawl (46k★) |
|---|---|---|---|
| Semantic Cache | ✅ 50-70% экономия | ❌ | ❌ |
| Self-Healing Parsers | ✅ | ❌ | ❌ |
| Script Generator | ✅ 0 токенов | ❌ | ❌ |
| API Detector | ✅ | ❌ | ❌ |
| P2P Cache Network | ✅ | ❌ | ❌ |
| MCP Server | ✅ | ❌ | ❌ |
| Cloudflare bypass | ✅ | ⚠️ | ✅ |
| Цена | Бесплатно | Бесплатно | $50/мес |
Донаты (крипто)
| Валюта | Адрес |
|---|---|
| BTC | bc1qd8sa7e4f696wmcyszuxh9snqt2n66zrhz9g80j |
| ETH | 0xD26f0efE6A8F11e127c3Af3D6163BD458a1693c3 |
| USDT (TON) | UQAoI2i8P9-JeZhvGSUwKnymVyY5cb-1Rg7pdqoWMNena7DP |
| SOL | 99EtqBVTeF5UNp9a1oPi18iVXbXptTG7YQ6JeJvXMUJK |
🇬🇧 English
Free, open-source AI web scraper with Cloudflare bypass & LLM extraction. An alternative to Firecrawl and Crawl4AI — no subscriptions, no API keys, 100% local.
Quick Start
pip install harvest-agent
# Scrape — one command
harvest scrape https://example.com
# LLM extraction — describe what you need
harvest llm-extract https://shop.com \
--prompt "Find all product names and prices"
# Detect hidden APIs
harvest detect-api https://shop.com
# P2P cache — automatic
harvest p2p-stats
# Health check
harvest doctor
Key Features
| Feature | Description |
|---|---|
| 🧠 Semantic Cache | 50–70% LLM token savings. Cache by meaning, not text |
| 🔧 Self-Healing Parsers | Auto-fix broken CSS selectors via LLM |
| 🤖 Script Generator | Analyze once → scrape forever (0 tokens) |
| 📊 Structural Diff | git diff for web pages |
| 🔍 API Detector | Discover hidden REST/GraphQL APIs from browser traffic |
| 🌐 P2P Cache Network | Decentralized cache sharing between users |
| 🔌 MCP Server | Works with Hermes, Claude, Cursor |
| 🛡️ Cloudflare bypass | JS challenges + Interstitial pages |
| ✨ 4 preprocessing modes | full / economy / hybrid / auto |
All Commands
| Command | Description |
|---|---|
harvest scrape <url> |
Page content (Markdown/text/HTML) |
harvest extract <url> --schema |
CSS extraction |
harvest llm-extract <url> --prompt |
AI extraction (describe → JSON) |
harvest detect-api <url> |
Discover hidden APIs |
harvest generate <url> |
Generate scraper script (0 tokens) |
harvest crawl <url> |
Crawl entire site |
harvest contacts <url> |
Collect emails/phones |
harvest monitor <url> |
Track page changes |
harvest snapshot <url> |
Capture DOM structure |
harvest diff <url> v1 v2 |
Compare snapshots |
harvest map <url> |
Discover all URLs |
harvest batch <file> |
Batch processing |
harvest cache-stats |
Cache statistics |
harvest p2p-stats |
P2P network stats |
harvest p2p-peers |
List connected peers |
harvest doctor |
Health check |
Benchmark
| Feature | Harvest | Crawl4AI (72k★) | Firecrawl (46k★) |
|---|---|---|---|
| Semantic Cache | ✅ 50-70% savings | ❌ | ❌ |
| Self-Healing Parsers | ✅ | ❌ | ❌ |
| Script Generator | ✅ 0 tokens | ❌ | ❌ |
| API Detector | ✅ | ❌ | ❌ |
| P2P Cache Network | ✅ | ❌ | ❌ |
| MCP Server | ✅ | ❌ | ❌ |
| Cloudflare bypass | ✅ | ⚠️ | ✅ |
| Price | Free | Free | $50/mo |
Donations (crypto only)
| Currency | Address |
|---|---|
| BTC | bc1qd8sa7e4f696wmcyszuxh9snqt2n66zrhz9g80j |
| ETH | 0xD26f0efE6A8F11e127c3Af3D6163BD458a1693c3 |
| USDT (TON) | UQAoI2i8P9-JeZhvGSUwKnymVyY5cb-1Rg7pdqoWMNena7DP |
| SOL | 99EtqBVTeF5UNp9a1oPi18iVXbXptTG7YQ6JeJvXMUJK |
License
MIT
1 maintainer
Metadata
1 maintainer
Release files for harvest-agent 0.8.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| harvest_agent-0.8.0.tar.gz | 111.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| harvest_agent-0.8.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 225.9 kB
Release files / harvest_agent-0.8.0.tar.gz
| Download URL | harvest_agent-0.8.0.tar.gz |
|---|---|
| Size | 111.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ad80eda0c4303508fd775bbeb73bbef2e400e14c0db645a5efd9e43f3f8d7fd0
|
|
BLAKE2b-256 checksum How to use checksums |
2aa7ad840df49026ba64e1a454904092bc0124b9073f6a250a2eac1029ba4899
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / harvest_agent-0.8.0-py3-none-any.whl
| Download URL | harvest_agent-0.8.0-py3-none-any.whl |
|---|---|
| Size | 114.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bec23537fe8ec4784e9fe040a92d157651c6ae7c6e50a1bf898d5d2063e8b409
|
|
BLAKE2b-256 checksum How to use checksums |
e586dcc47193045c0a6cfd318902615cda1f6892630215b96742cbf4c2cb462f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
