crawlo 1.7.0

Crawlo: A high-performance asynchronous Python web crawling framework with distributed support.。

Crawlo

A Modern High-Performance Python Async Web Scraping Framework

Python 3.8+ · Python 3.14 Compatible

中文 · English

Quick Start · Key Features · Docs · Examples

---

✨ Quick Start (3 Steps)

1. Install

pip install crawlo

2. Create a Spider

crawlo startproject myproject
cd myproject
crawlo genspider example example.com

3. Run

crawlo run example

👉 5-Minute Quickstart Tutorial →

---

🚀 Key Features

⚡ High-Performance Async Architecture

• Built on asyncio + aiohttp/httpx/curl-cffi multi-protocol downloaders
• Smart concurrency control, connection pool reuse, auto throughput optimization
• HTTP/2 support, TLS fingerprint emulation (bypass JA3 detection)

🛡️ Robust Anti-Bot Capabilities

• HybridDownloader: 6-level detection priority, auto-switch protocol/browser engine
• Cloudflare Auto-Bypass: Detects challenge pages and auto-switches to stealth browser
• 5 Browser Downloaders: Playwright / Camoufox / CloakBrowser / DrissionPage / Chrome
• BROWSER_ Unified Config Layer*: One set of params for all browser downloaders
• Adaptive Selectors: Auto-relocate elements when site structure changes (selector self-healing) — Guide →

🤖 AI Integration (MCP Server)

• Claude / Cursor directly invoke Crawlo scraping capabilities
• Three scraping modes: basic (1-3s) → stealth (3-10s) → max-stealth (10s+)
• Browser singleton pool: stealth/max-stealth modes reuse instances
• Structured error responses: distinguish TIMEOUT / CONNECTION_ERROR / STEALTH_UNAVAILABLE, with suggestions

📊 Four-Level Backpressure Defense

• Engine layer: request generation control (enqueue + TaskManager dual checks)
• QueueManager layer: strategy-driven (QueueSizeStrategy / AdaptiveStrategy / CompositeStrategy)
• MemoryQueue layer: Mixin delegation + fallback logic
• Hard limit: direct rejection when queue is full
• Smart enhancement: IntelligentBackpressureCalculator + BackpressureMonitor optional integration

📬 Multi-Channel Notification

• 5 Channels: DingTalk / Feishu / WeCom / Email / SMS
• 30+ Preset Templates: task start/stop, anomaly alerts, progress updates, DB monitoring
• Async Delivery: async_send_* functions, run_in_executor wrapper to avoid blocking event loop
• Message dedup + rate limiting to prevent notification storms

🔄 Three Deployment Modes

Mode	Config	Coordination	Use Case
Memory Mode	RUN_MODE='standalone' QUEUE_TYPE='memory'	None (auto exit)	Dev/debug, quick validation
Multi-Node ⭐	RUN_MODE='auto' QUEUE_TYPE='redis'	Competing consumption (BZPOPMIN)	Multi-machine, task loss acceptable
Distributed	RUN_MODE='distributed' QUEUE_TYPE='redis_stream'	ACK + heartbeat + failover	Production, high reliability

All three modes share the same priority model — switch without modifying spider code. Learn More → · Production Deployment →

---

📚 Documentation

🎯 By Role

You are?	Recommended Reading
Beginner	5-Min Quickstart → Installation
Developer	Configuration Guide → Scheduling Guide
Ops	Run Mode Deep Dive → Checkpoint System → Production Deployment

📖 Full Docs Navigation

• 🚀 Getting Started - Install, create your first spider
• 📚 Tutorials - Complete guides from basics to production
• 🎯 Guides - Scenario-based deep dives
  • Configuration, Scheduling
  • Backpressure, Run Modes
  • Adaptive Selector — selector self-healing after site redesign
• 📖 Concepts - Architecture, lifecycle, error handling
  • Distributed Architecture — Redis Streams, failover, coordinated shutdown
• 🖥 Production Deployment - Linux server setup, systemd, monitoring
• 🔧 API Reference - Complete API docs
• 💡 Examples - Real-world examples and best practices
• ❓ FAQ - FAQ and troubleshooting

👉 Browse Complete Docs →

---

💡 Examples

Check out the examples/ directory:

• Basic - Quick start
• Advanced - Complex scenarios
• Production - Ready for production

👉 View All Examples →

---

🤝 Contributing

Issues and Pull Requests are welcome!

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

---

📄 License

Licensed under BSD 3-Clause - see the LICENSE file for details.

---

⭐ If this project helps you, please give us a Star!