Fast async web crawler & scraping framework, supporting deduplication, and extensible middleware.
Project description
qcrawl is a fast async web crawling & scraping framework for Python to extract structured data from web-pages.
It is cross-platform and easy to install via pip or conda.
Follow the documentation.
qCrawl features
- Async architecture - High-performance concurrent crawling based on asyncio
- Performance optimized - Queue backend on Redis with direct delivery, messagepack serialization, connection pooling, DNS caching
- Powerful parsing - CSS/XPath selectors with lxml
- Middleware system - Customizable request/response processing
- Flexible export - Multiple output formats including JSON, CSV, XML
- Flexible queue backends - Memory, Disk, Redis-based schedulers for different scale requirements
- Item pipelines - Data transformation, validation, and processing pipeline
- Pluggable downloaders - HTTP (aiohttp), Camoufox (stealth browser) for JavaScript rendering and anti-bot evasion
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
qcrawl-0.4.0.tar.gz
(247.6 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
qcrawl-0.4.0-py3-none-any.whl
(127.0 kB
view details)
File details
Details for the file qcrawl-0.4.0.tar.gz.
File metadata
- Download URL: qcrawl-0.4.0.tar.gz
- Upload date:
- Size: 247.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.14.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a1c58d4948d3d373f7df44e59ea9c79a9ee321e78f15af4914653dd5343130f3
|
|
| MD5 |
d2153bf994857e50ce1e0a8a74ec300e
|
|
| BLAKE2b-256 |
5464066ff131a67962bfa9ace0d698ff353eef410b1e969d93bf154fd20d6c89
|
File details
Details for the file qcrawl-0.4.0-py3-none-any.whl.
File metadata
- Download URL: qcrawl-0.4.0-py3-none-any.whl
- Upload date:
- Size: 127.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.14.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
dfa1007821bbcfbd2452efe00c9e2bb7f53559a2fe322c798923106de51af3d8
|
|
| MD5 |
faf91228abf5cedbd47a8d00e0f82946
|
|
| BLAKE2b-256 |
2ecdce47ad5cfd14b274b312ea60c410e3458e272e1c4dce5c4497c648f90785
|
