qcrawl is a fast async web crawling & scraping framework for Python to extract structured data from web-pages.
It is cross-platform and easy to install via pip or conda.
Follow the documentation.
qCrawl features
- Async architecture - High-performance concurrent crawling based on asyncio
- Performance optimized - Queue backend on Redis with direct delivery, messagepack serialization, connection pooling, DNS caching
- Powerful parsing - CSS/XPath selectors with lxml
- Middleware system - Customizable request/response processing
- Flexible export - Multiple output formats including JSON, CSV, XML
- Flexible queue backends - Memory, Disk, Redis-based schedulers for different scale requirements
- Item pipelines - Data transformation, validation, and processing pipeline
- Pluggable downloaders - HTTP (aiohttp), Camoufox (stealth browser) for JavaScript rendering and anti-bot evasion
1 maintainer
Metadata
1 maintainer
Release files for qcrawl 0.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| qcrawl-0.4.0.tar.gz | 247.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| qcrawl-0.4.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 374.6 kB
Release files / qcrawl-0.4.0.tar.gz
| Download URL | qcrawl-0.4.0.tar.gz |
|---|---|
| Size | 247.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a1c58d4948d3d373f7df44e59ea9c79a9ee321e78f15af4914653dd5343130f3
|
|
BLAKE2b-256 checksum How to use checksums |
5464066ff131a67962bfa9ace0d698ff353eef410b1e969d93bf154fd20d6c89
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.5
|
Release files / qcrawl-0.4.0-py3-none-any.whl
| Download URL | qcrawl-0.4.0-py3-none-any.whl |
|---|---|
| Size | 127.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
dfa1007821bbcfbd2452efe00c9e2bb7f53559a2fe322c798923106de51af3d8
|
|
BLAKE2b-256 checksum How to use checksums |
2ecdce47ad5cfd14b274b312ea60c410e3458e272e1c4dce5c4497c648f90785
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.5
|
