icrawler
Introduction
Documentation: http://icrawler.readthedocs.io/
This package is a mini framework of web crawlers. With modularization design, it can be extended conveniently. It supports media data like images and videos very well, and can also be applied to texts and other type of files. Scrapy is heavy and powerful, while icrawler is tiny and flexible.
With this package, you can write a multiple thread crawler easily by focusing on the contents you want to crawl, avoiding some troublesome problems like exception handling, thread scheduling and communication.
It also provides built-in crawlers for popular image sites like flickr and search engines such as Google, Bing and Baidu. If you want to add your own crawlers into built-in, welcome for pull requests.
Requirements
Python 2.7+ or 3.4+ (recommended).
Samples
Using builtin crawlers is very simple.
from icrawler.builtin import GoogleImageCrawler
google_crawler = GoogleImageCrawler(parser_threads=2, downloader_threads=4,
storage={'root_dir': 'your_image_dir'})
google_crawler.crawl(keyword='sunny', max_num=1000,
date_min=None, date_max=None,
min_size=(200,200), max_size=None)Writing your own crawlers with this framework is also convenient, see the tutorials.
Structure
A crawler consists of 3 main components (Feeder, Parser and Downloader), they are connected with each other with FIFO queues. The workflow is shown in the following figure.
url_queue stores the url of pages which may contain images
task_queue stores the image url as well as any meta data you like, each element in the queue is a dictionary and must contain the field img_url
Feeder puts page urls to url_queue
Parser requests and parses the page, then extracts the image urls and puts them into task_queue
Downloader gets tasks from task_queue and requests the images, then saves them in the given path.
Feeder, parser and downloader are all thread pools, so you can specify the number of threads they use.
Release files for icrawler 0.3.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| icrawler-0.3.6.tar.gz | 20.0 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| icrawler-0.3.6-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size: 50.9 kB
Release files / icrawler-0.3.6.tar.gz
| Download URL | icrawler-0.3.6.tar.gz |
|---|---|
| Size | 20.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a3725bee15a00e0e46b0b616bb0b3d701f036a12f1d0c3d25d0b44fae8884791
|
|
BLAKE2b-256 checksum How to use checksums |
dcb308f17a744b82a2318b36359e37c27dcf704d69c419b7a089ed16fcd9afee
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / icrawler-0.3.6-py2.py3-none-any.whl
| Download URL | icrawler-0.3.6-py2.py3-none-any.whl |
|---|---|
| Size | 30.9 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
540ed7b34ceaf68ae2a3b1b4a31c54f967d03989a7388c32edb91d75c4da643f
|
|
BLAKE2b-256 checksum How to use checksums |
c50cd14f01a473e317210bb974695fe36621cc428d2f75d43d52410d8e306fca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for icrawler 0.3.6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
