Generic crawling for web galleries
Project description
https://github.com/idlesign/gallerycrawler
Description
Generic crawling for galleries
Crawler starts from gallery listing URL;
It visits every details page mentioned on current listing page;
It gathers information from each details page;
It moves to the next listing URL.
Etc.
from galerycrawler.toolbox import Crawler, dump
# Define crawler.
class MyCrawler(Crawler):
selector_listing_next: str = '.page-next a'
selector_listing_thumbnails: str = '.thumbnail img'
selector_details: str = '.page-details a'
selector_details_title: str = '.page-title'
selector_details_img: str = '.image img'
selector_details_author: str = '.image-author'
# Run dumping.
dump(
crawler=MyCrawler,
url='https://mysite.some/gallery/',
fpath='dumped.html',
probe=True, # Use this to quick test your crawler
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
gallerycrawler-0.1.0.tar.gz
(7.0 kB
view hashes)
Built Distribution
Close
Hashes for gallerycrawler-0.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65d7067fb463e6dee25e9ec081f75c9bb1938fae7fdad1c36ae332c370f381a3 |
|
MD5 | 38d598dda5699cbce5236f1048e8b37a |
|
BLAKE2b-256 | 658c0f4bee660ac5412380c62b9dc32400732521989c6aaa0fb56865ae23335a |