gallerycrawler 0.1.0

Generic crawling for web galleries

https://github.com/idlesign/gallerycrawler

Description

Generic crawling for galleries

1. Crawler starts from gallery listing URL;

2. It visits every details page mentioned on current listing page;

3. It gathers information from each details page;

4. It moves to the next listing URL.

5. Etc.

from galerycrawler.toolbox import Crawler, dump

# Define crawler.
class MyCrawler(Crawler):

    selector_listing_next: str = '.page-next a'
    selector_listing_thumbnails: str = '.thumbnail img'
    selector_details: str = '.page-details a'
    selector_details_title: str = '.page-title'
    selector_details_img: str = '.image img'
    selector_details_author: str = '.image-author'

# Run dumping.
dump(
    crawler=MyCrawler,
    url='https://mysite.some/gallery/',
    fpath='dumped.html',
    probe=True,  # Use this to quick test your crawler
)