20 projects
deprecated
Python @deprecated decorator to deprecate old python classes, functions or methods.
queuelib
Collection of persistent (disk-based) queues
splash
A javascript rendered with a HTTP API
dateparser
Date parsing library designed to parse dates from HTML pages
w3lib
Library of web-related functions
scrapy-crawlera
Crawlera middleware for Scrapy
Scrapy
A high-level Web Crawling and Web Scraping framework
webstruct
A library for creating statistical NER systems that work on HTML data
scrapinghub
Client interface for Scrapinghub API
slybot
Slybot crawler
shub
Scrapinghub Command Line Client
scrapely
A pure-python HTML screen-scraping library
scrapyd
A service for running Scrapy spiders, with an HTTP API
hubstorage
Client interface for Scrapinghub HubStorage
scrapylib
Scrapy helper functions and processors
adblockparser
Parser for Adblock Plus rules
scrapy-dotpersistence
Scrapy extension to sync `.scrapy` folder to an S3 bucket
scrapyjs
JavaScript support for Scrapy using Splash
flatson
Tool to flatten stream of JSON-like objects, configured via schema
crawl-frontier
A flexible frontier for web crawlers
