Skip to main content

my simple crawler

Project description


pip3 install simple-crawler

Set environment AUTO_CHARSET=1 to pass bytes to beautifulsoup4 and let it detect the charset.


  • URL: define a URL
  • URLExt: class to handle URL
  • Page: define a request result of a URL
    • url: type URL
    • content, text, json: response content properties from library requests
    • type: the response body type, is a enum which allows BYTES, TEXT, HTML, JSON
    • is_html: check whether is html accorrding to the response headers's Content-Type
    • soup: BeautifulSoup contains html if is_html
  • Crawler: schedule the crawler by calling handler_page() recusively


from simple_crawler import *

class MyCrawler(Crawler):
    name = 'output.txt'
    aysnc def custom_handle_page(self, page):
        tags ="#container")
        tag = tags and tags[0]
        with open(, 'a') as f:
        # do some async call

    def filter_url(self, url: URL) -> bool:
        return url.url.startswith("")

loop = get_event_loop(True)
c = MyCrawler("", loop, concurrency=10)
schedule_future_in_loop(c.start(), loop=loop)


  • Speed up using async or threading

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release. See tutorial on generating distribution archives.

Built Distribution

simple_crawler-1.2-py3-none-any.whl (4.0 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page