Skip to main content

Web Scraping Framework

Project description

## IOWeb Framework

Python framework to build web crawlers.

Good things:

  • system designed to run large number of network threads (like 100 or 500) on

    single CPU core

  • feature to combine things in chunks and then doing something with

    chunks (like mongodb bulk write)

  • asynchronous network operations are powered by gevent

  • network requests are handled with urllib3

  • HTML is parsed with lxml

  • ability to do CSS/XPATh queries to DOM tree of downloaded HTML document

  • ability to extract cert details

  • ability to resolve particular domain to custom IP

  • stat module to count events

  • logging statistics to influxdb

  • retrying on network errors

Bad things:

  • not fully covered with tests

  • no documentation

## Feedback

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ioweb-0.0.15.tar.gz (22.5 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page