Configurable Python Web Scraper
Project description
Minimalist Python DOM Scraper
Description
This module is an easy to use HTML/XML scraper. It supports both XPath and Regular Expression retrieval.
Once you have a file you want to extract information from, you can extract multiple pieces of information with a simple function call.
You should obtain the files you want to scrape by your own ways.
Installation
pip install scraper
Usage
Scrape using xpath:
import scraper import requests content = requests.get('https://github.com/explore').content conf = {'trending-repos' : {'xpath' : '//ol/li/h3/a[2]/@href'}} scraper.scrapes(content, conf) >>> {'trending-repos': ['/jamescryer/grumble.js', '/dominictarr/JSON.sh', '/JamieLottering/DropKick', '/harvesthq/chosen', '/velvia/ScalaStorm']}
Scrape using regexp:
import scraper import requests content = requests.get('http://wiki.nomasnumeros900.com/Air_Liquide').content conf = { 'numbers': {'regexp': '91[\s\d]+', 'transf': [lambda x: x.strip()], 'encoding': 'utf-8'} } scraper.scrapes(content, conf) >>> {'numbers': [u'915 029 300', u'915 029 560', u'915 029 330', u'91']}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scraper-0.1.0.tar.gz
(2.6 kB
view hashes)