A HT/XML web scraping tool
___ __ __ __ / (_) /_ ___ _ __/ /__________ ______/ /_ / / / __ \/ _ \| |/_/ __/ ___/ __ `/ ___/ __/ / / / /_/ / __/> </ /_/ / / /_/ / /__/ /_ /_/_/_.___/\___/_/|_|\__/_/ \__,_/\___/\__/
Libextract is a statistics-enabled data extraction library that works on HTML and XML documents and written in Python. Originating from eatiht, the extraction algorithm works by making one simple assumption: data appear as collections of repetitive elements. You can read about the reasoning here.
- libextract.api.extract(document, encoding=’utf-8’, count=5)
Given an html document, and optionally the encoding, return a list of nodes likely containing data (5 by default).
pip install libextract
Due to our simple definition of “data”, we open up a single interfaceable method. Post-processing is up to you.
from requests import get from libextract.api import extract r = get('http://en.wikipedia.org/wiki/Information_extraction') textnodes = list(extract(r.content))
Using lxml’s built-in methods for post-processing:
>> print(textnodes.text_content()) Information extraction (IE) is the task of automatically extracting structured information...
The extraction algo is agnostic to article text as it is with tabular data:
height_data = get("http://en.wikipedia.org/wiki/Human_height") tabs = list(extract(height_data.content))
>> [elem.text_content() for elem in tabs.iter('th')] ['Country/Region', 'Average male height', 'Average female height', ...]
This project is still in its infancy; and advice and suggestions as to what this library could and should be would be greatly appreciated
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.