HTML data extraction library
Project description
Pickaxe
Pickaxe is a Python package for structured data extraction from HTML documents. It provides a simple and intuitive API for parsing HTML documents, and automatically extracting structured data from them.
Features
- Written in Rust: Pickaxe is written in Rust, which makes it fast and memory-efficient.
- Robust: Pickaxe uses the
html5everandselectorscrate for browser-grade HTML parsing and CSS selector matching. - Data Maps: Pickaxe can automatically generate CSS selectors for structured data extraction using Data Maps.
- CSS Selectors & XPath: Pickaxe supports both CSS selectors and (simple) XPath expressions for querying HTML documents.
Quick Start
Installation
pip install python-pickaxe
Basic Usage
from pickaxe import HtmlDocument
# Parse an HTML document
document = HtmlDocument.from_str("<html><body><h1>Hello, World!</h1></body></html>")
# Access elements using CSS selectors or XPath expressions
heading = document.find("h1")
print(heading.inner_text) # Output: Hello, World!
heading = document.find_xpath("//h1")
print(heading.inner_text) # Output: Hello, World!
Data Maps
Data Maps are a powerful feature of Pickaxe that allow you to automatically find the best (most concise) CSS selectors for an HTML document based on samples.
from httpx import AsyncClient
from pickaxe import Attribute, HtmlDocument, generate_data_map
# We first generate a data map using a sample HTML document, and
# examples of the data we want to extract
async with AsyncClient() as client:
response = await client.get("http://quotes.toscrape.com/author/Albert-Einstein/")
document = HtmlDocument.from_str(response.text)
# In this example we want to extract the Name and Birth date of the authors.
# The HTML documents that are used as examples must have a corresponding sample in each attribute, even if the expected value
# is None.
# Note: You can specify the amount of iterations to run the algorithm, but typically 1-3 is enough.
data_map = generate_data_map(
[document],
[
Attribute("name", ["Albert Einstein"]),
Attribute("birth_date", ["March 14, 1879"]),
]
)
# From this data map we can extract data from other HTML documents
async with AsyncClient() as client:
response = await client.get("https://quotes.toscrape.com/author/J-K-Rowling/")
document = HtmlDocument.from_str(response.text)
data = data_map.extract(document)
print(data.to_dict()) # Output: {'name': 'J.K. Rowling', 'birth_date': 'July 31, 1965'}
# You can serialize and deserialize the data map to JSON.
data_map_json = data_map.to_json()
data_map = DataMap.from_json(data_map_json)
# The result of `extract()` is a `StructuredData` object, which can be converted to a dictionary or JSON.
print(data.to_dict())
print(data.to_json())
print(StructuredData.from_json(my_json).to_dict())
License
This project is licensed under MIT License.
Support & Feedback
If you encounter any issues or have feedback, please open an issue. We'd love to hear from you!
Made with ❤️ by Emergent Methods
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file python_pickaxe-0.5.0-cp39-abi3-manylinux_2_34_x86_64.whl.
File metadata
- Download URL: python_pickaxe-0.5.0-cp39-abi3-manylinux_2_34_x86_64.whl
- Upload date:
- Size: 1.7 MB
- Tags: CPython 3.9+, manylinux: glibc 2.34+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.6.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
111ce0f142d93ff4e52570500a30cfdb103e52ecf4015f970f0e480e0bec50b8
|
|
| MD5 |
c5797be2b8a96c446152e2e9a3d91713
|
|
| BLAKE2b-256 |
56e935c383f5ca44829aee7026c1beafb0db12781d3af2b4b083f2e99add0a06
|
