perse converts HTML content into structured JSON data
Project description
Perse
Perse converts HTML to JSON using a mix of traditional html parsing and LLM based data extraction.
Features
It's core features includes:
- Identify important fields to extract from html
- Building a JSON schemas that handles nested fields
- Process html tokens and fill the JSON schema object
You can install Perse using pip:
pip install zf-perse
export PERSE_OPENAI_API_KEY="your-openai-api-key"
And run it from CLI:
perse --url https://google.com
Optimizations
It performs a few optimizations after fetching the html while preventing any accidental removal of important data.
These optimizations includes:
- Removal of styling, scripting and svg tags
- Collapsing Tags (e.g. divs) with only one child
Comparison
There are a few other libraries but none of them provide a solution for reliable data extraction from html.
HTML to JSON
html2json library is a simple html to json converter that doesn't handle nested fields, nor does it remove unnecessary tags.
When ran on exactly the same html, Perse provides a more structured and cleaner output and at least 50% less verbose output.
| HTML to JSON | Perse |
|---|---|
|
|
HTML to Markdown
Reader-LM is a language model that converts html to markdown. It doesn't provide a json output catering only to the reader mode which is not suitabel for data extraction, analysis and automations.
Usage
Process HTML content and get a Dictionary
html_content = "<html>...</html>"
json_dict = perse(html_content)
print(json_dict)
Process HTML content and get a JSON string
html_content = "<html>...</html>"
json_string = perses(html_content)
print(json_string)
Exclude specific tags from the JSON output
html_content = "<html>...</html>"
json_dict = perse(html_content, exclude_tags={"script", "style"})
print(json_dict)
Clean up the HTML content for side usage
html_content = "<html>...</html>"
clean_soup = simmer(html_content) # or use simmers for a string output
print(clean_soup.prettify())
Examples
Google's Homepage
$ perse --url https://google.com
{
"image": "/images/branding/googleg/1x/googleg_standard_color_128dp.png",
"title": "Google",
"search_form": {
"action": "/search",
"method": "GET",
"autocomplete": "off",
"query": "",
"buttons": [
{
"button_1": {
"label": "Google Search",
"value": "Google Search"
},
"button_2": {
"label": "I'm Feeling Lucky",
"value": "I'm Feeling Lucky"
}
}
]
}
}
Zeff Muks's Homepage
$ perse --url https://zeffmuks.com
{
"title": "Zeff Muks",
"description": "Antifragile Entropy Assassin \ud83e\udd77",
"og_data": {
"type": "website",
"title": "Zeff Muks",
"description": "Antifragile Entropy Assassin \ud83e\udd77",
"url": "https://zeffmuks.com/",
"image": "https://www.zeffmuks.com/images/ZeffMuks-1920.png",
"site_name": "Zeff Muks"
},
"twitter_data": {
"card": "summary_large_image",
"site": "@zeffmuks",
"title": "Zeff Muks",
"description": "Antifragile Entropy Assassin \ud83e\udd77",
"image": "https://www.zeffmuks.com/images/ZeffMuks-1920.png"
},
"user_section": {
"header": {
"profile_image_url": "/images/ZeffMuks-6912.png",
"title": "Antifragile Entropy Assassin \ud83e\udd77",
"signature": ""
},
"builds": [
{
"date": "08/30/2024",
"name": "Cursor Git",
"description": "Enhanced Git for Cursor AI Editor",
"download_link": "https://zf-static.s3.us-west-1.amazonaws.com/cursor-git-0.1.12.vsix",
"preview_image": "https://zf-static.s3.us-west-1.amazonaws.com/cursor-git-logo128.png",
"alternative_link": ""
},
{
"date": "08/18/2024",
"name": "PyZF",
"description": "Enhancements for Python",
"download_link": "https://pypi.org/project/PyZF",
"preview_image": "https://zf-static.s3.us-west-1.amazonaws.com/pyzf-logo128.png",
"alternative_link": ""
},
{
"date": "08/05/2024",
"name": "Xanthus",
"description": "X (formerly Twitter) Assistant",
"download_link": "https://pypi.org/project/zf-xanthus",
"preview_image": "https://zf-static.s3.us-west-1.amazonaws.com/xanthus-logo128.png",
"alternative_link": ""
},
{
"date": "07/24/2024",
"name": "Jenga",
"description": "Fast JSON5 Python Library",
"download_link": "https://pypi.org/project/zf-jenga",
"preview_image": "",
"alternative_link": ""
},
{
"date": "07/12/2024",
"name": "Pegasus",
"description": "Next Generation Tech Stack",
"download_link": "https://zf-static.s3.us-west-1.amazonaws.com/pegasus.zip",
"preview_image": "https://zf-static.s3.us-west-1.amazonaws.com/pegasus-logo128.png",
"alternative_link": ""
},
...
{
"date": "11/01/2023",
"name": "Z",
"description": "Next Generation Content Platform",
"download_link": "https://x.com/zeffmuks/status/1718507463321010429",
"preview_image": "https://zf-static.s3.us-west-1.amazonaws.com/z-logo128.png",
"alternative_link": "https://alpha.thez.ai/try"
}
]
}
}
License
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file zf-perse-0.2.1.tar.gz.
File metadata
- Download URL: zf-perse-0.2.1.tar.gz
- Upload date:
- Size: 11.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 406bc1735406df4fee28b5a403fbf8b9938d366e105f178b2e712d795d3a87e7 |
|
| MD5 | ef59dfae869186d8c19b9626d1e97783 |
|
| BLAKE2b-256 | 447acc5a2eb21a40649c158e618400ef6bafbc1ffac64f9582464ad735429731 |
File details
Details for the file zf_perse-0.2.1-py3-none-any.whl.
File metadata
- Download URL: zf_perse-0.2.1-py3-none-any.whl
- Upload date:
- Size: 10.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8c54ba0f25edd59e206131b1f07d9dd963ae8d69b59b0197d22ec4a957422d63 |
|
| MD5 | cdff9f90b9edce32f50f5867930ccd40 |
|
| BLAKE2b-256 | 541a7cdd8fcde2c43dcd8b0ea0ffb2261b09b64dd03496e28b6fe41dc06a808e |
