site2markdown
A robust Python utility to convert web pages into clean, structured Markdown. Optimized for generating high-quality context for Large Language Models (LLMs).
Installation
pip install site2markdown
Usage
Quick Start
from site2markdown import UrlToMarkdown
converter = UrlToMarkdown()
# Convert URL to Markdown
markdown = converter.convert(
url="https://www.example.com",
inline_title=True, # Include the page title as an H1
ignore_links=False, # Preserve hyper-links
improve_readability=True # Use readability algorithm to filter boilerplate
)
print(markdown)
Converting HTML Content
from site2markdown import UrlToMarkdown
converter = UrlToMarkdown()
html_content = "<html><body><h1>Example</h1><p>Text content.</p></body></html>"
markdown = converter.convert_html(
html=html_content,
url="https://www.example.com", # Optional: base URL for resolving relative paths
inline_title=True,
improve_readability=True
)
print(markdown)
Core Features
- LLM Optimized: Produces clean text with minimal noise, ideal for RAG and LLM context.
- Smart Filtering: Automatic removal of scripts, styles, and non-content elements.
- Specialized Handling: Custom parsing logic for sites like Stack Overflow, Wikipedia, and Medium.
- Readability Engine: Integration with Mozilla's Readability algorithm to extract the main article body and skip navigation/sidebars.
- Structure Preservation: Reliable conversion of code blocks, tables, and nested lists.
- URL Resolution: Automatically converts relative links and image paths to absolute URLs.
Configuration Options
| Option | Type | Description | Default |
|---|---|---|---|
inline_title |
bool |
Includes the page title as an H1 heading at the top. | True |
ignore_links |
bool |
Removes all hyperlinks from the output. | False |
improve_readability |
bool |
Processes the page through a readability filter before conversion. | True |
License
This project is licensed under the MIT License.
Metadata
Release files for site2markdown 1.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| site2markdown-1.0.2.tar.gz | 10.2 kB | Without provenance | Details for site2markdown-1.0.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| site2markdown-1.0.2-py3-none-any.whl | Python 3 | none | any | Details for site2markdown-1.0.2-py3-none-any.whl |
Total release size: 21.3 kB
Release files / site2markdown-1.0.2.tar.gz
| Download URL | site2markdown-1.0.2.tar.gz |
|---|---|
| Size | 10.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
58e128bc8fd48428e830330ae776d7e7e29660369ed0740241264f3dd3198653
|
|
BLAKE2b-256 checksum How to use checksums |
f1c988e78ffd405fe9000f04ad7502f0ee7d0da707b0904ac183463365f6da45
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.0
|
Release files / site2markdown-1.0.2-py3-none-any.whl
| Download URL | site2markdown-1.0.2-py3-none-any.whl |
|---|---|
| Size | 11.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c30729a8dadf94096f69ec41ab0bdeb41350134854ae177487cffa7c6b853314
|
|
BLAKE2b-256 checksum How to use checksums |
2fce4efa64595608b6f905c9f668bee6b42269e0efa42f9d18a6418d928437e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.0
|
Security for site2markdown 1.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.