墨探 (omni-article-markdown)
轻松将网页文章（博客、新闻、文档等）转换为 Markdown 格式。
简介
墨探的开发初衷，是为了解决一个问题：如何将来自互联网上各种不同网站的文章内容，精准且高效地转换成统一的Markdown格式。
众所周知，万维网上的网站设计风格迥异，其HTML结构也呈现出千差万别的特点。这种多样性给自动化内容提取和格式转换带来了巨大的困难。要实现一个能够适应各种复杂HTML结构的通用解决方案，并非易事。
我的想法是：从特定的网站开始适配，以点到面，逐步抽取出通用的解决方案，最后尽可能多的覆盖更多网站。
功能介绍
- 支持大部分 html 元素转换
- 部分页面支持katex公式转换（示例：https://quantum.country/qcvc）
- 部分页面支持github gist（示例：https://towardsdatascience.com/hands-on-multi-agent-llm-restaurant-simulation-with-python-and-openai）
- 支持保存成文件或输出至
stdout - 支持突破某些网站的防爬虫策略（通过
playwright）
以下是一些网站示例，大家可以自己测试下效果。
| 站点 | 链接 | 备注 |
|---|---|---|
| Medium | link | |
| csdn | link | |
| 掘金 | link | |
| 公众号 | link | |
| 网易 | link | |
| 简书 | link | |
| Towards Data Science | link | |
| Quantamagazine | link | |
| Cloudflare博客 | link | |
| 阿里云开发者社区 | link | |
| 微软技术文档 | link | |
| InfoQ | link | |
| 博客园 | link | |
| 思否 | link | |
| 开源中国 | link | |
| Forbes | link | |
| 少数派 | link | |
| 语雀 | link | |
| 腾讯云开发者社区 | link | |
| 人人都是产品经理 | link | |
| Jetbrains博客 | link | |
| Claude文档 | link | |
| Anthropic | link | |
| Meta博客 | link | |
| Android Developers Blog | link | |
| Spring Blog | link | |
| Hackernoon | link | |
| 领英博客 | link | |
| 华尔街见闻 | link | |
| 苹果开发者文档 | link | |
| 百家号 | link | |
| Snowflake 技术博客 | link | |
| 知乎专栏 | link | |
| 今日头条 | link | |
| X Articles | link | |
| 飞书 | link | |
| Google for Developers | link | |
| Dropbox.Tech | link | |
| Wikipedia | link | |
| 虎嗅网 | link | |
| link | 已失效 |
安装与升级
# pipx
pipx install omni-article-markdown
pipx upgrade omni-article-markdown
# pip
pip install omni-article-markdown
pip install -U omni-article-markdown
# uv
uv tool install omni-article-markdown
uv tool install omni-article-markdown --upgrade
安装完成后即可使用：
mdcli -h
基本用法
仅转换
mdcli https://example.com
保存到当前目录
mdcli https://example.com -s
保存到指定路径
mdcli https://example.com -s /home/user/
架构说明
墨探主要分为三个模块：
- Reader 模块的功能是读取整个网页内容
- Extractor 模块的功能是提取正文内容，清理无用数据
- Parser 模块的功能是将 HTML 转换为 Markdown
贡献与反馈
- 发现解析问题？欢迎提交 Issue
- 改进解析？欢迎贡献 Pull Request
赞助
如果你觉得墨探对你有帮助，可以给我家猫咪买点罐头 ❤️
License
MIT License
1 maintainer
Metadata
1 maintainer
Release files for omni-article-markdown 0.2.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| omni_article_markdown-0.2.2.tar.gz | 43.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| omni_article_markdown-0.2.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 110.2 kB
Release files / omni_article_markdown-0.2.2.tar.gz
| Download URL | omni_article_markdown-0.2.2.tar.gz |
|---|---|
| Size | 43.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d5be98778a974fc0de5047d384774e1e95b7c12f946568c12d7ca6855b8a79f3
|
|
BLAKE2b-256 checksum How to use checksums |
dc134dcf2406c25653ef387ed266c71a15af631106f8a9908f07b110a0fd99b1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / omni_article_markdown-0.2.2-py3-none-any.whl
| Download URL | omni_article_markdown-0.2.2-py3-none-any.whl |
|---|---|
| Size | 67.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
cab13f8ff19b919fbfd3ead3e5cebbdbe3f4d9110c0bc3404cbee7cdcbd06028
|
|
BLAKE2b-256 checksum How to use checksums |
5c617eeece9a231994d7b7c5738084cdd0a101d92aad5568358128fd8cd15227
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
