melon-parse

Parse a Wikipedia page into Melon-mode: a clean, distraction-free, paginated focus-mode reader, emitted as machine-readable JSON.

This is a Python port of the encyclopedic reader from the Melon browser — the same rules that strip Wikipedia's chrome (navboxes, hatnotes, edit links, [1] markers, the "From Wikipedia, the free encyclopedia" tagline in ~50 languages), capture the infobox / data tables / image gallery / references / interlanguage links as structured data, and split the article into one coherent top-level section per page for focus-mode reading.

Install

pip install melon-parse

Use — library

import melon_parse

article = melon_parse.parse("https://en.wikipedia.org/wiki/Watermelon")
# or a bare title:  melon_parse.parse("Watermelon", lang="en")

print(article.title, "—", len(article.sections), "focus-mode pages")
for s in article.sections:
    print(f"[{s.index}] {s.title}  ({len(s.blocks)} blocks)")

data = article.to_dict()          # fully JSON-serializable

Use — CLI

melon-parse https://en.wikipedia.org/wiki/Watermelon        # JSON to stdout
melon-parse Watermelon --lang en -o watermelon.json         # bare title
melon-parse Watermelon --sections-only --indent 0           # just the pages, compact
melon-parse --html saved.html --url https://en.wikipedia.org/wiki/Watermelon   # offline

Output schema (machine-readable)

{
  "url": "https://en.wikipedia.org/wiki/Watermelon",
  "title": "Watermelon",
  "lang": "en",
  "encyclopedic": true,
  "lead_image": "https://upload.wikimedia.org/…jpg",
  "section_count": 9,
  "sections": [                       // ← the paginated FOCUS MODE
    {
      "index": 0,
      "title": "Watermelon",          // lead page titled by the article
      "anchor": null,                 // original-page DOM id (scroll-sync), when present
      "blocks": [
        {"kind": "paragraph", "text": "Watermelon (Citrullus lanatus) is a flowering plant…"},
        {"kind": "heading", "level": 3, "text": "Etymology", "anchor": "Etymology"},
        {"kind": "list", "items": ["…", "…"]},
        {"kind": "image", "src": "https://…jpg", "caption": "A cut watermelon"},
        {"kind": "quote", "text": "…"}
      ]
    }
    // …one entry per top-level (H2) section; H3/H4 subsections stay inline via {kind:"heading",level:3}
  ],
  "infobox":   {"title": "Watermelon", "image": "…", "rows": [{"label": "Kingdom", "value": "Plantae"}]},
  "tables":    [{"caption": "…", "section": "Nutrition", "rows": [["Energy", "127 kJ"], …]}],
  "gallery":   [{"src": "…", "caption": "…"}],
  "references": {"1": "Smith, J. (2020)…", "2": "…"},          // [N] markers hidden from body
  "languages": [{"lang": "fr", "title": "Pastèque", "url": "https://fr.wikipedia.org/wiki/Pastèque"}]
}

Block kinds

paragraph · heading (level 2–4, anchor) · image (src, caption) · list (items[]) · quote

Pagination (focus mode)

Each top-level (H1/H2) heading starts a new page; H3/H4 subsections stay inline within their parent page. A page needs ≥ 220 chars of real body before the next heading spins up a new page (empty/redundant sections like a structurally-captured "References" are dropped, not shown as header-only fragments); a thin tail page folds back into the previous one. Nothing is dropped — every real block lands in exactly one section.

Scope

Currently Wikipedia / Wikipedia-style encyclopedic pages (*.wikipedia.org/wiki/<Title>). The Melon browser applies the same reader idea to news, blogs, and other page types; those engines may follow here.

License

MIT © alvations