Skip to main content

Select and reverse-Markdown (html2text) web page fragments.

Project description

frag2text

Markdown gives you HTML from plain text and html2text reverses the process. If you want the plain text version of a specific section of a web page (an HTML fragment), you would normally do the selecting (parsing) first, then generate the Markdown text to preserve some formatting.

I made frag2text to:

  • easily select a web page fragment by CSS selector or XPath expression

  • get the plain text of the fragment with some formatting intact for later use with Markdown

  • not shell out to a another program (like lynx -dump)

  • not parse HTML or text directly

  • use html5lib for robust parsing

  • have a simple python module that is easy to maintain (encapsulating the tricky business of subclassing parsers, treebuilders and serializers, by keeping it simple – this shouldn’t require BeautifulSoup)

This is a problem I’ve tinkered with for some time and solved in many different ways. It seems trivial but gets ridiculous quickly. If you have any suggestions or want to share your experiences with other tools, please let me know.

Usage

python

>>> from frag2text import frag2text
>>> help(frag2text)
Help on function frag2text in module frag2text:

frag2text(endpoint, stype, selector, clean=False, raw=False, verbose=False)
    returns Markdown text of selected fragment.

    Args:
        endpoint: URL, file, or HTML string
        stype: { 'css' | 'xpath' }
        selector: CSS selector or XPath expression
    Returns:
        Markdown text
    Options:
        clean: cleans fragment (lxml.html.clean defaults)
        raw: returns raw HTML fragment
        verbose: show http status, encoding, headers

shell

$ frag2text.py -h
usage: frag2text.py [-h] [-c] [-r] [-v] endpoint {css,xpath} selector

reverse Markdown (html2text) HTML fragments.

positional arguments:
  endpoint       URL, file, or HTML string
  {css,xpath}    fragment selector type
  selector       CSS select statement or XPath expression

optional arguments:
  -h, --help     show this help message and exit
  -c, --clean    clean fragment (lxml.html.clean defaults)
  -r, --raw      output raw fragment
  -v, --verbose  print status, encoding, headers

Examples

python

from frag2text import frag2text
info = frag2text('http://wikipedia.org/wiki/Amanita', 'css', '.infobox')

shell

$ frag2text.py "<ht?+><borkt><h1>hello" xpath //h1
...
# hello

CSS select

$ frag2text.py http://wikipedia.org/wiki/Amanita css .infobox
_Amanita_
---
![Fliegenpilz-1.jpg](//upload.wikimedia.org/wikipedia/commons/thumb/d/d1
/Fliegenpilz-1.jpg/230px-Fliegenpilz-1.jpg)
_[Amanita muscaria](/wiki/Amanita_muscaria)_
Albin Schmalfuß, 1897
[Scientific classification](/wiki/Biological_classification)
Kingdom: | [Fungi](/wiki/Fungi)
Division: | [Basidiomycota](/wiki/Basidiomycota)
Class: | [Agaricomycetes](/wiki/Agaricomycetes)
Order: | [Agaricales](/wiki/Agaricales)
Family: | [Amanitaceae](/wiki/Amanitaceae)
Genus: | _**Amanita**_
[Pers.](/wiki/Christian_Hendrik_Persoon) (1794)
[Type species](/wiki/Type_species)
_[Amanita muscaria](/wiki/Amanita_muscaria)_
([L.](/wiki/Linnaeus)) [Lam.](/wiki/Lam.) (1783)
[Diversity](/wiki/Biodiversity)
[c.600 species](/wiki/List_of_Amanita_species)

XPath expression

$ frag2text.py http://en.wikipedia.org/wiki/Amanita xpath '//p[1]'

The [genus](/wiki/Genus) _**Amanita**_ contains about 600 [species](/wik
i/Species) of [agarics](/wiki/Agarics) including some of the most [toxic
](/wiki/Toxic) known [mushrooms](/wiki/Mushrooms) found worldwide, as we
ll as some well-regarded edible species. This genus is responsible for a
pproximately 95% of the fatalities resulting from [mushroom poisoning](/
wiki/Mushroom_poisoning), with the [death cap](/wiki/Death_cap) accounti
ng for about 50% on its own. The most potent toxin present in these mush
rooms is α[-amanitin](/wiki/%CE%91-amanitin).

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

frag2text-0.0.1.tar.gz (6.0 kB view details)

Uploaded Source

Built Distribution

frag2text-0.0.1-py2.py3-none-any.whl (7.4 kB view details)

Uploaded Python 2Python 3

File details

Details for the file frag2text-0.0.1.tar.gz.

File metadata

  • Download URL: frag2text-0.0.1.tar.gz
  • Upload date:
  • Size: 6.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for frag2text-0.0.1.tar.gz
Algorithm Hash digest
SHA256 1d110634ae10c3fcb8f56b74759177cc3911dfc69eee3f216c3f79733182493a
MD5 913ec71850aa2e9771245fb8781bd58d
BLAKE2b-256 76276040ff205737c5e9af76f471a71b6b2dc7ab1c85b359391555dec0037861

See more details on using hashes here.

File details

Details for the file frag2text-0.0.1-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for frag2text-0.0.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 c9ca17249a4b00cb5e58df996e122e5bdc6c2e9bd442e3fc0e7b53446b1b6ceb
MD5 860f476d4619de0bcc1b92acbb7cbe62
BLAKE2b-256 91d3697705ae0303bb8b7ca6476494e97e020c368e41d2058996caf97ca804dc

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page