Skip to main content

Helpers to fetch & parse text on pages with requests, lxml, & beautifulsoup4

Project description


Install system requirements for lxml

% sudo apt-get install -y libxml2 libxslt1.1 libxml2-dev libxslt1-dev zlib1g-dev


% brew install libxml2

Install with pip

% pip3 install parse-helper

Optionally install ipython with ``pip3 install ipython`` to enable
``ph-soup-explore`` command


The ph-ddg, ph-download-files, ph-download-file-as, and ph-soup-explore scripts are provided

$ venv/bin/ph-ddg --help
Usage: ph-ddg [OPTIONS] [QUERY]

  Pass a search query to duckduckgo api

  --help  Show this message and exit.

$ venv/bin/ph-download-files --help
Usage: ph-download-files [OPTIONS] [ARGS]...

  Download all links to local files

  - args: urls or filenames containing urls

  --help  Show this message and exit.

$ venv/bin/ph-download-file-as --help
Usage: ph-download-file-as [OPTIONS] URL [LOCALFILE]

  Download link to local file

  - url: a string - localfile: a string

  --help  Show this message and exit.

$ venv/bin/ph-soup-explore --help
Usage: ph-soup-explore [OPTIONS] [URL_OR_FILE]

  Create a soup object from a url or file and explore with ipython

  --help  Show this message and exit.
In [1]: import parse_helper as ph

In [2]: ph.USER_AGENT
Out[2]: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/58.0.3029.110 Chrome/58.0.3029.110 Safari/537.36'

In [3]: ph.duckduckgo_api('adventure time')
2019-08-27 06:21:05,303: Fetching JSON from
[{'text': 'Adventure Time An American animated television series created by Pendleton Ward for Cartoon Network.',
  'thumbnail': '',
  'link': ''},
 {'text': '"Adventure Time" (pilot) An animated short created by Pendleton Ward, as well as the pilot to the Cartoon Network series...',
  'thumbnail': '',
  'link': ''},
 {'text': "Adventure Time (1959 TV series) A local children's television show on WTAE-TV 4 in Pittsburgh, Pennsylvania, from 1959 to 1975.",
  'thumbnail': '',
  'link': ''},
 {'text': "Adventure Time (1967 TV series) A Canadian children's adventure television series which aired on CBC Television in 1967 and 1968.",
  'thumbnail': '',
  'link': ''},
 {'text': 'Adventure Time (album) The second album for the rock/pop trio The Elvis Brothers.',
  'thumbnail': '',
  'link': ''}]

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for parse-helper, version 0.1.21
Filename, size File type Python version Upload date Hashes
Filename, size parse_helper-0.1.21-py3-none-any.whl (7.5 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page