Skip to main content

DOM Traversing and Scraping using GraphQL

Project description

GDOM is the next generation of web-parsing, powered by GraphQL syntax and the Graphene framework.

Install it typing in your console:

pip install gdom

DEMO: Try GDOM online

Usage

You can either do gdom --test to start a test server for testing queries or

gdom QUERY_FILE

This command will write in the standard output (or other output if specified via --output) the resulting JSON.

Your QUERY_FILE could look similar to this:

{
  page(url:"http://news.ycombinator.com") {
    items: query(selector:"tr.athing") {
      rank: text(selector:"td span.rank")
      title: text(selector:"td.title a")
      sitebit: text(selector:"span.comhead a")
      url: attr(selector:"td.title a", name:"href")
      attrs: next {
         score: text(selector:"span.score")
         user: text(selector:"a:eq(0)")
         comments: text(selector:"a:eq(2)")
      }
    }
  }
}

Advanced usage

If you want to generalize your gdom query to any page, just rewrite your query file adding the $page var. So should look to something like this:

query ($page: String) {
  page(url:$page) {
    # ...
  }
}

And then, query it like:

gdom QUERY_FILE http://news.ycombinator.com

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for gdom, version 1.0.0
Filename, size File type Python version Upload date Hashes
Filename, size gdom-1.0.0.tar.gz (4.2 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page