scrapqd 1.0.1b0

ScrapQD - Scraper Query Definition (beta)

ScrapQD consists of query definition created for scraping web data using GraphQL-Core which is port of GraphQL.js.

Library intends to focus on how to locate data from website and eliminate backend process of crawling. So people can just have xpath and get data right away.

It supports scraping using requests for traditional websites and selenium for modern websites (js rendering). Under selenium it supports Google Chrome and FireFox drivers.

ScrapQD library only uses lxml parser and xpath used to locate elements.

• Explore the docs »

• Source Code »

• Report Bug »

• Request Feature »

Getting Started

Query

Sample query is loaded to GraphQL UI and sample page is available within the server to practice.

query test_query($url: String!, $name: GenericScalar!) {
  result: fetch(url: $url) {
    name: constant(value: $name)
    summary: group {
      total_emp_expenses: text(xpath: "//*[@id='emp-exp-total']", data_type: INT)
      total_shown_expenses: text(xpath: "//*[@id='exp-total']/span[2]", data_type: INT)
      total_approved_expenses: text(xpath: "//*[@id='emp-exp-approved']/span[2]", data_type: INT)
    }
    exp_details: list(xpath: "//div[@class='card']") {
      name: text(xpath: "//div[contains(@class,'expense-emp-name')]")
      amount: group {
        money: text(xpath: "//h6[contains(@class,'expense-amount')]/span[1]", data_type: INT)
        name: text(xpath: "//h6[contains(@class,'expense-amount')]/span[2]")
      }
    }
  }
}

query variable

// url will be used in the above query
query_variables = {
    "url": "http://localhost:5000/scrapqd/sample_page/",
    "name": "local-testing"
}

Result

{
  "data": {
    "result": {
      "name": "local-testing",
      "summary": {
        "total_emp_expenses": 309,
        "total_shown_expenses": 40,
        "total_approved_expenses": 4
      },
      "exp_details": [
        {
          "name": "Friedrich-Wilhelm, Langern",
          "amount": {
            "money": 8800,
            "name": "egp"
          }
        },
        {
          "name": "Sebastian, Bien",
          "amount": {
            "money": 3365,
            "name": "mkd"
          }
        },
        {
          "name": "Rosa, Becker",
          "amount": {
            "money": 6700,
            "name": "xof"
          }
        },
        {
          "name": "Ines, Gröttner",
          "amount": {
            "money": 8427,
            "name": "npr"
          }
        }
      ]
    }
  }
}

Executing with client

from scrapqd.client import execute_sync

query = r"""
        query test_query($url: String!, $name: GenericScalar!) {
          result: fetch(url: $url) {
            name: constant(value: $name)
            summary: group {
              total_shown_expenses: regex(xpath: "//*[@id='exp-total']", pattern: "(\\d+)")
            }
          }
        }"""

query_variables = {
    "url": "http://localhost:5000/scrapqd/sample_page/",
    "name": "local-testing"
}
result = execute_sync(self.query, query_variables)

Integrating with existing Flask app

from flask import Flask

name = __name__
app = Flask(name)

@app.route("/")
def hello_world():
    return "<p>Hello, World!</p>"

from scrapqd.app import register_scrapqd
register_scrapqd(app,
                 register_sample_url=True,
                 redirect_root=True)

Test (for development)

• Clone the github repository

  git clone https://github.com/dduraipandian/scrapqd.git

• create virtual environment to work

  pip3 install virtualenv
  virtualenv scrapqd_venv
  source scrapqd_venv/bin/activate

• install tox

  pip install tox

• run tox from the project root directory

  • current tox have four python version - py37,py38,py39,py310

  • check your python version

    python3 --version
    
    # Python 3.9.10

  • once you get your version (example: use py39 for 3.9) to run tox

    tox -e py39

FAQs

• How to copy query from graphql ui to python code.

  • you can normally copy code from ui to python code to execute using client.

  • if you hav regex query, patterns needs to escaped in the python code. In such, use python raw strings, where backslashes are treated as literal characters, as above example.

• How to suppress webdriver logs

  • If you see webdriver logs like below, set WDM_LOG_LEVEL=0 as environment variable and run

    [INFO] [97002] [2022-03-14T02:18:26+0530] [SCRAPQD] [/webdriver_manager/logger.py:log():26] [WDM] [Driver [/99.0.4844.51/chromedriver] ...]

• How to change log level for scrapqd library

  • ERROR level is default logging. You can change this with SCRAPQD_LOG_LEVEL environment variable.

License

This project is licensed under the MIT License - see the LICENSE file for details