Skip to main content

Skyrunner scrapes with json definition

Project description

Skyrunner

Introduction

Skyrunner scrapes with json definition

Installation

$ pip install skyrunner

How to use

  1. Import skyrunner package at your project.
  2. You define steps of scraping in json file.
  3. You create class for share on scraping tasks and define self.attributes = dict() at constructor.
class TaskSharedData:
    def __init__(self):
        self.attributes = dict()
  1. You call skyrunner.setup_task_manager('scraping_steps_defined.json', TaskSharedData()) at your program.

sample of Scraping task definition json

{
   "description": "tasks description",
   "driver": {
     "name": "webdriver",
     "browser": "firefox"
   },
   "tasks": [
     {
       "description": "access to google",
       "action": "link",
       "params": {
         "url": "https://google.com"
       }
     },
     {
       "description": "search for mileshare",
       "action": "input",
       "params": {
         "type": "id",
         "path": "lst-ib",
         "data": "mileshare"
       }
     },
     {
       "description": "search(press the enter key)",
       "action": "enter",
       "params": {
         "type": "id",
         "path": "lst-ib"
       }
     }
   ]
 }

other support actions

  • click
{
  "description": "click task",
  "action": "click",
  "params": {
    "type": "id or xpath",
    "path": "target id or xpath"
  }
}
  • custom
{
   "description": "custom task for complex process, update the shared data etc..",
   "action": "custom",
   "params": {
     "type": "python",
     "script_file": "custom python script(class) path",
     "class_name": "custom python class name"
   }
 }
  • shared_input
{
  "description": "shared input task",
  "action": "shared_input",
  "params": {
    "type": "id or xpath",
    "path": "target id or xpath",
    "data": "dict key of shared data"
  }
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for skyrunner, version 0.0.4
Filename, size File type Python version Upload date Hashes
Filename, size skyrunner-0.0.4-py3-none-any.whl (10.1 kB) File type Wheel Python version py3 Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page