Skip to main content

A Cli tool for Grepsr Developers

Project description

A Cli tool for Grepsr Developers

Installation

$ pip install grepsr-cli

Usage

passing parameters to amazon_com service.

gcli crawler test -s amazon_com -p '{"urls":["https://amazon.com/VVUH4HJ","https://amazon.com/FV4434"]}'

if JSON is complex, use file instead

# contents of /tmp/amazon_params.json
{"urls": ["https://amazon.com/VV%20UH4HJ"], "strip": ["'", "\"", "\\"]}

gcli crawler test -s amazon_com --params-file '/tmp/amazon_params.json'

Hacks Used.

If the json parameter has a space, it might break parameter parsing. If the json parameter has a dash - and any character after it has a space, it will break parameter parsing. Cause: no double quoting around $@ in run_service.php:5:49 here This is fixed hackily by replacing string with its unicode \u0020 sequence. This works beacause $@ does not split on \u0020.

inject custom command.

Say, for example you wanted to a inject a php function so that it could be called from inside you service_code when testing locally. Note: All these files should only be created inside ~/.grepsr/tmp. Creating it outside will not work.

  1. Create a file called inject.php inside ~/.grepsr/tmp/
  2. Implement your function inside ~/.grepsr/tmp/inject.php
function addRowLocal($arr) {
    ...
    ...
}
  1. Create a file called inject.sh inside ~/.grepsr/tmp/
  2. inside inject.sh add:
alias php='php -d auto_prepend_file=/tmp/inject.php'

Note: the file location is /tmp/inject.php instead of ~/.grepsr/tmp/inject.php. This is because, the local path ~/.grepsr/tmp gets mapped to /tmp in the docker container. And inject.sh runs inside docker, instead of the local filesystem. 5. Add an entry in ~/.grepst/config.yml like so:

    php:
        ...
        sdk_image: ...
        pre_entry_run_file: inject.sh      # relative and limited to the tmp/ dir
  1. Now you can use addRowLocal() in your any of your files.
public function main($params) {
    ...
    $arr = $this->dataSet->getEmptyRow();
    addRowLocal($arr); // won't throw error
    ...
}

Development

Be sure to uninstall gcli first, with pip uninstall grepsr-cli

git clone git@bitbucket.org:zznixt07/gcli.git grepsrcli
cd grepsrcli
pip install -e .

Features Added

  • drop stash after pushed successully. Before this, all stashes were always kept.
  • run a custom shell file before running your crawler. This allows possiblity like always injecting a php function in all your crawlers.
  • auto add Dependencies: ... that your crawler class extends (dependecies that are not extended by crawler classes but used elsewhere is upcoming)

TODO:

  • Experiment with git rebase on deploy fail. git rebase origin/master --autostash && git push
  • Handle Prioritization of same plugin name across multiple repo more deterministically. (maybe prioritize cwd path?)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

grepsr_cli-0.9.16.tar.gz (31.1 kB view details)

Uploaded Source

Built Distribution

grepsr_cli-0.9.16-py3-none-any.whl (316.9 kB view details)

Uploaded Python 3

File details

Details for the file grepsr_cli-0.9.16.tar.gz.

File metadata

  • Download URL: grepsr_cli-0.9.16.tar.gz
  • Upload date:
  • Size: 31.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.10.4

File hashes

Hashes for grepsr_cli-0.9.16.tar.gz
Algorithm Hash digest
SHA256 6cd95b350797915739d75d8bb01d338a42f585ea01920aaf674d8f4b9d234243
MD5 8ab4c5c639c4414a7bb2b85ffe7c82ae
BLAKE2b-256 5aaad74d03248bd006b0e4b46d1e8705153b2019cc9f6d285cd427a00b9ace7b

See more details on using hashes here.

File details

Details for the file grepsr_cli-0.9.16-py3-none-any.whl.

File metadata

  • Download URL: grepsr_cli-0.9.16-py3-none-any.whl
  • Upload date:
  • Size: 316.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.10.4

File hashes

Hashes for grepsr_cli-0.9.16-py3-none-any.whl
Algorithm Hash digest
SHA256 f93e3bda9c91464f9d87836d33afd2b48f328fe883bbef321f260656326617d4
MD5 2878a2afb2608b79e77febe31a4bf1a3
BLAKE2b-256 03c7121639a5ce75d60f26abbcce7fdb64a8393906b417da9817e37e934cbd9c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page