gallop

🐎 New level of python develop. Source code from github

More dynamic in design, more configuration driven

   [-;-/=_
    `-; \=_       ___        _________    __    __    ____  ____ 
      ) ,"-...--./===--__   / ____/   |  / /   / /   / __ \/ __ \ 
    __|/      /  ]`        / / __/ /| | / /   / /   / / / / /_/ /
   /;--> >...-\ <\_       / /_/ / ___ |/ /___/ /___/ /_/ / ____/ 
   `- <<,      7/-.\,     \____/_/  |_/_____/_____/\____/_/      
       `-     /(   `-     

Install

pip install gallop

Configuration Management

from gallop.config import BaseConfig

config = BaseConfig(a=1)
config.a = 2
config.b = 3

config.to_json("some/path.json")
config.to_yaml("some/path.yaml")

Run python task from config

You can turn any callable execution in to configuration, eg save the following

func_name: use:logging.warning
args:
  - "hello world"

to sometask.yaml and run gallop sometask

Is the same to run the python script

import logging
logging.warning("hello world")

param: configuration

We can change the value on the run, while pointing to the position in the config, using a chain of keys (keys to dict or list).

eg. to change the key args, of its 1st element, we can run

gallop sometask --param:args.0 changed_world

Advanced usage

Some simple grammar

• checkin: somekey, save the result to a centralized dictionary with key somekey
• checkout: somekey, use the result from the centralized dictionary with key somekey
• checkout: env:DATA_HOME, use the result from the environment variable DATA_HOME
• use:some.module, use or import the module some.module as the callable, eg
  • func_name: use:os.path.join, use the function os.path.join
  • func_name: use:pandas.DataFrame, use the class pandas.DataFrame

Examples

Inference BERT model

If you have transformers installed, you can run the following example directly to featurize a sentence

pred_task:
  - func_name: use:transformers.AutoModel.from_pretrained
    args:
      - bert-base-uncased
    checkin: model
    description: |
      Load pretrained model
  - func_name: use:transformers.AutoTokenizer.from_pretrained
    args:
      - bert-base-uncased
    checkin: tokenizer
    description: |
      Load pretrained tokenizer
  - func_name: tokenizer
    args:
      - - "Hello, the capital of [MASK] is Paris"
    kwargs:
      return_tensors: pt
      max_length: 128
      truncation: True
      padding: True
    checkin: inputs
  - func_name: use:logging.warning
    args:
      - func_name: model
        kwargs:
          input_ids:
            checkout: inputs.input_ids
          attention_mask:
            checkout: inputs.attention_mask
        checkin: features

This equals to the following python script

import logging
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
inputs = tokenizer(
    "Hello, the capital of [MASK] is Paris",
    return_tensors="pt",
    max_length=128,
    truncation=True,
    padding=True,
)
logging.warning(model(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
))

Save the yaml to run_bert.yaml and you can use gallop run_bert --output features to run the task.

Run in commandline with changed value, and printout one of the checkout value

gallop run_bert --loglevel debug --output features

And run it with CHANGED VALUE

gallop run_bert 、
     --param:pred_task.2.args.0.0 "The [MASK] house is where the POTUS live and work"   

Related project

From the same lead author

• Python category management accelerated with Rust
• Data science basic toolset forgebox