Skip to main content

Task handlinlg.

Project description

TaskSquirrel

TaskSquirrel executes tasks from a Tasqfile.yml. It is similar to make or invoke. It also supports Kubernetes workflows like Argo and Airflow.

Differences from other tools are:

  • configuration is in YAML
  • it supports multiple scripts / languages
  • it supports Jinja2 templating
  • it has built-in support for executing code in Docker and on Kubernetes clusters
  • it is completely serverless: all it requires for interacting with a K8s cluster is Kubectl

TaskSquirrel is a good replacement for invoke and make even if you don't use the container-related features, but it is particularly well suited to data analysis and deep learning workflows.

Commands

  • tasq --help -- display help
  • tasq list -- list all commands
  • tasq run command -- run a command

Example

Simple replacement for invoke

tasks:
    venv:
      - doc: build the virtual environment
      - script: |
          test -d venv || virtualenv venv
          venv/bin/pip3 install -U -r requirements.txt

    newversion:
      - doc: increase the patch version number
      - python: |
          version = map(int, open("VERSION").read().strip().split("."))
          version[-1] += 1
          with open("VERSION", "w") as stream:
            stream.write(".".join(map(str, version)))

    publish:
      - doc: publish to PyPI
      - script: twine upload

Kubernetes Example

params:

  cluster: "mycluster"
  image: "mycompute"

tasks:

    image:
      docker_container: {{ image }}
        dockerfile: ./Docker
        push: tmbdev/{{ image }}
        context:
          - simple.key
          - scripts/start.sh

    kutest:
      - doc: start a job on K8s and wait for its completion
      - task: image
      - with_braceexpand: "{000..999}"
        param: shard
        tasks:
          - kubectl_apply: hello
            template_name: myjob
            params:
              id: hello-{{ shard }}
              script: |
                echo hello-{{ shard }};
                sleep 30
      - kubectl_wait:
        kind: jobs
        prefix: hello-
      - script: kubectl delete jobs --all

k8s_templates:

  myjob: |
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: "{{ id }}"
    spec:
      template:
        spec:
          containers:
          - name: main
            image: "{{ image }}"
            command: ["/bin/bash", "-c", "{{ script }}"]
          restartPolicy: Never
      backoffLimit: 4

Status

It's pre-alpha.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for tasq, version 0.0.3
Filename, size File type Python version Upload date Hashes
Filename, size tasq-0.0.3-py3-none-any.whl (4.9 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size tasq-0.0.3.tar.gz (4.7 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page