Skip to main content

Task handlinlg.

Project description

TaskSquirrel

TaskSquirrel executes tasks from a Tasqfile.yml. It is similar to make or invoke. It also supports Kubernetes workflows like Argo and Airflow.

Differences from other tools are:

  • configuration is in YAML
  • it supports multiple scripts / languages
  • it supports Jinja2 templating
  • it has built-in support for executing code in Docker and on Kubernetes clusters
  • it is completely serverless: all it requires for interacting with a K8s cluster is Kubectl

TaskSquirrel is a good replacement for invoke and make even if you don't use the container-related features, but it is particularly well suited to data analysis and deep learning workflows.

Commands

  • tasq --help -- display help
  • tasq list -- list all commands
  • tasq run command -- run a command

Example

Simple replacement for invoke

tasks:
    venv:
      - doc: build the virtual environment
      - script: |
          test -d venv || virtualenv venv
          venv/bin/pip3 install -U -r requirements.txt

    newversion:
      - doc: increase the patch version number
      - python: |
          version = map(int, open("VERSION").read().strip().split("."))
          version[-1] += 1
          with open("VERSION", "w") as stream:
            stream.write(".".join(map(str, version)))

    publish:
      - doc: publish to PyPI
      - script: twine upload

Kubernetes Example

params:

  cluster: "mycluster"
  image: "mycompute"

tasks:

    image:
      docker_container: {{ image }}
        dockerfile: ./Docker
        push: tmbdev/{{ image }}
        context:
          - simple.key
          - scripts/start.sh

    kutest:
      - doc: start a job on K8s and wait for its completion
      - task: image
      - with_braceexpand: "{000..999}"
        param: shard
        tasks:
          - kubectl_apply: hello
            template_name: myjob
            params:
              id: hello-{{ shard }}
              script: |
                echo hello-{{ shard }};
                sleep 30
      - kubectl_wait:
        kind: jobs
        prefix: hello-
      - script: kubectl delete jobs --all

k8s_templates:

  myjob: |
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: "{{ id }}"
    spec:
      template:
        spec:
          containers:
          - name: main
            image: "{{ image }}"
            command: ["/bin/bash", "-c", "{{ script }}"]
          restartPolicy: Never
      backoffLimit: 4

Status

It's pre-alpha.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tasq-0.0.3.tar.gz (4.7 kB view details)

Uploaded Source

Built Distribution

tasq-0.0.3-py3-none-any.whl (4.9 kB view details)

Uploaded Python 3

File details

Details for the file tasq-0.0.3.tar.gz.

File metadata

  • Download URL: tasq-0.0.3.tar.gz
  • Upload date:
  • Size: 4.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.5

File hashes

Hashes for tasq-0.0.3.tar.gz
Algorithm Hash digest
SHA256 c29204789658808ef9d0696eb274da597a3829e68f80853a9ac21e6675cd996e
MD5 c6b7313d7fb571de904ab801f0050d16
BLAKE2b-256 dd2e9c3d657d4d7504f48abdf1407f95143701465db691fdaad77caad1b76581

See more details on using hashes here.

File details

Details for the file tasq-0.0.3-py3-none-any.whl.

File metadata

  • Download URL: tasq-0.0.3-py3-none-any.whl
  • Upload date:
  • Size: 4.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/45.2.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.5

File hashes

Hashes for tasq-0.0.3-py3-none-any.whl
Algorithm Hash digest
SHA256 8d7d36c22b14c572bbaef56151928a211fdac34571d4c7cc925bf8cf602b7900
MD5 5486e53b92284137a2bb6918f4e0ff72
BLAKE2b-256 52d35aa0d59481ef2026be6fcedce9fc1d468a8d65a17a0a83d54eb25baf538e

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page