o-o 0.4.1

A command line interface for running cloud jobs

o-o

o‑o is a command line interface for running jobs on ephemeral cloud instances with tracked inputs and outputs. Building data or MLOps pipelines is as simple as stringing together multiple commands, as easily as running locally, but with the power of cloud compute and storage.

Highlights

• Easily run any command in any language in cloud compute environments
• Flexible. Define your own run environments with docker images and machine types
• Traceable. Trace all inputs to the commands that produced them
• You control your data. Data and source code is stored in your managed buckets
• Support for Scaleway and Google Cloud (more to come...)

Getting Started

Follow the installation instructions to configure o-o. Once setup, define environments and datastores in an .ooconfig file:

project: test-project
environments:
  scaleway:
    provider: Scaleway
    image: docker.io/debian:stable-slim
    machinetype: STARDUST1-S
    region: fr-par-1
  gcp:
    provider: gcp
    image: docker.io/debian:stable-slim
    machinetype: e2-highcpu-2
    region: northamerica-northeast1-b
datastores:
  my-datastore:
    provider: Scaleway
    bucket: o-o-data
    region: fr-par

Lets start with a simple example that creates a data.txt file containing "Hello World" (under the hood, o-o will start and configure our scaleway environment, execute our command, copy output files to our datastore, and finally, delete the Scaleway instance).

$ o-o run --environment scaleway --message "create data.txt" -- \
    'echo "Hello World" > o://output/data.txt'

We completed our first run! o-o run --list displays the run's id (in this case ntus965ryy) and message:

$ o-o run --list
ntus965ryy create data.txt

You are free to run commands in any configured environment. Let's run a second command in our gcp environment that uses the output of ntus965ryy as input, and simply prints the contents of data.txt:

$ o-o run --environment gcp --message "print data.txt" -- \
    cat o://ntus965ryy/data.txt
Hello World

$ o-o run --list
ntus965ryy create data.txt
cxdbx8am38 print data.txt

And that's the basics. Find out more in the documentation and with o-o --help.