noetlctl 2.5.3

NoETL CLI (Rust) - Command-line interface for NoETL workflow automation

NoETL CLI

NoETL workflow automation CLI - Execute playbooks locally or orchestrate distributed server-worker pipelines.

Installation

Via Cargo

cargo install noetl

Via Homebrew (macOS)

brew tap noetl/tap
brew install noetl

Via APT (Ubuntu/Debian)

echo 'deb [trusted=yes] https://noetl.github.io/apt jammy main' | sudo tee /etc/apt/sources.list.d/noetl.list
sudo apt-get update
sudo apt-get install noetl

Quick Start

Create a simple playbook:

# hello.yaml
apiVersion: noetl.io/v2
kind: Playbook
metadata:
  name: hello_world

workflow:
  - step: start
    tool:
      kind: shell
      cmds:
        - "echo 'Hello from NoETL!'"
    next:
      - step: end
  - step: end

Run it:

noetl run hello.yaml

Features

• Local Playbook Execution - Run workflows without server infrastructure
• HTTP Actions - Make REST API calls with automatic pagination
• Conditional Flow - Dynamic routing with case/when/then/else
• Playbook Composition - Call sub-playbooks for modularity
• Server/Worker Management - Start/stop distributed services
• Kubernetes Operations - Deploy to K8s clusters
• Database Management - Initialize and validate schemas

Usage

Check version:

noetl --version

Local Execution

# Run playbook locally
noetl run playbook.yaml

# Pass variables
noetl run playbook.yaml --set env=prod --set version=v2.5.3

# Verbose output
noetl run playbook.yaml --verbose

Server Management

Start NoETL server:

noetl server start
noetl server start --init-db  # Initialize database on startup

Stop NoETL server:

noetl server stop
noetl server stop --force  # Force stop without confirmation

Worker Management

Start NoETL worker:

noetl worker start
noetl worker start --max-workers 4

Stop NoETL worker:

noetl worker stop  # Interactive selection if multiple workers
noetl worker stop --name my-worker
noetl worker stop --name my-worker --force

Database Management

Initialize database schema:

noetl db init

Validate database schema:

noetl db validate

Build Management

Build NoETL Docker image:

noetl build
noetl build --no-cache  # Build without using cache

The build command:

• Builds the Docker image with a timestamp-based tag
• Saves the tag to .noetl_last_build_tag.txt for deployment use
• Streams build output to console
• Replaces task docker-build-noetl

Kubernetes Management

Deploy NoETL to kind cluster:

noetl k8s deploy

Remove NoETL from cluster:

noetl k8s remove

Rebuild and redeploy:

noetl k8s redeploy
noetl k8s redeploy --no-cache  # Rebuild without cache

Full reset (schema reset + redeploy + test setup):

noetl k8s reset
noetl k8s reset --no-cache  # Reset with clean build

The k8s commands:

• deploy: Applies Kubernetes manifests to kind cluster
• remove: Deletes NoETL resources from cluster
• redeploy: Builds image, loads to kind, and deploys (replaces task noetl:k8s:redeploy)
• reset: Full workflow - resets database schema, redeploys, runs test setup (replaces task noetl:k8s:reset)

Configuration and Contexts

noetl supports multiple contexts to manage different server environments.

Add a Context

noetl context add local --server-url http://localhost:8082 --set-current
noetl context add prod --server-url http://noetl-server:8082

List Contexts

noetl context list

Switch Context

noetl context use prod

Show Current Context

noetl context current

CLI Mode

Catalog Management

Register a resource (auto-detects kind: Credential or Playbook):

noetl catalog register tests/fixtures/playbooks/data_transfer/http_to_postgres_transfer/http_to_postgres_transfer.yaml

Get resource details:

noetl catalog get tests/fixtures/playbooks/data_transfer/http_iterator_save_postgres

List resources:

noetl catalog list Playbook --json

Execution

Execute a playbook:

noetl execute playbook tests/fixtures/playbooks/regression_test/master_regression_test --json

Get execution status:

noetl execute status 522107710393811426 --json

Credentials

Get credential details:

noetl get credential gcs_service_account --include-data

SQL Query Execution

Execute SQL queries via NoETL Postgres API:

# Query with table format (default)
noetl query "SELECT * FROM noetl.keychain LIMIT 5"

# Query with specific schema
noetl query "SELECT execution_id, credential_name FROM noetl.keychain WHERE execution_id = 12345" --schema noetl

# Query with JSON output
noetl query "SELECT * FROM noetl.event ORDER BY created_at DESC LIMIT 10" --format json

# Query public schema tables
noetl query "SELECT * FROM users LIMIT 5" --schema public --format table

Output Formats:

• table (default): Formatted ASCII table with borders
• json: Pretty-printed JSON output

Example Output (table format):

┌────────────────────┬────────────────┬──────────────┐
│ execution_id       │ credential_name│ access_count │
├────────────────────┼────────────────┼──────────────┤
│ 507861119290048685 │ openai-api-key │ 0            │
│ 507861119290048686 │ postgres-creds │ 2            │
└────────────────────┴────────────────┴──────────────┘
(2 rows)

Registering (Legacy/Explicit)

Register a Credential:

noetl register credential -f tests/fixtures/credentials/pg_k8s.json

Register a Playbook:

noetl register playbook -f tests/fixtures/playbooks/api_integration/auth0/provision_auth_schema.yaml

Direct Execution/Status/List

Execute a Playbook:

noetl run playbook api_integration/auth0/provision_auth_schema

Get Execution Status:

noetl status <execution_id>

List Resources:

noetl list Playbook

Interactive TUI Mode

Run noetl with the -i or --interactive flag:

noetl --interactive

• Navigation: Use Up/Down arrows or j/k to navigate lists.
• Refresh: Press r to refresh the data.
• Quit: Press q to exit.

Docker Integration

The noetl binary is built into the Docker image using a multi-stage build:

# Rust builder stage compiles the CLI
FROM rust:1.75-slim as rust-builder
WORKDIR /build
COPY noetlctl/ ./
RUN cargo build --release

# Production stage includes the binary
COPY --from=rust-builder /build/target/release/noetl /usr/local/bin/noetl

The Kubernetes manifests use the Rust CLI for server and worker management:

Server deployment:

command: ["noetl"]
args: ["server", "start"]

Worker deployment:

command: ["noetl"]
args: ["worker", "start"]

This provides a unified binary for both local development and containerized deployments.

Command Mapping

The Rust CLI replaces several task commands:

Task Command	noetl Command
task docker-build-noetl	noetl build
task noetl:k8s:deploy	noetl k8s deploy
task noetl:k8s:redeploy	noetl k8s redeploy
task noetl:k8s:reset	noetl k8s reset
task noetl:server:start	noetl server start
task noetl:worker:start	noetl worker start