extty

Terminal-first ML experiment tracking: a Python SDK for logging runs and a Rust TUI for browsing them.

The TUI and S3 are both optional. Runs are stored locally, and the Python SDK alone can log and read them. Configure S3 to sync runs to a bucket.

Install

Python SDK

Requires Python 3.10+.

pip install extty            # or: uv add extty
pip install "extty[image]"   # adds Image logging

TUI

Prebuilt binaries for macOS, Linux and Windows are attached to each GitHub release. On macOS or Linux, set target to your platform and extract extty onto your PATH:

# aarch64-apple-darwin, x86_64-apple-darwin,
# x86_64-unknown-linux-gnu or aarch64-unknown-linux-gnu
target=aarch64-apple-darwin
mkdir -p ~/.local/bin
curl -LsSf "https://github.com/ekorman/extty/releases/latest/download/extty-$target.tar.xz" \
  | tar xJ --strip-components=1 -C ~/.local/bin "extty-$target/extty"

On Windows, download extty-x86_64-pc-windows-msvc.zip from the release and put extty.exe on your PATH.

To build from source instead:

cargo install --locked --git https://github.com/ekorman/extty extty

Logging

import extty

extty.init("my-project", name="run-1", config={"lr": 1e-3})
extty.log({"train/loss": 0.5}, step=0)
extty.finish()

The value's type decides how it is stored and rendered. Use / in names to group streams (train/loss, val/loss).

extty.log(
    {
        "train/loss": loss,
        "val/samples": extty.Example(
            prompt="Capital of France?",
            responses=["Paris", "Lyon"],
            rewards=[1.0, 0.0],
            groundtruth="Paris",
        ),
        "eval/cm": extty.ConfusionMatrix.from_array(cm, labels=["cat", "dog"]),
        "eval/roc": extty.Chart.from_arrays(fpr, tpr, ("fpr", "tpr")),
        "val/detections": extty.Image(frame, caption="epoch 3"),
    },
    step=step,
)

BatchExample takes parallel lists, one entry per prompt. from_array and from_arrays accept numpy or torch arrays.

In the TUI run view, focus a card with Enter and scrub through steps with ↑↓ (Shift jumps 10).

Reading runs

run = extty.get_run("my-project", "run-1")
run.metric_names
run.metric("train/loss")          # [MetricPoint(step, timestamp, value), ...]
run.examples("val/samples")
run.confusion_matrix("eval/cm")
run.chart("eval/roc")
run.image_bytes(run.images("val/detections")[-1])

With S3 configured, get_run reads a run straight from S3 if it isn't local.

Checkpoints

extty.save_checkpoint(step, state_dict=model.state_dict(), optimizer_state_dict=opt.state_dict())
extty.save_checkpoint(step, path="ckpt.pt")   # or a file written by torch.save

ckpt = extty.load_checkpoint(step)                              # active run
ckpt = extty.load_checkpoint_from("my-project", "run-1", step)  # any run
model.load_state_dict(ckpt["model_state_dict"])

Checkpoints are saved in the run directory, so they work without S3. With S3 configured they are uploaded instead, and kept locally only with keep_local=True or if the upload fails. Loads use a complete local copy when there is one and download from S3 otherwise. extty push does not upload checkpoints.

extty.delete_local_checkpoint(project, run, step) and extty prune local free disk by deleting local copies that S3 also has. Neither deletes a checkpoint's only copy unless you pass force=True to delete_local_checkpoint.

S3 sync (optional)

Configure S3 in ~/.extty/s3/config.toml, which both the SDK and TUI read:

bucket = "my-bucket"
prefix = "extty"          # optional
region = "us-west-2"      # optional
access_key_id = "..."     # optional, defaults to the AWS credential chain
secret_access_key = "..."
endpoint_url = "..."      # optional, for MinIO etc.

The SDK also accepts EXTTY_S3_BUCKET, EXTTY_S3_PREFIX, EXTTY_S3_REGION, EXTTY_S3_ACCESS_KEY_ID, EXTTY_S3_SECRET_ACCESS_KEY and EXTTY_S3_ENDPOINT_URL, which take precedence over the file. Once configured, runs sync to S3 as they log. Without it, the SDK makes no network requests.

With the TUI installed:

extty                     # open the TUI
extty pull [project/[run]]
extty push [project/[run]]
extty sync project/run    # pull, then push
extty prune local         # delete local checkpoints/artifacts that S3 has

-n/--dry-run previews, and -f/--force overwrites instead of merging. From Python: extty.push("my-project/", dry_run=True).

Where data is stored

Runs, checkpoints, the artifact cache and the S3 config all live under ~/.extty. Set EXTTY_HOME to move them, e.g. to scratch space on a cluster with a small home quota. The SDK and TUI both honor it.

export EXTTY_HOME=/scratch/$USER/extty