worldjen 0.4.0

WorldJen Python SDK for running evaluation pipelines on video and world models

WorldJen Python SDK

WorldJen provides high-performance benchmarks for video and world models. It empowers researchers and builders with a unified platform for accurate, fast, and clear evaluations across dimensions like subject consistency, motion smoothness, physical mechanics, and more.

This package provides the Python SDK and CLI. Use it to create runs, manage models and API keys, and operate GPU runners from the command line or from your own code.

The SDK provides the same capabilities as the runs, models, and api-keys CLI commands, plus the high-level worldjen.run() for end-to-end pipeline execution. Runner commands (systemd, register, etc.) remain CLI-only.

Requirements: Python 3.10+

Installation

Install the core package (for programmatic runs and CLI):

pip install worldjen

With uv:

uv pip install worldjen

If you use a virtual environment, activate it so the worldjen CLI is on your PATH (e.g. source .venv/bin/activate).

To run the runner (Celery worker for GPU execution), install the optional runner extra:

pip install "worldjen[runner]"

With uv, activate a virtual environment first:

uv venv && source .venv/bin/activate
uv pip install "worldjen[runner]"

The runner requires a Linux system with systemd for service management.

Configuration

Set your API key via worldjen.config() or environment variables:

• WORLDJEN_API_KEY — API key (required for creating runs and for CLI commands that modify data)
• WORLDJEN_VIDEO_DIR — Directory for cached prompts and run outputs (default: ~/.worldjen/data/videos)

import worldjen

worldjen.config(api_key="your-api-key")
worldjen.config(video_dir="/path/to/videos")  # optional

Usage

Dimensions

Use the Dimensions enum when specifying evaluation dimensions:

from worldjen import Dimensions

dims = [Dimensions.SUBJECT_CONSISTENCY, Dimensions.SCENE_CONSISTENCY]
# or strings: dims = ["subject_consistency", "scene_consistency"]

Running a pipeline

Your pipeline must be one of:

• A callable: pipeline(prompt, **kwargs) -> list[frames]
• An object with a generate method: pipeline.generate(prompt, **kwargs) -> list[frames]
• An object with an infer method: pipeline.infer(prompt, **kwargs) -> list[frames]

Frames can be PIL Images or numpy arrays (HWC; the SDK handles writing to video).

import worldjen

def my_pipeline(prompt, **kwargs):
    # Generate frames from prompt (text-to-video)
    ...
    return frames  # list of PIL or numpy arrays

result = worldjen.run(
    my_pipeline,
    dimensions=[worldjen.Dimensions.SUBJECT_CONSISTENCY],
    run_name="my-eval-run",
    model_id="my-org/my-model-checkpoint-xyz",
    wait_for_evals=True,
    # optional pipeline kwargs: num_frames=16, etc.
)

print(result.run_id, result.status, result.video_paths, result.eval_results)
if result.error_message:
    print("Error:", result.error_message)

On failure (run creation, pipeline error, or upload), the run is marked failed and RunResult includes status="failed" and error_message.

Examples

See the worldjen-examples GitHub repo for runnable scripts (e.g. LTX-Video with real prompts from worldjen/prompts/ and a single dimension, with notes on Dimensions.ALL and multiple dimensions).

API summary

High-level (primary)

• worldjen.config(api_key=..., video_dir=..., api_url=..., timeout=...) — Set or override config; returns the config object.
• worldjen.run(pipeline, dimensions, run_name=None, model_id=None, wait_for_evals=True, **pipeline_kwargs) — End-to-end: create a run, run your pipeline locally, generate and upload videos, then optionally poll until evals are ready. Returns RunResult(run_id, video_paths, output_dir, eval_results, status, error_message).

Run management (worldjen.runs)

Lower-level API for inspecting and managing runs (equivalent to most worldjen runs CLI commands):

• worldjen.runs.create(name, dimensions, *, runner_id, model_id, run_instructions=None) — Create a run and enqueue it on the given runner (same as the web UI). Returns run ID. Requires keyword-only runner_id and model_id (MongoDB ids from the dashboard). Use worldjen.run() for end-to-end local pipeline execution; that path uses an internal API call and does not require a worker queue. For rare cases that only need a run record without enqueueing, use worldjen._api.create_run (omit runner_id).
• worldjen.runs.list_runs(status=None, page=1, limit=50) — List runs.
• worldjen.runs.get(run_id) — Fetch run metadata and result data.
• worldjen.runs.cancel(run_id) — Cancel a run.
• worldjen.runs.delete(run_id) — Delete a run.
• worldjen.runs.get_logs(run_id) — Fetch run logs.
• worldjen.runs.get_csv(run_id) — Fetch run CSV as bytes.
• worldjen.runs.get_videos(run_id) — List video metadata (id, prompt, url).
• worldjen.runs.download_videos(run_id, output_dir=".") — Download all run videos to a directory.

Model management (worldjen.models)

• worldjen.models.list_user(sort_by=None) — List your models.
• worldjen.models.list_base(sort_by=None) — List base (suggested) models.
• worldjen.models.list_ref(sort_by=None) — List reference models.
• worldjen.models.create(name, provider, hf_repo_id, model_type, encrypted_tokens=None) — Create a model. For private repos, use --hf-token via CLI or configure in the dashboard.
• worldjen.models.delete(model_id) — Delete a model.

API keys (worldjen.api_keys)

• worldjen.api_keys.list_keys() — List API keys.
• worldjen.api_keys.create(name=None, expires_in=None) — Create an API key.
• worldjen.api_keys.revoke(key_id) — Revoke an API key.

Create and revoke are also available in the dashboard.

CLI

All CLI commands that create or read data accept an API key via the WORLDJEN_API_KEY environment variable or --api-key. To point at a non-production API (developers only), set WORLDJEN_API_URL in the environment; it is intentionally not listed in worldjen --help.

Runs

• worldjen runs create --name NAME --dimensions DIM1,DIM2 --runner-id RUNNER_ID --model-id MODEL_ID [--run-instructions JSON_OR_PATH] — Create a run on a worker queue (same as the dashboard). --run-instructions is inline JSON or a path to a .json file.
• worldjen runs list [--status STATUS] [--page N] [--limit N] — List runs.
• worldjen runs get RUN_ID — Show run metadata.
• worldjen runs cancel RUN_ID — Cancel a run.
• worldjen runs delete RUN_ID — Soft-delete a run.
• worldjen runs logs RUN_ID — Print run logs.
• worldjen runs csv RUN_ID [--output FILE] — Download run CSV.
• worldjen runs videos RUN_ID — List video metadata (id, prompt, url).
• worldjen runs download-videos RUN_ID [--output-dir DIR] — Download all run videos.

Dimensions

• worldjen dimensions list [--json] — List all available evaluation dimensions (id, name, description). Pass --json for machine-readable output.

Models

• worldjen models list — List your models.
• worldjen models list-base — List base (suggested) models.
• worldjen models list-ref — List reference models.
• worldjen models create --name NAME --provider P --hf-repo-id REPO --type text_to_video [--hf-token TOKEN] — Create a model. For private repos, pass --hf-token (or set WORLDJEN_HF_TOKEN); the token is encrypted per registered runner so runners can pull the model.
• worldjen models delete MODEL_ID — Delete a model.

API keys

• worldjen api-keys list — List API keys. Create and revoke keys in the dashboard.

Runner

Runner commands require the worldjen[runner] extra. Local commands act on this machine (config file, systemd service). Backend commands act on your account’s runner resources via the API.

All runner commands accept --name INSTANCE to target a specific runner instance (default: "default"). This enables multiple runners on a single machine, each with its own config (runner-{name}.conf), key file, and systemd service (worldjen-runner@{name}).

Backend (API / your account) — require WORLDJEN_API_KEY or --api-key:

• worldjen runner create [--name NAME] [--data-dir DIR] — Create a runner resource in the backend, register this machine, install the systemd service, and start it.
• worldjen runner delete [--name NAME] [--runner-id ID] — Delete the runner from the backend; also stops and uninstalls the local service. Fails if the runner has queued runs.
• worldjen runner list — List runners for your account.
• worldjen runner list --local — List locally registered runner instances on this machine.

Local (this machine) — act on local config and systemd; Linux with systemd required for install/start/stop/logs:

• worldjen runner register --token TOKEN [--name NAME] [--api-url URL] — Register this machine using a one-time token from the web UI; writes instance config, then installs and starts the service.
• worldjen runner install [--name NAME] [--data-dir DIR] — Install and start the systemd service (requires an existing config from a prior register).
• worldjen runner uninstall [--name NAME] — Stop and uninstall the systemd service.
• worldjen runner start [--name NAME] — Start the installed service.
• worldjen runner stop [--name NAME] — Stop the service.
• worldjen runner status [--name NAME] — Show systemctl status for the runner service.
• worldjen runner logs [--name NAME] [-f] [-n N] — Tail logs of the systemd service.

The runner runs as a Celery worker. Jobs are submitted from the web UI or when you create a run and select a runner in the dashboard. Multiple runners share the same Python venv and model cache but consume from separate queues.

Links

• WorldJen — Web app and dashboard
• Documentation — Full docs and guides