JTC

jtc is one Python distribution with one import namespace and three cooperating subpackages:

jtc
├── common   # shared data models, IO, sources, templates, and workflows
├── eval     # scheduler-free evaluation workers, outcomes, and reporting
└── datadev  # data recipes, materialization, mixtures, and training configs

These are not independent packages. Installing jtc installs all three under the same namespace. The historical top-level imports jtc_data_commons, jtceval, and datadev are no longer part of the package API.

JTC implements data and evaluation work. It does not own training or evaluation infrastructure. PrimeBeaker loads JTC's Python workers, creates the LiteRegistry service topology, and schedules the jobs. RL trace inspection and export come from the standalone rltracer package. Checkpoint and evaluation provenance indexing comes from primebeaker watcher.

Install

Install the released jtc-core distribution (Python imports and commands remain jtc):

pip install jtc-core

JTC requires Python 3.12.

For development from this checkout:

pip install -e .

Install every optional JTC stack for development with:

pip install '.[all]'

Available extras are:

• evals: OpenAI, PyArrow, and RLTracer support used by evaluation tooling.
• harness: Verifiers/Harbor support used by verifier workflows.
• search: Pyserini for local-search workloads.
• trace: RLTracer without the other optional stacks.
• all: all optional JTC dependencies.

PrimeBeaker is intentionally not a JTC dependency. A runtime that schedules JTC work installs PrimeBeaker separately; the supported evaluation image pins and installs released versions of both packages from PyPI.

Package boundaries

• jtc.common: schemas, parquet IO, rubric execution, source adapters, shared tool integration, prompt templates, and registered Python workflows.
• jtc.eval: evaluation workload contracts and workers, outcome archives, audits, plots, and Evalboard export.
• jtc.datadev: typed SFT/RL data recipes, materialization, Pot sampling and optimization, and training-TOML generation.

Verifier environments and reward functions live in primebeaker.environments. Scheduling, dynamic ports, LiteRegistry deployment, readiness, supervision, resume, and shutdown coordination also live in PrimeBeaker. Runtime records, datasets, checkpoints, and evaluation results stay outside this repository.

Data and training-configuration flow

jtc.datadev turns versioned YAML data recipes into materialized SFT or RL datasets and then renders a compatible training TOML. It does not launch training:

data recipe YAML -> materialized dataset + manifest -> training TOML -> PrimeBeaker

The CLI is implemented with Fire:

# Validate without writing data.
datadev data sft examples/datadev/sft-data-smoke.yaml --dry-run

# Materialize data and render the corresponding training TOML.
datadev prepare rl \
  examples/datadev/rl-data-smoke.yaml \
  examples/datadev/rl.toml \
  /weka/gfaria/generated/example-rl.toml

Canonical recipe configs live under examples/configs/data/{sft,rl} and training configs under examples/configs/train/{sft,rl}. The small files under examples/datadev demonstrate the formats. Generated datasets and manifests must use an external artifact directory; the resulting TOML is handed to PrimeBeaker for SFT or RL scheduling.

The jtc command exposes common parquet inspection and sampling operations:

jtc inspect /weka/gfaria/records/example.parquet
jtc sample-jrows INPUT.parquet OUTPUT.parquet --max-input-objects=100

In Python, import through the unified namespace:

from jtc.common.schema import JRecord, JRow
from jtc.datadev import Pot, build_data
from jtc.eval import EvaluationArchive, EvaluationWorkloadConfig

Python-only evaluation workloads

jtc.eval owns benchmark selection and evaluation logic. PrimeBeaker owns the YAML descriptions and all infrastructure lifecycle behavior: Beaker scheduling, native LiteRegistry deployment, endpoint readiness, signal forwarding, and cleanup. The dependency direction is PrimeBeaker -> JTC.

The evaluator always runs in a Beaker image containing released PyPI wheels; JTC source is never imported or copied into the image. Build and publish the pinned image using JTC's docker/Dockerfile.eval, then put its immutable beaker://... URI in the PrimeBeaker-owned eval YAML or pass it with --image:

image: beaker://IMMUTABLE_PRIMEBEAKER_JTC_IMAGE
workspace: ai2/oe-agents-holmes
budget: ai2/oe-omai
evaluation:
  model: /weka/gfaria/models/policy/step_400
  run_dir: /weka/gfaria/jtc-artifacts/evals/policy-step-400-smoke
  datasets: [healthbench]
  max_rows: 4
  runs: 1
  verifier_tools: terminal
  verifier_prompt_chat_template_path: templates/jtc-io-terminal-release.json

Lifecycle YAMLs live in PrimeBeaker under examples/configs/eval. Preview and submit through PrimeBeaker's Fire CLI:

primebeaker evaluation preview --config=examples/configs/eval/python_only_smoke.yaml
primebeaker evaluation submit --config=examples/configs/eval/python_only_smoke.yaml

PrimeBeaker selects a packaged Python entry point for each workload:

• standard benchmark evaluation: jtc.eval.worker
• search-agent evaluation: jtc.eval.search_worker
• any registered JTC workflow: jtc.eval.workflow_worker
• iterative rejection sampling: jtc.eval.rejection_sampling_worker
• rubric-judge audit: jtc.eval.rubric_judge_audit_worker

The generic workflow worker accepts every stable name exported by jtc.common.workflows.registry. This includes articulated/grouped submit, TMAX/Podman and binary-submit workflows, inference, rubric generation, and the other verifier workflows in the registry.

JTC validates the selected workflow and executes its Python main() while PrimeBeaker injects managed registry and gateway endpoints. A newly registered workflow therefore uses the existing generic worker rather than a new scheduler module.

For rejection sampling, PrimeBeaker owns the rounds, service topology, resume, and cleanup lifecycle. The JTC worker runs the selected verifier workflow; the canonical filtering and accepted-trajectory compilation algorithms live in jtc.common.rejection_sampling. No launcher or fixed-port infrastructure lives in JTC.

Templates are runtime data, not image code. A PrimeBeaker YAML can use a newly created JSON template by YAML-relative, checkout-relative, or absolute path. PrimeBeaker validates it on the submitting machine and writes a content-addressed snapshot under <run_dir>/.jtceval-inputs/templates/ before it submits the evaluator. The /weka mount exposes that snapshot to the PyPI-only evaluation image without rebuilding JTC. A dry run reports the future path without writing.

Rubric-judge audits use the same boundary. JTC implements Python sampling and judgment recovery while PrimeBeaker schedules the shadow model, registry, and gateway. The jtc.eval.rubric_judge_audit_worker calls those implementations directly:

primebeaker evaluation preview \
  --config=examples/configs/eval/rubric_judge_audit/qwen35_4b_grouped_vs_qwen36_35b_steps005_400.yaml
primebeaker evaluation submit \
  --config=examples/configs/eval/rubric_judge_audit/qwen35_4b_grouped_vs_qwen36_35b_steps005_400.yaml

The coordinator can resume from a complete sample or response stream and rejects partial artifacts unless --force=True is explicitly supplied.

Prime-RL TOMLs point directly at PrimeBeaker's package-qualified environment modules:

[orchestrator.train.source.legacy]
id = "primebeaker.environments.jtc_tool_label_env"

Prime-RL loads the dotted module path directly. No environment implementation is copied into JTC, and there are no JTC-owned harness aliases; callers use the PrimeBeaker module path or its environment registry.

Discover and run workflows

jtc-workflow list

Programmatically:

from jtc.common.workflows import load_workflow

workflow = load_workflow("verifier-tool-use")
workflow.main(...)

Only import-tested workflows are registered. Broken historical variants are kept in the repository for provenance but excluded from the wheel registry.

Local evaluation utilities

The jtceval CLI does not schedule work. It imports and summarizes existing evaluation outputs; lifecycle operations stay in PrimeBeaker:

jtceval import-evaluations SOURCE_DIRECTORY OUTPUT.json --dry-run

The distribution also installs these evaluation-side entry points:

• jtceval-evalboard: Evalboard export and local site tooling.
• jtceval-bon and jtceval-bon-plot: best-of-N configuration and plotting.

All JTC CLIs use Fire. Scheduling commands remain under primebeaker evaluation, primebeaker sft, and primebeaker rl.

Discover verifier environments

Programmatically:

from primebeaker.environments import load_environment_module

module = load_environment_module("jtc-tool-label")
environment = module.load_environment(dataset="/path/to/train.jsonl")

The stable direct import for the same implementation is:

from primebeaker.environments.jtc_tool_label_env import (
    load_environment,
)

Runtime prompt assets

Workflow defaults still use familiar paths such as templates/search-agent.json. resolve_resource_path() first honors an existing caller-provided path and then falls back to templates bundled in the wheel. This preserves repository overrides while allowing jobs to run from any working directory.

from jtc.common import load_prompt_template, resolve_resource_path

path = resolve_resource_path("templates/search-agent.json")
template = load_prompt_template(path)

Release and PyPI

.github/workflows/publish.yml uses PyPI Trusted Publishing. Publishing a GitHub release builds both the source distribution and wheel with Python 3.12, then uploads them using GitHub's short-lived OIDC identity; no API token is stored in this repository.

Create the pypi environment in GitHub Settings → Environments. For the first release, add a pending publisher in PyPI account publishing with these values:

• PyPI project name: jtc-core
• GitHub owner: goncalorafaria
• GitHub repository: jtc
• Workflow: publish.yml
• Environment: pypi

Publish by updating the version in pyproject.toml, merging that commit, and creating a GitHub release whose tag matches the version, such as v0.3.0. The release event runs the publishing workflow. The tag must be v<version> or <version> and match pyproject.toml; a mismatch stops the build. The first successful upload creates the PyPI project automatically. For an existing project, add the same publisher in its PyPI publishing settings.

Manual workflow runs build and validate artifacts only. Builds use strict Twine metadata checks; publishing runs in a separate job with OIDC permission. Draft releases and tag pushes alone do not publish a package.

Before creating the release, run:

python -m pytest -q tests/test_packaging.py tests/test_config_layout.py
python -m build
python -m twine check dist/*

The package tests import every registry entry, verify the unified layout and packaged template fallback, and catch stale names before a Beaker rollout.

PrimeBeaker owns the watcher implementation and command. The standalone rltracer project is versioned and released independently from JTC.