jtc-client

jtc-client is a small, standalone Python client for Judges That Code tool services. The first public client is JudgeClient, which calls the /judge endpoint exposed by a JTC/LiteRegistry gateway.

The package intentionally contains no model-serving or dataset code. Its only runtime dependency is aiohttp.

Install

pip install jtc-client

For local development:

pip install -e ".[test,publish]"

Judge one output

import asyncio
import json

from jtc_client import JudgeClient


async def main() -> None:
    client = JudgeClient(
        "http://127.0.0.1:1212/judge",
        service_model_path="judge",
        timeout=600,
        max_retries=3,
    )
    response = await client.verify_output(
        input="What is the capital of France?",
        output="The capital of France is Paris.",
        rubrics=[
            "The response correctly identifies Paris as the capital of France."
        ],
        model="/models/my-rubric-judge",
    )
    print(json.dumps(response, indent=2))


asyncio.run(main())

model selects the LLM that performs the judgment. service_model_path selects the /judge service pool through the gateway. They are intentionally separate fields.

Lion-4B example

See examples/lion4b.py for the concrete Lion-4B judge configuration used by JTC.

The example expects a local LiteRegistry gateway on port 1212. For the existing Jupiter registry, launch one with:

literegistry gateway \
  --registry redis://jupiter-cs-aus-183.reviz.ai2.in:59936 \
  --port 1212

The Lion-4B model pool and the service registered as rubrichub-lion4b-step800-per-rubric-judge must already be alive in that registry.

Request contract

The client sends this JSON object to /judge:

{
  "input": "original task",
  "output": "candidate response",
  "rubrics": ["one or more textual criteria"],
  "model": "/registered/judge/model",
  "model_path": "judge-service-name"
}

The complete JSON object returned by the server is returned unchanged. It normally contains one judgment and trace per supplied rubric.

Build and publish

python -m build
python -m twine check dist/*
python -m twine upload dist/*

Publishing requires a PyPI account and token. Verify that the distribution name jtc-client is available before the first upload.