Yobro

A small language and Python library for fine-tuning LLMs that checks your project before it uses a GPU, reports honestly whether the model improved, refuses to ship a worse model, and turns production failures into reviewed training data.

Yobro does not replace the trainers. It drives Hugging Face TRL and PEFT for the heavy work and adds the checking, evaluation, release and monitoring around them.

yobro check project.yobro     # every check and estimate, no GPU
yobro run project.yobro       # train, compare with the base model, gate, ship
yobro monitor project.yobro   # watch live traffic, cluster failures, queue them for review

Version 0.1.1. Status is honest and specific; see What is tested.

Install

python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -e ".[train]"      # from this folder; installs torch, transformers, trl, peft, datasets
yobro doctor                   # shows versions and whether a GPU was found

Without [train] you can still run checks, splits, reports against a served model, gates and monitoring.

Connect your 3B model

There are two ways, and you will usually use both.

1. A local Hugging Face folder, needed for training. It must hold the model in Hugging Face format: config.json, *.safetensors, and the tokenizer files. Training needs these weights.

model base = "/path/to/your-3b-model" { params = "3B" }

2. A running server, for evaluation, judging and monitoring. Anything that speaks the OpenAI chat API works, including llama.cpp's llama-server, Ollama, vLLM and LM Studio.

model local = "http://localhost:8080/v1" { served_model = "your-3b" }

Answers from servers are cached. If you swap the model behind the same address, add version = "..." to the model or run with --fresh.

Or keep the base model local for training but answer evaluation questions through the server:

yobro run project.yobro --serve base=http://localhost:8080/v1#your-3b

If your 3B model only exists as a GGUF file (for example in Ollama or llama.cpp), you can evaluate and monitor it, but you cannot fine-tune a GGUF file. Download the same model's Hugging Face weights for training. After training, export "release@1" -> "gguf" converts the result back to GGUF so llama.cpp or Ollama can run it.

Quick start

yobro new myproject --model /path/to/your-3b-model
cd myproject
yobro check project.yobro            # fix anything marked [error]
yobro preview project.yobro --rows 3 # see rows exactly as the model will see them
yobro run project.yobro
yobro report r1                      # print the saved report

The starter project has only 30 example rows, so the release gate will refuse to ship. That is deliberate. Replace data/*.jsonl with your own data.

Data formats (JSONL, one object per line)

Chat, the main format:

{"id": "t1", "messages": [{"role": "system", "content": "You are a support agent."},
                          {"role": "user", "content": "How do I reset my password?"},
                          {"role": "assistant", "content": "Open Settings, then Security."}],
 "meta": {"topic": "account"}}

Instruction: {"id", "instruction", "input", "output", "meta"}. Preference (for DPO): {"id", "prompt", "chosen", "rejected", "meta"}.

meta is free-form. meta.topic (or any field you name in slices = by("...")) gives per-slice results. meta.source = "generated" marks model-written rows, and generated_share() caps their share. Safety probe rows use meta.expected = "refuse" or "comply".

JSON, CSV and Parquet are also read, as is hf://owner/dataset:split when the datasets package is installed.

Traces for monitoring:

{"trace_id": "t-001", "time": "2026-10-08T10:00:00Z", "model": "support-bot@1",
 "messages": [{"role": "user", "content": "..."}], "output": "...",
 "feedback": {"rating": -1, "correction": "the right answer"}, "meta": {"topic": "billing"}}

OpenTelemetry JSON span exports with gen_ai.* attributes are also read, best effort.

The language

Thirteen statements. Names are bound once; a new version gets a new name.

Rules the compiler enforces:

• A tune without eval does not compile.
• A test set cannot be used as training or evaluation data.
• A report on training data does not compile.
• ship without gated_by does not compile.
• A name cannot be bound twice, and a setting cannot be written twice.
• Unknown settings and misspelled arguments (smaple, early(patiance = 3)) are errors, not silently ignored.
• Training settings are checked before anything runs: a rank of -4, a learning rate of 0 or epochs = "two" do not compile.
• Every gate condition is a comparison that mentions a model; v1.task >= 0.5 (no field) or 1 >= 0 do not compile.
• Release names may only use letters, digits, ., _, - and one @version, because they become folder names on export.

What ship refuses at run time:

• a gate that failed;
• a model whose scores none of the gate's conditions tested (a gate on v2 cannot ship v1, and a row count such as v1.task.n does not count). In Python, built-in rules such as improves("v1") record their model; mark a custom rule with yobro.gate.tests("v1", rule);
• a model that was evaluated through a replacement such as --serve v1=..., since the gate did not see the weights that would ship;
• a dry run (--backend dry): the gate is rehearsed, nothing is recorded;
• a release name that already points at different weights. Shipping the same weights again does nothing; new weights need a new version (support-bot@2).

Every error says where the problem is, what is wrong, and how to fix it.

Checks: schema, min_rows(n), no_duplicates(near), no_overlap_with(...), no_empty_answers, label_conflicts, encoding, personal_data, generated_share(max_share), template_matches(model), truncation(model, max_length). schema, encoding, no_empty_answers, label_conflicts and personal_data always run.

Metrics: exact_match(), contains(), token_f1(), json_valid(keys = [...]), choice() (multiple choice), behaviour() (refuse or comply), judge(model = ..., rubric = "file.md"), user_feedback(), or any Python function f(output, reference[, row]) -> float.

Report settings: general = forgetting_check(data = "general.jsonl", sample = 200) (sample = 200 keeps 200 rows; sample = 10% keeps a tenth), general_metric = choice(), safety = safety_probe(data = "safety.jsonl"), slices = by("topic"), labels = "human_labels.jsonl" (checks judge accuracy; one {"id": ..., "label": 1} per line, where a label is 1/0, true/false or "pass"/"fail"), min_rows, min_slice_rows. Below 5 rows nothing is judged, whatever min_rows says.

A judge rubric can be text or a file. If it looks like a file name (rubrics/policy.md) and the file is missing, that is an error rather than a rubric made of the file name.

monitor ... { sample = 5% } scores a share of traces; sample = 200 scores up to 200. Trace times can be ISO 8601 (offsets are converted to UTC) or Unix timestamps in seconds, milliseconds or nanoseconds.

Gate fields: <model>.<metric>.score | low | high | diff | diff_low, <model>.general.drop, <model>.safety.drop, <model>[slice].score, and every s in r1.slices: .... Intervals are 95% paired bootstrap intervals over test rows. Slices smaller than min_slice_rows are not judged.

See examples/full_lifecycle.yobro for the whole loop, and examples/python_api.py for the same project in Python.

Commands

Everything is stored in .yobro/ next to your file: datasets by hash, runs with manifest.json (data hashes, settings, seed, library versions, hardware), reports (under reports/<project file name>/, so two projects in one folder never overwrite each other), releases (each records the hash of the report that allowed it), caches and review queues. Nothing leaves your machine unless your program sends it to a server.

yobro check and yobro run exit with code 1 when a blocking check fails, and 2 on any other error, so they can gate a CI job.

Answers are cached only for Yobro's own local and served generators, whose identity covers the weights, template and server. Answers from a Python function or your own Generator class are produced fresh each run (set cacheable = True on your class if its ident() changes whenever its answers could).

What is tested and what is not

157 automated tests pass (pytest). 19 of them guard against 15 bugs found by an independent review, and 81 against more than 50 problems found in a stress test and its follow-up verification (all on 8 October 2026); all are fixed. The stress test covered 18,000 randomly broken .yobro files, about 70 hand-written bad programs, 25 malformed data files, data sets of up to 50,000 rows, malformed server replies, odd trace files and every CLI command with bad input. The training tests use a tiny randomly initialised model built locally, on CPU.

Strings in .yobro files treat only \", \\, \n and \t as escapes. On Windows, use forward slashes in paths (C:/models/my-3b) to avoid \n or \t surprises.

Test datasets

DATASET_PROMPT.md is a prompt to give an AI assistant. It produces a full evaluation bundle covering:

• planted data bugs with an answer key
• clean train, val and test sets with a deliberately small slice
• forgetting and safety sets
• judge calibration answers
• structured-output rows
• preference pairs
• six weeks of drifting production traces

yobro bench scores the checks on the bundle; the rest feeds yobro run and yobro monitor.

Development

pip install -e ".[train,dev]"
pytest -q

Licence: Apache-2.0.