zyexp

English · 简体中文

A lightweight, local experiment record manager for coding agents.

Let your agent write and run experiment code in the way the task requires. zyexp gives that code a consistent way to organize runs and save parameters, metrics, checkpoints, and results. A later analysis can read those records without having to interpret a new collection of ad hoc logs.

The package includes an Agent Skill that explains how to structure experiments, record observations, validate outputs, and read results. Your agent writes the recording calls into the script; the script saves observations as the experiment runs. zyexp does not execute experiments or impose a training loop.

Python 3.10+. The only direct runtime dependency is jsonschema; the base installation requires neither PyTorch, scikit-learn, NumPy, nor zyplib. Records stay in local files, with no tracking server or account required.

Get started with your agent

Install zyexp in the Python environment your agent uses for the experiment:

pip install zyexp

Then ask the agent to read the bundled guide:

zyexp skill

This prints the Skill and the installed paths to its reference files. An agent with shell and file access can read them as needed. No separate Skill download is required.

For example, give your agent this task:

Compare logistic regression and random forest on Iris using the same data split. First run zyexp skill and follow its guidance. Write and execute the experiment, recording the parameters, data split identity, metrics, and saved models with zyexp. Validate the Run, then compare the models by reading the saved results. Report the Run directory and the evidence behind your comparison.

The agent chooses how to load data, build models, and run the comparison. It uses zyexp to save the facts needed to inspect the experiment later. Repeats, cross-validation, and parameter searches remain ordinary code, not a separate scheduling language.

Use the libraries your experiment needs. Optional installation shortcuts are available:

pip install 'zyexp[sklearn]'  # scikit-learn
pip install 'zyexp[torch]'    # PyTorch and torchvision

These extras are not needed to read or validate saved results.

What an experiment leaves behind

A comparison might use one Run for the execution and one Trial per model:

experiments/classifier-comparison/runs/run-001/
├── state.json
├── records.jsonl
├── report.json
└── models/
    ├── state.json
    ├── records.jsonl
    ├── logistic-regression/
    │   ├── state.json
    │   ├── records.jsonl
    │   └── checkpoints/
    └── random-forest/
        ├── state.json
        ├── records.jsonl
        └── checkpoints/

A Run is one execution. A Trial is a unit with its own metrics, checkpoints, result, and failure status. A Group, such as models above, organizes related Trials and may contain other Groups. Epoch and batch positions are metric coordinates inside a Trial, not directories.

The experiment script can record:

These facts must be submitted by the experiment code; zyexp does not infer data splits or scientific conclusions. Normal context exit marks a node completed. An escaping exception marks it failed and propagates to the caller.

Read and analyze results

After the experiment exits, the same agent or a new session can inspect its Run without rerunning training:

zyexp inspect <RUN_DIR>
zyexp records <RUN_DIR> --node models/logistic-regression --type metric
zyexp validate <RUN_DIR> --verify-artifacts

inspect prints the report tree, including final results and checkpoint metadata. records reads the observations. Validation checks structure and referenced files; --verify-artifacts also checks artifact and checkpoint hashes. It does not judge experimental design or the agent's interpretation.

Python readers work on the saved files too:

from zyexp import iter_metrics, load_run

run_dir = 'experiments/classifier-comparison/runs/run-001'
report = load_run(run_dir)
models = report['run']['children'][0]
for trial in models['children']:
    print(trial['name'], trial['result']['objectives'])

for observation in iter_metrics(run_dir, node_path='models/logistic-regression'):
    print(observation['name'], observation['coordinates'], observation['value'])

state.json and report.json hold results; records.jsonl holds submitted observations. Live metric recorders provide within-Trial aggregate views. After process exit, read raw metrics to reconstruct the views needed for analysis. The agent computes cross-Trial comparisons and may record its conclusions explicitly.

How scripts integrate

Recording calls sit beside the experiment's own execution code. This runnable example uses illustrative values, not a trained model:

from zyexp import open_run

with open_run(root='experiments', experiment='recording-demo', run_name='run-001') as run:
    run.record_hyper_params({'learning_rate': 0.01})
    with run.trial('training') as trial:
        metrics = trial.metrics(levels=('epoch',))
        metrics.define('validation/loss')
        for epoch, loss in enumerate([0.8, 0.5, 0.3]):
            # In an experiment, use the value computed by your evaluation code.
            metrics.record('validation/loss', loss, epoch=epoch)
        trial.record_result(objectives={'validation/loss': loss})

print(run.run_dir)

Choose a new run_name for another execution. Training frameworks, optimization order, and checkpoint serialization/loading stay in caller code. zyexp owns the paths and retention policy for managed Top-K checkpoints.

Current limits:

• One writer per Run; concurrent or distributed writing is not supported.
• An existing Run cannot be reopened for continued writing after a process restart.
• Each metric name has one coordinate depth. Compute non-decomposable metrics such as F1 or AUROC over the appropriate data; do not average batch scores.
• zyexp repair <RUN_DIR> marks abandoned running nodes interrupted and rebuilds the report. It does not resume execution or recover missing observations.

Guides and examples

• zyexp skill — installed Agent guide with paths to references for recording, checkpoints, profiling, and inspection.
• zyexp sourcecode — absolute path to the installed Python source, for checking an API question against the version your experiment uses. The Skill includes a source map.
• Python API guide — API examples for metrics, Top-K, artifacts, relations, framework-owned loops, and data profiling.
• sklearn Iris example — fit, evaluate, save a model, and record its lineage.
• PyTorch MNIST example — caller-owned training loop, batch/epoch metrics, and Top-K checkpoints.

Both READMEs, the Python API guide, and example scripts are included in the source distribution. The installed wheel includes the Agent Skill and its references.

License: AGPL-3.0-or-later.