SIB (Self-Improving Benchmark)

A framework for building self-improving AI agents that autonomously refine their performance on benchmark tasks.

Installation

pip install sib-agent

Usage

from sib import Generation, GenerationLog, ScoreTracker

# Track generations in a self-improvement loop
log = GenerationLog()

gen = log.new_generation(config={"model": "claude-sonnet", "temperature": 0.7})
# ... run your agent ...
gen.finish(result={"accuracy": 0.82})

gen = log.new_generation(config={"model": "claude-sonnet", "temperature": 0.5})
# ... run improved agent ...
gen.finish(result={"accuracy": 0.91})

# Find the best generation
best = log.best("accuracy")
print(best.generation_id, best.result)  # 1 {'accuracy': 0.91}

# Save and reload logs
log.save("run_log.json")
log = GenerationLog.load("run_log.json")

# Track scores across generations
tracker = ScoreTracker()
tracker.record("accuracy", 0.82)
tracker.record("accuracy", 0.91)

summary = tracker.summarise("accuracy")
print(summary.improvement)   # 0.09
print(summary.is_improving)  # True

License

MIT