AutoResearch SDK
Standalone Python SDK for autonomous experiment loops.
Try ideas, keep what works, discard what doesn't, never stop.
Inspired by karpathy/autoresearch and the OpenClaw AutoResearch plugin.
Features
- 🔄 Experiment Loop Engine — init → run → log → improve → repeat
- 📊 Statistical Confidence — MAD-based noise floor to distinguish real improvements from noise
- 📝 Ideas Backlog — Failed experiments automatically generate ideas for future exploration
- 📈 State Persistence — JSONL results log, survives restarts
- 🛠️ CLI — Full command-line interface with rich output
- 🐍 Pure Python — Zero external dependencies required (only
rich+click) - 📦 OpenClaw Compatible — Same JSONL format as the OpenClaw AutoResearch plugin
Installation
pip install autoresearch
Quick Start
CLI
# Initialize experiment
autoresearch init --name "optimize-latency" --metric latency_ms --direction lower --unit ms
# Run baseline
autoresearch run -c "./benchmark.sh"
# Log result
autoresearch log -d "Baseline measurement"
# Make changes, run again
autoresearch run -c "./benchmark.sh"
autoresearch log -d "Added caching layer"
# Discard bad idea
autoresearch run -c "./benchmark.sh"
autoresearch log -s discard -d "Thread pool didn't help" -i "Try async I/O instead"
# Check status
autoresearch status
Python API
from autoresearch import AutoResearch
# Initialize
ar = AutoResearch(
name="optimize-latency",
metric="latency_ms",
direction="lower", # lower is better
unit="ms",
)
ar.init()
# Run baseline
result = ar.run("./benchmark.sh")
print(f"Passed: {result.passed}, Duration: {result.duration_seconds:.1f}s")
ar.log("Baseline measurement")
# Run experiment
result = ar.run("./benchmark.sh")
if result.passed:
ar.log("Added Redis caching")
else:
ar.log("Crashed", status="crash")
# Discard with idea
result = ar.run("./benchmark.sh")
ar.discard_with_idea("Thread pool overhead", "Try event loop instead")
# Check confidence
print(f"Confidence: {ar.confidence}")
print(f"Best: {ar.best_metric} vs Baseline: {ar.baseline_metric}")
# Export results
import json
print(json.dumps(ar.get_status(), indent=2))
How It Works
The Loop
SCAN → HYPOTHESIS → EXPERIMENT → MEASURE → INTEGRATE → repeat
- SCAN — Identify optimization opportunities
- HYPOTHESIS — Form a testable hypothesis
- EXPERIMENT — Make the change and run the benchmark
- MEASURE — Parse metrics, compare to baseline
- INTEGRATE — Keep if better, discard if worse, log idea for future
Confidence Scoring
AutoResearch uses Median Absolute Deviation (MAD) as the noise floor:
| Confidence | Meaning |
|---|---|
| ≥ 2.0x | Improvement is likely real ✅ |
| 1.0-2.0x | Above noise but marginal ⚠️ |
| < 1.0x | Within noise, re-run to confirm ❓ |
This prevents false positives from natural variance.
Output Format
Your benchmark script should output METRIC name=value lines:
#!/bin/bash
set -euo pipefail
# Run your benchmark
result=$(./my_benchmark)
# Output metrics
echo "METRIC latency_ms=$result"
echo "METRIC throughput_qps=$(echo '10000 / ' $result | bc)"
Persistence
All results are stored in autoresearch.results.jsonl:
{"type": "config", "name": "optimize-latency", "metricName": "latency_ms", "metricUnit": "ms", "bestDirection": "lower"}
{"run": 1, "commit": "abc1234", "metric": 142.5, "metrics": {"latency_ms": 142.5}, "status": "keep", "baseline": true, "description": "Baseline", "timestamp": 1713888000, "segment": 0, "confidence": null}
Ideas Backlog
Discarded experiments append to autoresearch.ideas.md:
- Try async I/O instead of thread pool
- Consider connection pooling for database
- Pre-compute lookup table for hot path
Architecture
autoresearch/
├── src/autoresearch/
│ ├── __init__.py # Public API
│ ├── core/
│ │ ├── engine.py # Experiment loop engine
│ │ ├── confidence.py # Statistical confidence (MAD)
│ │ └── metrics.py # METRIC name=value parser
│ ├── cli.py # Rich CLI interface
│ ├── utils/
│ └── metrics/
├── tests/
│ └── test_autoresearch.py
├── pyproject.toml
└── README.md
Comparison with OpenClaw AutoResearch Plugin
| Feature | OpenClaw Plugin | Python SDK |
|---|---|---|
| Experiment loop | ✅ (via tools) | ✅ (API + CLI) |
| Confidence scoring | ✅ (MAD) | ✅ (MAD, same algorithm) |
| Ideas backlog | ✅ | ✅ |
| State persistence | ✅ (JSONL) | ✅ (JSONL, compatible) |
| Git integration | ✅ (auto checkout) | ⚠️ (commit tracking only) |
| Session management | ✅ (OpenClaw sessions) | ❌ (standalone) |
| Rich output | ✅ | ✅ (click + rich) |
| OpenClaw required | ✅ | ❌ |
| Python importable | ❌ | ✅ |
License
MIT — Use it however you want.
Author
Jose Manuel Sabarís García (@llllJokerllll)
Acknowledgments
- karpathy/autoresearch — Original concept
- OpenClaw AutoResearch Plugin — Confidence algorithm and JSONL format
Metadata
Release files for alfred-autoresearch 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| alfred_autoresearch-1.1.0.tar.gz | 14.1 kB | Without provenance | Details for alfred_autoresearch-1.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| alfred_autoresearch-1.1.0-py3-none-any.whl | Python 3 | none | any | Details for alfred_autoresearch-1.1.0-py3-none-any.whl |
Total release size: 29.7 kB
Release files / alfred_autoresearch-1.1.0.tar.gz
| Download URL | alfred_autoresearch-1.1.0.tar.gz |
|---|---|
| Size | 14.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
804e143443bcd7e08e85435430c00e49e2e1ce2c07e6c0b6ff6f6d739354794a
|
|
BLAKE2b-256 checksum How to use checksums |
2c966b5a009fc58f319d9032983c1afd37000ee53779ad8603e6b314d74a0353
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.4
|
Release files / alfred_autoresearch-1.1.0-py3-none-any.whl
| Download URL | alfred_autoresearch-1.1.0-py3-none-any.whl |
|---|---|
| Size | 15.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
92d358aac422f61504a91b00fca6ce828e2ef68506fce337f27ec7672502ae52
|
|
BLAKE2b-256 checksum How to use checksums |
75a2ed9419d6f58205d669ea7e26203db7c002192e10ae104115ca29002e7da7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.4
|
Security for alfred-autoresearch 1.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.