ExperimentLane

An evidence-driven experimentation engine for AI agents. Record experiments, compare them against baselines, and get a deterministic decision about what to change next — one variable at a time.

The Python package mirrors the TypeScript implementation (@talocode/experimentlane) 1:1.

What is it

ExperimentLane tracks content and agent experiments as receipts: a baseline, a run, and the platform-exposed metrics that come back. It compares them, names a winner, picks the strongest and weakest signals, and tells the agent exactly what to change next while everything else stays constant.

Why it exists

Most agents ship a variation, see a number, and guess. That is how B-tests quietly become A/B/C/D tests and nothing is ever attributable.

ExperimentLane encodes the discipline most agents are missing: change exactly one independent variable at a time. It is a local, deterministic, no-frills engine any agent can embed, so experiments stay honest, small, and comparable.

Open engine first. The hosted power can come later — the same schema is shaped for the Talocode API surface.

Install

pip install talocode-experimentlane

Python 3.8+, no runtime dependencies.

Quickstart

from talocode_experimentlane import ExperimentLane, JsonStore

lane = ExperimentLane(JsonStore(".experimentlane"))
lane.init()

# 1. Define a lane for a platform's exposed metrics
shorts = lane.create_lane(name="youtube-shorts", platform="youtube_short")

# 2. Record a baseline (the control asset's platform-exposed metrics)
base = lane.create_baseline(
    lane_id=shorts["id"],
    metrics={"views": 480, "avg_view_percent": 75, "shares": 10, "subscribers": 2},
)

# 3. Create the experiment — exactly one variable changes
asset = lane.create_asset(lane_id=shorts["id"], title="reliability-short-2")
exp = lane.create_experiment(
    lane_id=shorts["id"], asset_id=asset["id"], baseline_id=base["id"], variable="hook"
)

# 4. Record the receipt after the run
lane.add_receipt(
    experiment_id=exp["id"],
    metrics={"views": 624, "avg_view_percent": 87, "shares": 14, "subscribers": 3},
)

# 5. Compare and get one deterministic decision
result = lane.compare(experiment_id=exp["id"])
print(result["comparison"]["winner"])      # "experiment"
print(result["decision"]["line"]["nextExperiment"])  # "change only <weakest signal>"
print(result["decision"]["line"]["everythingElse"])  # "hold constant"

Canonical import: from talocode_experimentlane import ExperimentLane, JsonStore.

Auth / env

The local store needs no API key. Optional environment variable:

• EXPERIMENTLANE_DIR — override the default data directory (default .experimentlane).

API surface

Everything is comparable-metrics only — absent metrics are "unavailable", never a failure.

Platform schemas

Platforms (youtube_short, tiktok, instagram_reel, x_post, generic) map their exposed metrics. shares_per_view is derived from views + shares when both are present. A receipt may only contain metrics listed for its lane platform — unknown keys raise a clear error.

Guardrail codes

CLI

experimentlane lane create --name my-shorts --platform youtube_short
experimentlane receipt add --experiment <id> --views 624 --shares 14
experimentlane compare --experiment <id>
experimentlane next --experiment <id> --propose hook,duration   # guardrail check

Data lives in .experimentlane (override with EXPERIMENTLANE_DIR).

Related packages

Sibling installs for the Talocode ecosystem:

Talocode ecosystem

More: github.com/talocode · talocode.site · docs.talocode.site

License

MIT © Talocode.