ablate 0.2.3

Creating a Report

To create your first Report, define one or more experiment sources. For example, the built in Mock can be used to simulate runs:

from ablate.sources import Mock

source = Mock(
    grid={"model": ["vgg", "resnet"], "lr": [0.01, 0.001]},
    num_seeds=2,
)

Each run in the mock source has accuracy, f1, and loss metrics, along with a seed parameter as well as the manually defined parameters model and lr. Next, the runs can be loaded and processed using functional-style queries to e.g., sort by accuracy, group by seed, aggregate the results by mean, and finally collect all results into a single list:

from ablate.queries import Metric, Param, Query

runs = (
    Query(source.load())
    .sort(Metric("accuracy", direction="max"))
    .groupdiff(Param("seed"))
    .aggregate("mean")
    .all()
)

Now that the runs are loaded and processed, a Report comprising multiple blocks can be created to structure the content:

from ablate import Report
from ablate.blocks import H1, Table

report = Report(runs)
report.add(H1("Model Performance"))
report.add(
    Table(
        columns=[
            Param("model", label="Model"),
            Param("lr", label="Learning Rate"),
            Metric("accuracy", direction="max", label="Accuracy"),
            Metric("f1", direction="max", label="F1 Score"),
            Metric("loss", direction="min", label="Loss"),
        ]
    )
)

Finally, the report can be exported to a desired format such as Markdown:

from ablate.exporters import Markdown

Markdown().export(report)

This will produce a report.md file with the following content:

# Model Performance

| Model   |   Learning Rate |   Accuracy |   F1 Score |    Loss |
|:--------|----------------:|-----------:|-----------:|--------:|
| resnet  |           0.01  |    0.94285 |    0.90655 | 0.0847  |
| vgg     |           0.01  |    0.92435 |    0.8813  | 0.0895  |
| resnet  |           0.001 |    0.9262  |    0.8849  | 0.0743  |
| vgg     |           0.001 |    0.92745 |    0.90875 | 0.08115 |

Combining Sources

To compose multiple sources, they can be added together using the + operator as they represent lists of Run objects:

runs1 = Mock(...).load()
runs2 = Mock(...).load()

all_runs = runs1 + runs2 # combines both sources into a single list of runs

Selector Expressions

ablate selectors are lightweight expressions that access attributes of experiment runs, such as parameters, metrics, or IDs. They support standard Python comparison operators and can be composed using logical operators to define complex query logic:

accuracy = Metric("accuracy", direction="max")
loss = Metric("loss", direction="min")

runs = (
    Query(source.load())
    .filter((accuracy > 0.9) & (loss < 0.1))
    .all()
)

Selectors return callable predicates, so they can be used in any query operation that requires a condition. All standard comparisons are supported: ==, !=, <, <=, >, >=. Logical operators & (and), | (or), and ~ (not) can be used to combine expressions:

from ablate.queries import Id

select = (Param("model") == "resnet") | (Param("lr") < 0.001) # select resnet or LR below 0.001

exclude = ~(Id() == "run-42") # exclude a specific run by ID

runs = Query(source.load()).filter(select & exclude).all()

Functional Queries

ablate queries are functionally pure such that intermediate results are not modified and can be reused:

runs = Mock(...).load()

sorted_runs = Query(runs).sort(Metric("accuracy", direction="max"))

filtered_runs = sorted_runs.filter(Metric("accuracy", direction="max") > 0.9)

sorted_runs.all() # still contains all runs sorted by accuracy
filtered_runs.all() # only contains runs with accuracy > 0.9

Composing Reports

By default, ablate reports populate blocks based on the global list of runs passed to the report during initialization. To create more complex reports, blocks can be populated with a custom list of runs using the runs parameter:

report = Report(sorted_runs.all())
report.add(H1("Report with Sorted Runs and Filtered Runs"))
report.add(H2("Sorted Runs"))
report.add(
    Table(
        columns=[
            Param("model", label="Model"),
            Param("lr", label="Learning Rate"),
            Metric("accuracy", direction="max", label="Accuracy"),
        ]
    )
)
report.add(H2("Filtered Runs"))
report.add(
    Table(
        runs = filtered_runs.all(), # use filtered runs only for this block
        columns=[
            Param("model", label="Model"),
            Param("lr", label="Learning Rate"),
            Metric("accuracy", direction="max", label="Accuracy"),
        ]
    )
)