amos

Version
Status
Documentation
Tools
Compatibility
Stats

What is amos?

Automated Modeling for Open Scholarship. Like the Rocinante's mechanic in The Expanse, amos handles the dirty jobs of a data science project: cleaning, describing, splitting, preprocessing, modeling, evaluating, and drawing. It is designed for academic research, where a result has to be explained and reproduced.

You describe a study in a plain settings file (or a Python dict). amos builds the workflow with chrisjen, runs it on your data, compares every combination of the methods you list, and records everything needed to reproduce the result.

import sklearn.datasets

import amos

cancer = sklearn.datasets.load_breast_cancer(as_frame = True)
settings = {
    "general": {"seed": 43},
    "cancer_project": {"cancer_workers": "analyst, critic"},
    "analyst": {
        "design": "experiment",
        "criterion": "roc_auc",
        "steps": "split, scale, model",
        "split_techniques": "stratified",
        "scale_techniques": "standard, min_max",
        "model_techniques": "baseline, logit",
    },
    "critic": {"techniques": "scorecard, confusion"},
}
project = amos.Project.create(settings, item = cancer)
print(project.result)
# Dataset(rows=569, columns=31, label='target', task='classify', train=426, test=143, model=LogisticRegression)

That project split the data, tried four combinations of scaling and models, kept the one with the best ROC AUC on the test rows, and scored it. The comparison of all four is stored in a table:

table = project.result.tables["analyst_comparison"]
print(list(table.columns))
# ['rank', 'score', 'roc_auc']
print(table["roc_auc"].iloc[-1])
# 0.5

Why use amos?

Intuitive

amos is accessible to researchers at all levels. It uses one vocabulary for every stage of a project. A project is made of workers (its stages), which use techniques (the actual methods), sometimes organized in steps. The settings file only names things, so it doubles as a readable description of your methods:

The stages are only a convention: any technique can be used in any worker, and you can name your workers whatever you like.

Reproducible

Open scholarship means that others can check your work. amos helps by:

• Passing one seed to every tool that takes a random_state, so a projectngives the same result every time.
• Applying a project to a copy of your data, so running it twice gives thensame answer and your data is never changed.
• Keeping a history of every technique that was applied, with the exact tool and parameters it used.
• Isolating training and testing data. Every scaler, encoder, imputer, and sampler is fitted to the training rows and then applied to the test rows, so nothing leaks from the test set.
• All settings and parameters can be implemented through a single settings file that will produce the same results on any computer.

Universal

amos wraps the major Python data science packages behind one interface, so you can compare their methods side by side without learning each package's quirks:

Robust and Transparent

A result that depends on one arbitrary choice of preprocessing or model is fragile. The experiment design tries every combination of the techniques you list for each step and reports how each one did, so readers can see that your conclusions are robust (or not). The critic's scorecard puts every combination side by side, on every metric, in a table ready for a paper (as csv, Markdown, LaTeX, HTML, Word, or an image).

Name the columns that identify groups of people or places as groups, and amos keeps them out of the model's features while it uses them to check fairness across groups, to add fixed effects and cluster standard errors, and to keep each group in one set when it splits the data. Any step can include none, which tests whether a technique helps at all.

Extensible

Every technique is a small Python class that is added to a library as soon as it is defined, so it can be named in settings right away. To wrap a tool that amos does not include, name its import path:

transformer = amos.Transformer(
    name = "yeo_johnson", contents = "sklearn.preprocessing.PowerTransformer")
print(transformer.name)
# yeo_johnson

Getting started

Requirements

amos requires Python 3.11 or later. It runs on Linux, macOS, and Windows. It is built on chrisjen, numpy, pandas, and scikit-learn, which are installed automatically.

Installation

To install amos, use pip:

pip install amos

The other packages that amos wraps are optional. A technique imports its package only when it is used, and tells you which extra to install if the package is missing. Install them all (except TabPFN) with pip install amos[all], or choose:

Usage

Describe your study

Most users describe a study in a settings file. This is the Wisconsin breast cancer study from above, with every stage, as an ini file:

[general]
seed = 43
label = target

[cancer_project]
cancer_workers = wrangler, explorer, analyst, critic, artist

[wrangler]
techniques = drop_duplicates, drop_constant

[explorer]
techniques = summarize, missing_values, label_balance

[analyst]
design = experiment
criterion = roc_auc
steps = split, scale, sample, model
split_techniques = stratified
scale_techniques = standard, min_max
sample_techniques = none, smote
model_techniques = logit, random_forest

[critic]
techniques = scorecard, confusion, permutation_importance

[artist]
techniques = roc_curve, importance_plot

[random_forest_parameters]
n_estimators = 100

[roc_curve_parameters]
title = How well the best model separates the classes

[importance_plot_parameters]
title = The ten features that matter most
limit = 10

The "general" section names the label (the column that models predict) and the seed. The project section lists the workers. In the "analyst", each {step}_techniques setting lists alternatives, and the experiment design tries every combination: 2 scalers, 2 samplers, and 2 models make 8 combinations. Parameters for any technique go in a {technique}_parameters section.

Run it

Pass the settings and the data (a pandas.DataFrame, a path to a data file, or a scikit-learn dataset) to Project.create:

project = amos.Project.create("cancer.ini", item = cancer, id = "first_run")
result = project.result
print(result.metrics["roc_auc"] > 0.95)
# True
print(sorted(result.figures))
# ['importance_plot', 'roc_curve']

The artist drew two figures, titled by the roc_curve_parameters and importance_plot_parameters sections. The ROC curve shows how well the best combination tells the two kinds of tumors apart on the test rows it never learned from. The importance plot shows how much its score drops when each feature is shuffled, which works for any model:

The result includes a Dataset, the fitted model, its predictions, metrics, tables, figures, and the history of every technique.

Compare every branch

The critic's scorecard compares every branch of the analysis: each combination of techniques that the experiment tried, ranked by the criterion, with every standard metric for the task computed from that branch's own predictions on the test rows. It can be saved as a csv file, a Markdown table, a Word document, or an image:

scorecard = project.scorecard
print(len(scorecard.table), list(scorecard.table.columns[:5]))
# 8 ['rank', 'split', 'scale', 'sample', 'model']
scorecard.to_csv("scorecard.csv")
scorecard.to_markdown("scorecard.md")
scorecard.to_word("scorecard.docx")
scorecard.to_image("scorecard.png")

This is the scorecard of the study, saved as an image. The best branch is shaded:

Export the results

export saves everything needed to report and reproduce the results in a folder named for the run, including the scorecard in all four formats:

folder = project.export()
print(sorted(path.name for path in folder.iterdir()))
# ['environment.json', 'figures', 'history.json', 'metrics.csv', 'predictions.csv', 'report.txt', 'scorecard.csv', 'scorecard.docx', 'scorecard.md', 'scorecard.png', 'settings.json', 'tables']

There is much more to amos, including hyperparameter searches, statistical inference with statsmodels, every technique and its parameters, and how to write your own. See the documentation, especially the tutorial, the advanced user guide, and the recipes.

Contributing

Contributors are always welcome. Feel free to grab an issue to work on or make a suggested improvement. If you wish to contribute, please read the Contribution Guide and Code of Conduct.

Similar Projects

• scikit-learn: its Pipeline and GridSearchCV combine preprocessing and models and search over hyperparameters. amos uses scikit-learn throughout, and adds named columns, supervised encoders, samplers, comparisons of whole combinations of methods, and settings files.
• PyCaret: a low-code library that compares many models with a few function calls. It automates more choices for you, while amos asks you to list your methods so that they can be reported.
• Kedro: a framework for reproducible data pipelines, with more structure for large, long-running projects.
• MLflow and DVC: track experiments, data, and models over time. They work well alongside amos.

License

Use of this repository is authorized under the Apache Software License 2.0.