skald 0.3.0

📟 a simple and efficient experiment logger for Python 🐍

Skáld

📟 a simple and efficient experiment logger for Python 🐍

“Ósnotr maðr þykkist allt vita.” - An unwise man thinks he knows all.

Skáld - An Old Norse word for a poet, usually applied to a Norwegian or Icelandic court poet or bard of the period from the 9th century to the 13th. Skaldic verse is marked by its elaborate patterns of metre, rhyme, and alliteration, and by its use of kennings.

---

📃 Table of Contents

• 💡 Motivation
• 👀 Concepts
  • 📂 Logging Structure
• 📦 Installation
• 🚀 Usage

---

💡 Motivation

During my PhD, I tried different Experiment/Metrics loggers including:

• TensorBoard
• MLFlow
• Weights and Biases
• DVCLive
• other logging solutions from deep learning and continual learning frameworks (mostly stdout and csv loggers)

While those are quite mature logging solutions, which often offer beautiful dashboards, I was looking for something light-weight, local and file-based as DVCLive, but with a more ergonomic and tidy [^1] structure of the logs for simpler consumption and analysis.

The latest of my workflows before Skáld involved using a mixture of DVCLive and a custom logger from FACIL [^2] in combination with log crawling CLI scripts and analysis and visualizations performed in jupyter notebooks 🤨

Another problem I faced with those solutions is that all of them offered a single "step" identifier for each logged metric, which is not sufficient for deep learning use-cases outside the conventional epoch-based training loops.

Because I like building python packages and I felt the need to tidy my experiment logs, I created skald as a small side project. I hope that some people find some enjoyment in using the package too ❤️

👀 Concepts

Skáld is an experiment logger, that offers a standardized logging structure and interface to log different aspects of an experiment including:

• parameters|arguments - meta information and variables, that don't change during the experiment
• metrics|scalars - single-valued, numeric variables that change during the experiment
• artifacts - additional result files like images or plots

Each metric has a unique name and a user-defined set of id variables, that identify the value of the metric at a certain step.

While a stateful version of Skáld is planned, that updates an id/step variables through some manual call (often in a Callback). The first version is very explicit and requires every id to be logged in each call.

📂 Logging Structure

Logs of metrics will be represented by tidy dataframes that are stored as readable .csv or more space efficient .parquet files.

To save space, parameters will not be included in these dataframes, but in a separate file (params.yaml) by default.

Artifacts will be stored in a separate sub-directory (artifacts/ by default).

📦 Installation

The package can be installed with:

pip install skald

🧑‍💻 to install a development environment (which you need if you want to work on the package, instead of just using the package), cd into the project's root directory and call:

poetry install --sync --compile

🚀 Usage

The API of Skáld is very similar to DVCLive and other loggers.

A basic example:

from skald import Logger

# get experiment parameters from CLI arguments, parameter files ...
params: dict = { ... }

# instanciate a logger with a certain run name
with Logger("test-run-1") as logger:
    logger.log_params(params)
    # experiment logic
    metric: float = evaluate(model)
    logger.log_metric("accuracy", metric)

---

📄 References

[^1]: H. Wickham, “Tidy Data,” Journal of Statistical Software, vol. 59, pp. 1–23, Sep. 2014, doi: 10.18637/jss.v059.i10. [^2]: M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. van de Weijer, “Class-incremental learning: survey and performance evaluation on image classification.” arXiv, Oct. 11, 2022. doi: 10.48550/arXiv.2010.15277.