Skip to main content

"Log as append-only source" logger

Project description

Log as append-only source package

Build Status

Logs as append-only source: write your ML training results in Python without having to worry about crashes. Loading is a breeze: the logs are native Python code. The package supports unstructured data. The data can easily be imported into Jupyter Notebooks or elsewhere.


To install using pip, use:

pip install laaos

To run the tests, use:

python test

Append-only source logs

Storing training results as Python dictionaries or JSON files is problematic because the formats are not append-only, which means that you have to rewrite the file every time something changes. (Or you only write results at the end, which does not play well with interruptions or intermediate failures.)

Alternatively, we can simply write the operations that create a structure to a file in an append-only fashion. If the data structure itself is growing and not mutated, this only increases file-size by a constant factor.

The advantage of this library is that the file format is very simple: it's valid Python code.

The only requirement is that you only store primitive types, lists, sets, dicts and immutable types.

Custom wrappers can be added by registering TypeHandlers when creating a Store. See WeakEnumHandler and StrEnumHandler.


from laaos import open_file_store, safe_load

store = open_file_store("test", suffix="", truncate=True)
print("Output file: ", store.uri)

store['losses'] = []
losses = store["losses"]

for i in range(10):


The resulting file laaos/ contains valid Python code:

store = {}

It can be loaded either with:

form laaos.test import store

or with the more secure:


Slightly more sensible example

from laaos import open_file_store

initial_data = dict(config=dict(dataset="MNIST", learning_rate=1e-4, seed=1337), losses=[])

store = open_file_store("experiment_result", suffix="", initial_data=initial_data)

if store["config"] != initial_data["config"]:
    raise ValueError("Experiment mismatch!")

print("Output file: ", store.uri)

losses = store["losses"]

for i in range(len(losses), 10):
    print("Epoch ", i)
    losses.append(1 / (i + 1))

    if i % 3 == 0:
        raise SystemError("Preemption!")


Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for laaos, version 2.0.4
Filename, size File type Python version Upload date Hashes
Filename, size laaos-2.0.4-py3-none-any.whl (7.3 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size laaos-2.0.4.tar.gz (7.3 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page