Skip to main content

Log as append-only source package

Build Status

Logs as append-only source: write your ML training results in Python without having to worry about crashes. Loading is a breeze: the logs are native Python code. The package supports unstructured data. The data can easily be imported into Jupyter Notebooks or elsewhere.

Installation

To install using pip, use:

pip install laaos

To run the tests, use:

python setup.py test

Append-only source logs

Storing training results as Python dictionaries or JSON files is problematic because the formats are not append-only, which means that you have to rewrite the file every time something changes. (Or you only write results at the end, which does not play well with interruptions or intermediate failures.)

Alternatively, we can simply write the operations that create a structure to a file in an append-only fashion. If the data structure itself is growing and not mutated, this only increases file-size by a constant factor.

The advantage of this library is that the file format is very simple: it's valid Python code.

The only requirement is that you only store primitive types, lists, sets, dicts and immutable types.

Custom wrappers can be added by registering TypeHandlers when creating a Store. See WeakEnumHandler and StrEnumHandler.

Example

from laaos import open_file_store, safe_load

store = open_file_store("test", suffix="", truncate=True)
print("Output file: ", store.uri)

store['losses'] = []
losses = store["losses"]

for i in range(10):
    losses.append(1/(i+1))

store.close()

The resulting file laaos/test.py contains valid Python code:

store = {}
store['losses']=[]
store['losses'].append(1.0)
store['losses'].append(0.5)
store['losses'].append(0.3333333333333333)
store['losses'].append(0.25)
store['losses'].append(0.2)
store['losses'].append(0.16666666666666666)
store['losses'].append(0.14285714285714285)
store['losses'].append(0.125)
store['losses'].append(0.1111111111111111)

It can be loaded either with:

form laaos.test import store

or with the more secure:

safe_load('laaos/test.py')

Slightly more sensible example

from laaos import open_file_store

initial_data = dict(config=dict(dataset="MNIST", learning_rate=1e-4, seed=1337), losses=[])

store = open_file_store("experiment_result", suffix="", initial_data=initial_data)

if store["config"] != initial_data["config"]:
    raise ValueError("Experiment mismatch!")

print("Output file: ", store.uri)

losses = store["losses"]

for i in range(len(losses), 10):
    print("Epoch ", i)
    losses.append(1 / (i + 1))

    if i % 3 == 0:
        raise SystemError("Preemption!")

store.close()

Metadata

Release files for laaos 2.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for laaos 2.1.1
File Size Uploaded Provenance Actions
laaos-2.1.1.tar.gz 7.9 kB Without provenance Details for laaos-2.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for laaos 2.1.1
File Interpreter ABI Platform Actions
laaos-2.1.1-py3-none-any.whl Python 3 none any Details for laaos-2.1.1-py3-none-any.whl

Total release size: 15.7 kB

Release files / laaos-2.1.1.tar.gz

Download URL laaos-2.1.1.tar.gz
Size 7.9 kB
Tags Source
SHA-256 checksum
How to use checksums
e33a9689b72e3e3b2050869df95b8c712ff8ce9842ac475f1392b910bb358a9f
BLAKE2b-256 checksum
How to use checksums
cc76e483e1aefec8ca375485979dc0c06ca58e605c033fbb66e07a8a541bc1f6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.3.0 pkginfo/1.5.0.1 requests/2.25.0 setuptools/52.0.0.post20210125 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.9

Release files / laaos-2.1.1-py3-none-any.whl

Download URL laaos-2.1.1-py3-none-any.whl
Size 7.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8a2d43b924cf58923f7cabc16f52ec87069f26f5d7599fa86e89a69120abbec4
BLAKE2b-256 checksum
How to use checksums
d70b7df7aaa9e6b46bd172faeacaf711cade0959102d82e5459cea95be8b7c57
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.3.0 pkginfo/1.5.0.1 requests/2.25.0 setuptools/52.0.0.post20210125 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.7.9

Security for laaos 2.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page