Plättli

Readers and writers for the Plättli metric format. There is a fundamental issue in metric logging: reads are columnar (metrics), writes are rows (steps). Plättli solves this by making the format on disk columnar (like parquet) with an optional row-wise "hot log" (like jsonl) for recent writes.

It consists of one file per metric (raw homogeneous array or jsonl), plus a metrics manifest (plattli.json) that describes dtype and indices, a config.json with info about the run, and an optional hot.jsonl during live logging.

At some point I will take the time to write more details about it, but essentially it combines the best of parquet and jsonl while keeping everything very simple.

Install

pip install plattli

Requires Python 3.11+ (tested on 3.11-3.14).

CLI

A tool to convert jsonl (a common adhoc format) to plattli is provided, see

jsonl2plattli --help

By default it writes in-place as <run_dir>/metrics.plattli. With --outdir, it writes <run_name>.plattli into the output tree.

API

from plattli import CompactingWriter, DirectWriter

w = CompactingWriter("/experiments/123456", hotsize=200, config={"lr": 3e-4, "depth": 32})
w.write(loss=1.2)  # First write creates new metric, auto-guesses dtype (float32 here)
w.write(note="ok")  # strings work too. Writes are non-blocking.
w.end_step()  # Increments step by one. Flushes hot log.

w.write(loss=1.3)  # Next write appends
# Not every metric needs to be written every step.
w.write(accuracy=0.73)
w.end_step()

# Data is written ASAP, so almost nothing is lost on crash/preemption.
del w

# If we specify a start step and destination exists,
# existing metrics will be truncated to that and we continue from there.
w = CompactingWriter("/experiments/123456", step=1, hotsize=200, config={"lr": 3e-4, "depth": 32})
w.write(loss=1.1)

# You can also write json, btw (stored as jsonl).
w.write(prediction={"qid": "42096", "answer": "Yes"})

# When finishing cleanly, we can hindsight-optimize the data for faster consumption.
# This writes /experiments/123456/metrics.plattli and removes /experiments/123456/plattli.
w.finish()

# For fast local disks, write directly to columnar files:
d = DirectWriter("/experiments/123456", config={"lr": 3e-4, "depth": 32})
d.write(loss=1.2)
d.end_step()
d.finish()

Note: this library is meant to be called from a single thread. DirectWriter uses threads internally to be non-blocking, and CompactingWriter compacts in the background. Calling end_step from a different thread would lead to silently inconsistent data.

DirectWriter and CompactingWriter also work as context managers: __exit__ flushes pending work, it does not finish() the run, so it stays resumable.

DirectWriter(outdir, step=0, write_threads=16, config="config.json", allow_resume_finalized=False)

• Prepares the writer to write under outdir/plattli, creating the dir and writing the config there.
• If outdir/plattli/plattli.json already exists, all metric files are truncated to step so you can resume a run and overwrite later data safely.
• If outdir/metrics.plattli exists, the constructor refuses to proceed unless allow_resume_finalized=True, which validates the archive paths, unzips into outdir/plattli, and removes the zip.
• write_threads=0 disables background writes.
• config is a dict written to config.json, or a string path (resolved relative to outdir) to symlink config.json to (default: "config.json").
• If the target path does not exist, an empty config is written; pass None to force an empty config.

CompactingWriter(outdir, step=0, *, hotsize, config="config.json", allow_resume_finalized=False)

• Hot mode: writes rows to hot.jsonl and compacts them into columnar files in the background.
• hotsize must be > 0 and is the compaction trigger: once the hot log holds hotsize completed steps, all completed rows are compacted in one background batch.
• Backpressure: writes normally never block on compaction; if the filesystem cannot keep up with the write rate, completed rows accumulate in memory (and in the hot log, so nothing is lost on crash) and batches get bigger. Once the backlog reaches 10x hotsize, end_step blocks until the in-flight batch lands, so memory stays bounded and logging degrades to filesystem speed instead of exhausting RAM.
• config follows the same rules as DirectWriter.
• allow_resume_finalized follows the same rules as DirectWriter.

BulkWriter(outdir, step=0, config="config.json", overwrite=False)

• Buffers a complete run in memory and writes the columnar export on finish().
• Refuses to replace an existing outdir/plattli directory or outdir/metrics.plattli unless overwrite=True.

DirectWriter.write(...)

• Appends each metric at the current step (pass at most one dict or keyword metrics; the dict form is needed for slash-named metrics like detail/thing0).
• Auto-dtype rules:
  • array-like scalars -> use their dtype if supported
  • bool -> jsonl
  • float -> f32
  • int -> i64
  • explicit numpy types (eg np.float64) are taken as-is.
  • everything else -> jsonl
• Force a dtype by casting the value (for example: write(dim=np.float32(128))).
• Only scalar values are supported (including 0-d array-likes).
• Only standard dtypes are supported for now: no bf16, nvfp4, fp8; no complex/composite.

CompactingWriter.write(..., flush=False)

• Appends each metric at the current step (pass at most one dict or keyword metrics).
• flush=True forces a hot.jsonl rewrite without advancing the step (use write(flush=True) to flush only).
• Uses the same auto-dtype rules and scalar restrictions as DirectWriter.write.

end_step()

• Increments step counter by one.
• DirectWriter waits for all previous step writes to finish and checks for errors.
• CompactingWriter flushes the hot row for the current step.

set_config(config)

• Replaces config.json with the provided json-dumpable config.

finish(optimize=True, zip=True)

• DirectWriter flushes writes; CompactingWriter compacts any remaining hot rows and removes hot.jsonl.
• Updates plattli.json.
• If optimize=True:
  • Tightens numeric dtypes (floats -> keep original float width, ints -> smallest fitting int/uint).
  • Converts monotonically spaced indices into {start, stop, step} and removes the .indices file.
  • Writes run_rows (max rows across metrics) into the manifest.
• If zip=True, zips the run folder to <outdir>/metrics.plattli (stored, not compressed).
• When zipping, outdir/plattli is removed after the zip is written.

Reader(path, kind=None)

from plattli import Reader

with Reader("/experiments/123456") as r:
    print(r.metrics())
    print(r.rows("loss"), r.approx_max_rows(), r.when_exported())
    steps, values = r.metric("loss")
    step, value = r.metric("loss", idx=-1)

Callers that already know the exact storage path can pass kind="dir" for a plattli/ directory or kind="zip" for a .plattli archive. This bypasses filesystem discovery, so the path and kind must be trusted.

• Prefers metrics.plattli if present, otherwise reads the plattli/ directory.
• Keeps zip files open until close() (use a with block or call close() manually).
• List all available metric names with metrics().
• Read a metric with one of metric(name, idx=None) -> (indices, values), metric_indices(name), metric_values(name), which return numpy arrays.
• An integer idx (like idx=-1 for the latest value) reads just that one row, without loading the whole column.
• Some useful metadata: config() returns the attached config dict; when_exported() is a timestamp, and rows(name) is the exact row count (not last step!) in the given metric.
• approx_max_rows(nprobes=12) cheaply estimates the row count of the most-frequent metric. Closed index specs and finalized run_rows metadata need no probes; otherwise it checks at most nprobes explicit-index or open numeric columns, distributed across their index cadences. The result becomes exact for stable columnar data when the budget covers every candidate, but may otherwise underestimate; hot rows and open JSONL columns are excluded.
• While the data format is simple, the reader code is a bit more complex because it tolerates corrupt tails, such that it's fine to read plattli's while they are being written.
• Metadata (manifest, config, hot rows, row counts, jsonl values) is cached on first use, while raw data files are read fresh on every call. On a long-lived Reader of a live run, call refresh() to drop the caches and pick up new metrics and hot rows. Zip readers are immutable snapshots.

Aligned reads: table()

table(names, on="step", **selectors) reads several metrics step-aligned, keeping only steps present in every requested metric (inner join on steps). It returns (steps, {name: values}) with all arrays of equal length.

Selectors (see below) select rows of the on column: with the default on="step" they apply to the aligned table itself, while on="some_metric" selects that metric's rows — including by value via vstart/vstop — and the other columns follow. Columns other than on are only read within the selected step window, so zoomed reads stay cheap even when metrics were logged at different cadences.

with Reader("/experiments/123456") as r:
    steps, cols = r.table(["loss", "accuracy"])                    # aligned full read
    steps, cols = r.table(["walltime", "loss"], on="walltime", vstart=10.0, vstop=20.0)
    x, y = cols["walltime"], cols["loss"]

Advanced API topics

Range selectors

Range selectors can be passed to any metric read:

• start/stop read this range of step values. stop is inclusive, like label slicing with pandas .loc.
• vstart/vstop read this range of metric values. vstop is inclusive. Mostly useful for monotonic metrics.
• istart/istop read this range of physical row positions. istop is exclusive and negative positions count from the end, like Python slices (istart=-100 reads the last 100 rows).

These cannot be mixed.

from plattli import Reader

with Reader("/experiments/123456") as r:
    zoomed_loss_steps, zoomed_loss_values = r.metric("loss", start=100, stop=200)
    x_steps, x_values = r.metric("walltime", vstart=10.0, vstop=20.0)

Helpers

• plattli.is_run(path) -> whether the path is a plattli run (a correct folder structure, or a metrics.plattli zipfile).
• plattli.is_run_dir(path) -> whether the folder path contains plattli metrics (be it as subfolder or zipped).
• plattli.resolve_run_dir(path) -> resolved directory that contains plattli.json (returns either path or path/plattli), or None.

Data format

Each run directory contains a plattli/ folder, while the .plattli archive contains the same files at the top level:

run_dir/
  plattli/
    config.json
    plattli.json
    <metric>.indices
    <metric>.<dtype>   # or <metric>.jsonl
    hot.jsonl           # present during live logging if hotsize is enabled
    hot.compacting.jsonl  # transient: rows being compacted right now; unlinked when done
  metrics.plattli

Manifest (plattli.json)

JSON object keyed by metric name, plus metadata keys like run_rows and when_exported:

{
  "loss": {"indices": "indices", "dtype": "f32"},
  "note": {"indices": "indices", "dtype": "jsonl"},
  "run_rows": 1234,
  "when_exported": "2026-01-03T12:34:56Z"
}

Fields:

• indices: "indices", a list of {start, stop, step} segments (canonical), or a single {start, stop, step} (legacy). During live compacting writes, the final segment may omit stop; readers derive it from the value file length.
• dtype: one of f{32,64}, {i,u}{8,16,32,64}, or jsonl.
• monotonic: optional "inc" or "dec" for numeric metrics whose stored values are monotonic; flat-only metrics use "inc".
• run_rows: optional max rows across all metrics (written on finish only).
• when_exported: timestamp updated on manifest writes.

Indices (<metric>.indices)

Raw little-endian uint32 array. Each entry is the step value for that metric write. If optimize=True during finish(), the file may be removed and replaced by a list of {start, stop, step} segments (canonical) or a single {start, stop, step} (legacy) in the manifest. Live compacted runs may omit stop from the final segment until finish() closes it.

Config (config.json)

Arbitrary JSON object (dict), written when a config is provided.

Values (<metric>.<dtype>)

Raw little-endian typed array. One scalar is appended per write call.

JSONL values (<metric>.jsonl)

One JSON value per line:

{"event":"start"}
{"event":"done"}

Metric names and subfolders

Metric names are used as file paths. A slash creates subfolders: detail/thing0 -> detail/thing0.f32. Names must be non-empty relative paths. Absolute paths, backslashes, NULs, empty or ./.. path components, and non-string names are rejected. The following names are reserved: step, run_rows, when_exported, hot, and hot.compacting.