lairs

A read/write dataset client for the Layers format, built on didactic.

Tutorial · Guides · Concepts · API · Development

lairs is a Python client for reading and writing data in the Layers format. It downloads pub.layers.* records from ATProto Personal Data Servers, validates them against models generated from the Layers lexicons, holds them in memory or in a local content-addressed store, and exposes them through a datasets-like API with tooling for the modalities Layers relates: audio, video, and time-series signals. On the write side it constructs records, uploads media blobs, and publishes records in bulk to the authenticated user's own repository, with the local store doubling as schema-aware version control.

lairs is built on didactic, which is built on panproto. Structured values in lairs are didactic models built programmatically from the pub.layers.* ATProto lexicons.

Installation

lairs can be installed from pypi:

pip install lairs                 # core

The core install does not include integration dependencies. Each integration is an optional extra, discovered at runtime through entry points, so importing lairs never imports an integration's dependency.

pip install "lairs[hf]"           # HuggingFace datasets and Hub
pip install "lairs[torch]"        # PyTorch exporter
pip install "lairs[audio]"        # audio decoding
pip install "lairs[conllu]"       # the CoNLL-U codec

Usage

import lairs
from lairs.atproto import PdsClient

# read a corpus straight from the public Layers PDS
with PdsClient("https://repo.layers.pub") as client:
    corpus = lairs.load_corpus(
        "at://did:plc:myu6umexofclib2tvwn23gsc/pub.layers.corpus.corpus/92cfd1034bcef513c2796962",
        source="pds",
        pds_client=client,
    )

print(len(corpus.expressions))
print(corpus.expressions[0].text)

The lairs command vendors lexicons, regenerates models, and pulls, materialises, publishes, and inspects corpora:

lairs gen --check          # fail if the committed models drift from the lexicons
lairs pull did:plc:abc     # ingest an account's records into a local repository
lairs materialize <uri>    # build Arrow and Parquet views
lairs publish --repo ... --revision v0.1 --to did:plc:abc   # dry-run plan by default

Documentation

The documentation is available at layers.pub/lairs/. It can be built locally with:

uv run --group docs mkdocs serve

Development

lairs uses uv along with ruff, ty, and pytest for development.

uv sync
uv run ruff format --check lairs tests
uv run ruff check lairs tests
uv run ty check
uv run pytest                    # unit tests only
uv run pytest --run-integration  # include integration tests (docker, network, extras)

See CONTRIBUTING.md for the full contribution guide and the Development section of the documentation for testing, code generation, and the release process. All participants are expected to follow the Code of Conduct.

Changelog

Notable changes are recorded in CHANGELOG.md.

License

lairs is released under the MIT License.

Acknowledgments

lairs was developed with substantial assistance from Claude Code.