Arrakis Schema Specification

This repository defines the schema for all data, metadata and API requests for the arrakis LIGO data distribution system.

Schemas are defined in one of two places:

• endpoints/: Endpoints served from Arrakis server
• publication/: Data published into Kafka

Endpoints

The Arrakis server responds to API requests corresponding to the four main actions exposed by the client API:

• stream: endpoints/stream
• describe: endpoints/describe
• find: endpoints/find
• count: endpoints/count

as well as two actions which aid in publication:

• partition: endpoints/partition
• publish: endpoints/publish

All API requests are done in a two-stage approach by first sending an Arrow Flight descriptor to the server, returning back a Flight info object which contains the request and the server to contact, contained within a Flight ticket. This ticket is then sent to receive back the expected payload with a specific Arrow flight schema dependent on the request, serialized in the Arrow .

The Flight descriptors sent to the server in the first stage are all specified here as JSON packets which are UTF-8-encoded, using the command variant of the Flight descriptor, which can be used to specify any application-specific command.

The Flight descriptor schemas are described within each endpoint in descriptor.json, while the payload schemas are described via schema.txt. In addition, a generic descriptor specification for all endpoints is described in endpoints/descriptor.json.

Versioning

The request schema set is versioned by the package version (arrakis_schema.SCHEMA_VERSION), which follows semver: releases that change the schemas bump the minor version when additive and the major version when breaking (while pre-1.0, the minor position carries breaking changes, per the usual caret convention). Clients stamp this version into the request envelope (the optional top-level version field) so servers can report actionable errors for requests newer than they support.

Publication

Publication is done by first registering a publisher via the publish endpoint with a publisher ID. If authorized to do so, the server will send a response with connection details to connect via Kafka to publish data.

The data is published via Kafka with the schema described in publication/schema.txt.

Packaging

The schema set is packaged twice from this repository, for the two client languages:

• Python: the arrakis-schema wheel, with the schema documents as package data.
• Rust: the arrakis-schema crate, embedding the same documents at compile time and mirroring the Python API (SCHEMA_VERSION, load_schema, parse_version, supports).

The version is single-sourced: Cargo.toml holds the one literal (Cargo cannot read an external version) and the Python packaging derives its version from it, so one release tag publishes both. The version-comparison test vectors (arrakis_schema/tests/version_vectors.json) are shared by both test suites so the two implementations are pinned to identical semantics. A nix flake provides a development environment covering both packagings (nix develop).

Usage

Python

The generic Flight descriptor schema is described within each endpoint in {endpoint}.json. In addition, a generic descriptor specification for all endpoints is described in descriptor.json.

from arrakis_schema import load_schema

schema = load_schema("count.json")