arrakis-schema 0.5.0

Schemas for the Arrakis API

Arrakis Schema Specification

This repository defines the schema for all data, metadata and API requests for the arrakis LIGO data distribution system.

Schemas are defined in one of two places:

• endpoints/: Endpoints served from Arrakis server
• publication/: Data published into Kafka

Endpoints

The Arrakis server responds to API requests corresponding to the four main actions exposed by the client API:

• stream: endpoints/stream
• describe: endpoints/describe
• find: endpoints/find
• count: endpoints/count

as well as two actions which aid in publication:

• partition: endpoints/partition
• publish: endpoints/publish

All API requests are done in a two-stage approach by first sending an Arrow Flight descriptor to the server, returning back a Flight info object which contains the request and the server to contact, contained within a Flight ticket. This ticket is then sent to receive back the expected payload with a specific Arrow flight schema dependent on the request, serialized in the Arrow .

The Flight descriptors sent to the server in the first stage are all specified here as JSON packets which are UTF-8-encoded, using the command variant of the Flight descriptor, which can be used to specify any application-specific command.

The Flight descriptor schemas are described within each endpoint in descriptor.json, while the payload schemas are described via schema.txt. In addition, a generic descriptor specification for all endpoints is described in endpoints/descriptor.json.

Versioning

The request schema set is versioned by the package version (arrakis_schema.SCHEMA_VERSION), which follows semver: releases that change the schemas bump the minor version when additive and the major version when breaking (while pre-1.0, the minor position carries breaking changes, per the usual caret convention). Clients stamp this version into the request envelope (the optional top-level version field) so servers can report actionable errors for requests newer than they support.

Publication

Publication is done by first registering a publisher via the publish endpoint with a publisher ID. If authorized to do so, the server will send a response with connection details to connect via Kafka to publish data.

The data is published via Kafka with the schema described in publication/schema.txt.

Packaging

The schema set is packaged twice from this repository, for the two client languages:

• Python: the arrakis-schema wheel, with the schema documents as package data.
• Rust: the arrakis-schema crate, embedding the same documents at compile time and mirroring the Python API (SCHEMA_VERSION, load_schema, parse_version, supports).

The version is single-sourced: Cargo.toml holds the one literal (Cargo cannot read an external version) and the Python packaging derives its version from it, so one release tag publishes both. The version-comparison test vectors (arrakis_schema/tests/version_vectors.json) are shared by both test suites so the two implementations are pinned to identical semantics. A nix flake provides a development environment covering both packagings (nix develop).

Usage

Python

The generic Flight descriptor schema is described within each endpoint in {endpoint}.json. In addition, a generic descriptor specification for all endpoints is described in descriptor.json.

from arrakis_schema import load_schema

schema = load_schema("count.json")