arrakis-admin

Admin CLI for the Arrakis low-latency timeseries data distribution platform

Administrative tooling for operating Arrakis deployments: manage the Kafka topics backing data partitions, manage the server's static publisher tokens, probe data topics for debugging, and migrate server metadata caches.

Resources

• Documentation
• Source Code
• Issue Tracker

Installation

With pip:

pip install arrakis-admin

Quickstart

Every command supports --help. Commands that modify the broker show what they will do and prompt for confirmation, and support --dry-run to preview changes without applying them.

Manage topics

Create Kafka topics for a publisher's data partitions, with partition assignments queried from the Arrakis server:

➜ arrakis-admin topics create H1-lldd -b kafka1:9092 -s grpc://arrakis:31206

topics to create (2):
  broker: kafka1:9092
  retention: 7d (604_800_000ms), replication factor: 1

  + arrakis-H1-lldd_float32
  + arrakis-H1-lldd_float64

metadata topic (compact, indefinite retention):
  + arrakis-metadata-H1-lldd

Is this ok [y/N]:

Partition assignments can instead be read from a publisher TOML config with --config, and replay-context publishers are supported via --replay-id.

Sync data topics against the channel metadata published on the broker, creating missing topics and removing stale ones:

➜ arrakis-admin topics sync -b kafka1:9092 --dry-run

Remove a publisher's topics (or all Arrakis topics with --all):

➜ arrakis-admin topics remove -b kafka1:9092 -p H1-lldd

Manage static publisher tokens

Manage the static token file of an arrakis-server running with --auth publish --static-token-file (see the arrakis-server authentication documentation). Register a publisher by minting a fresh 256-bit token bound to it:

➜ arrakis-admin tokens add H1-lldd -f /etc/arrakis/tokens.toml
token added for H1-lldd in /etc/arrakis/tokens.toml (fingerprint 3fa4b2c1)
f3a9c2...

The token is printed to stdout, once — deliver it to the publisher's host (e.g. an ARRAKIS_TOKEN= environment file for its service, which --env emits directly) and nowhere else. Edits touch only the target publisher's line, are written atomically, and are validated with the server's own parser, so a hand-maintained file's comments and layout survive; the server hot-reloads the file, so changes need no restart.

Rotate a token in two phases — tokens rotate mints a new token while keeping the old one valid, then tokens prune drops the old one once the publisher is redeployed. tokens remove revokes a publisher, tokens list shows entries as fingerprints (never secrets), and tokens check validates the file.

Probe data topics

A debugging tool that consumes a partition's data topic directly from Kafka and unpacks each message's Arrow payload, reporting block times, per-channel content, and latencies:

➜ arrakis-admin probe -p H1-lldd_float64 -b kafka1:9092 -v
probing topic 'arrakis-H1-lldd_float64' on kafka1:9092 (Ctrl-C to stop)

offset=5231 time=1418063158.000000000 latency=0.412s transport=0.198s size=125.4kB channels=42 samples=131072
  [   0]  float64  16384 samples
  [   1]  float64  16384 samples
  ...

latency is the delay between the GPS block time and receipt; transport is the delay relative to the Kafka message timestamp, separating publisher lag from broker lag.

A channel name can be given instead of a partition ID, in which case its partition is resolved through the Arrakis server and the channel's rows are highlighted in the output:

➜ arrakis-admin probe -c H1:GDS-CALIB_STRAIN -s grpc://arrakis:31206 -b kafka1:9092

Migrate metadata caches

Convert an arrakis-server TOML metadata cache to SQLite, or upgrade an existing SQLite cache to the current schema:

➜ arrakis-admin cache migrate metadata.toml
migrated 12483 channels from metadata.toml to metadata.db

➜ arrakis-admin cache migrate metadata.db
upgraded metadata.db from schema v2 to v3

Clear all data from a cache while preserving its schema:

➜ arrakis-admin cache clear metadata.db

Remove publishers from a cache. The server keeps a publisher whose config file disappears, so moving publishers to another server means taking them out of this one's cache (their channels, the partition counters only they used, their publisher info and sync offsets), with the server stopped:

➜ arrakis-admin cache remove metadata.db H1-CAL-O4E2E_202409 H1-CALCLEAN-O4E2E_202409