arrakis-admin

Admin CLI for the Arrakis low-latency timeseries data distribution platform

Administrative tooling for operating Arrakis deployments: manage the Kafka topics backing data partitions, manage the server's static publisher tokens, probe data topics for debugging, and migrate server metadata caches.

Resources

• Documentation
• Source Code
• Issue Tracker

Installation

With pip:

pip install arrakis-admin

Quickstart

Every command supports --help. Commands that modify the broker show what they will do and prompt for confirmation, and support --dry-run to preview changes without applying them.

Manage topics

Create Kafka topics for a publisher's data partitions, with partition assignments queried from the Arrakis server:

➜ arrakis-admin topics create H1-lldd -b kafka1:9092 -s grpc://arrakis:31206

topics to create (2):
  broker: kafka1:9092
  retention: 7d (604_800_000ms), replication factor: 1

  + arrakis-H1-lldd_float32
  + arrakis-H1-lldd_float64

metadata topic (compact, indefinite retention):
  + arrakis-metadata-H1-lldd

Is this ok [y/N]:

Partition assignments can instead be read from a publisher TOML config with --config, and replay-context publishers are supported via --replay-id.

Sync data topics against the channel metadata published on the broker, creating missing topics and removing stale ones:

➜ arrakis-admin topics sync -b kafka1:9092 --dry-run

Remove a publisher's topics (or all Arrakis topics with --all):

➜ arrakis-admin topics remove -b kafka1:9092 -p H1-lldd

Manage static publisher tokens

Manage the static token file of an arrakis-server running with --auth publish --static-token-file (see the arrakis-server authentication documentation). Register a publisher by minting a fresh 256-bit token bound to it:

➜ arrakis-admin tokens add H1-lldd -f /etc/arrakis/tokens.toml
token added for H1-lldd in /etc/arrakis/tokens.toml (fingerprint 3fa4b2c1)
f3a9c2...

The token is printed to stdout, once — deliver it to the publisher's host (e.g. an ARRAKIS_TOKEN= environment file for its service, which --env emits directly) and nowhere else. Edits touch only the target publisher's line, are written atomically, and are validated with the server's own parser, so a hand-maintained file's comments and layout survive; the server hot-reloads the file, so changes need no restart.

Rotate a token in two phases — tokens rotate mints a new token while keeping the old one valid, then tokens prune drops the old one once the publisher is redeployed. tokens remove revokes a publisher, tokens list shows entries as fingerprints (never secrets), and tokens check validates the file.

Probe data topics

A debugging tool that consumes a partition's data topic directly from Kafka and unpacks each message's Arrow payload, reporting block times, per-channel content, and latencies:

➜ arrakis-admin probe -p H1-lldd_float64 -b kafka1:9092 -v
probing topic 'arrakis-H1-lldd_float64' on kafka1:9092 (Ctrl-C to stop)

offset=5231 time=1418063158.000000000 latency=0.412s transport=0.198s size=125.4kB channels=42 samples=131072
  [   0]  float64  16384 samples
  [   1]  float64  16384 samples
  ...

latency is the delay between the GPS block time and receipt; transport is the delay relative to the Kafka message timestamp, separating publisher lag from broker lag.

A channel name can be given instead of a partition ID, in which case its partition is resolved through the Arrakis server and the channel's rows are highlighted in the output:

➜ arrakis-admin probe -c H1:GDS-CALIB_STRAIN -s grpc://arrakis:31206 -b kafka1:9092

Migrate metadata caches

Convert an arrakis-server TOML metadata cache to SQLite, or upgrade an existing SQLite cache to the current schema:

➜ arrakis-admin cache migrate metadata.toml
migrated 12483 channels from metadata.toml to metadata.db

➜ arrakis-admin cache migrate metadata.db
upgraded metadata.db from schema v2 to v3

Clear all data from a cache while preserving its schema:

➜ arrakis-admin cache clear metadata.db