arrakis-admin

Admin CLI for the Arrakis low-latency timeseries data distribution platform

Administrative tooling for operating Arrakis deployments: manage the Kafka topics backing data partitions, probe data topics for debugging, and migrate server metadata caches.

Resources

• Documentation
• Source Code
• Issue Tracker

Installation

With pip:

pip install arrakis-admin

Quickstart

Every command supports --help. Commands that modify the broker show what they will do and prompt for confirmation, and support --dry-run to preview changes without applying them.

Manage topics

Create Kafka topics for a publisher's data partitions, with partition assignments queried from the Arrakis server:

➜ arrakis-admin topics create H1-lldd -b kafka1:9092 -s grpc://arrakis:31206

topics to create (2):
  broker: kafka1:9092
  retention: 7d (604_800_000ms), replication factor: 1

  + arrakis-H1-lldd_float32
  + arrakis-H1-lldd_float64

metadata topic (compact, indefinite retention):
  + arrakis-metadata-H1-lldd

Is this ok [y/N]:

Partition assignments can instead be read from a publisher TOML config with --config, and replay-context publishers are supported via --replay-id.

Sync data topics against the channel metadata published on the broker, creating missing topics and removing stale ones:

➜ arrakis-admin topics sync -b kafka1:9092 --dry-run

Remove a publisher's topics (or all Arrakis topics with --all):

➜ arrakis-admin topics remove -b kafka1:9092 -p H1-lldd

Probe data topics

A debugging tool that consumes a partition's data topic directly from Kafka and unpacks each message's Arrow payload, reporting block times, per-channel content, and latencies:

➜ arrakis-admin probe -p H1-lldd_float64 -b kafka1:9092 -v
probing topic 'arrakis-H1-lldd_float64' on kafka1:9092 (Ctrl-C to stop)

offset=5231 time=1418063158.000000000 latency=0.412s transport=0.198s size=125.4kB channels=42 samples=131072
  [   0]  float64  16384 samples
  [   1]  float64  16384 samples
  ...

latency is the delay between the GPS block time and receipt; transport is the delay relative to the Kafka message timestamp, separating publisher lag from broker lag.

A channel name can be given instead of a partition ID, in which case its partition is resolved through the Arrakis server and the channel's rows are highlighted in the output:

➜ arrakis-admin probe -c H1:GDS-CALIB_STRAIN -s grpc://arrakis:31206 -b kafka1:9092

Migrate metadata caches

Convert an arrakis-server TOML metadata cache to SQLite, or upgrade an existing SQLite cache to the current schema:

➜ arrakis-admin cache migrate metadata.toml
migrated 12483 channels from metadata.toml to metadata.db

➜ arrakis-admin cache migrate metadata.db
upgraded metadata.db from schema v2 to v3

Clear all data from a cache while preserving its schema:

➜ arrakis-admin cache clear metadata.db