top_level: true description: The GAR WebSocket protocol in full: handshake, subscribing, filtering, binary and JSON encodings, batching, heartbeats.

Generic Active Records Protocol Documentation

Overview

The GAR protocol is a WebSocket-based messaging system built on top of TCP. It is used for streaming and snapshot-style data delivery and operates over a single bi-directional socket connection. When establishing a connection, clients must specify the subprotocol "gar-protocol":

Sec-WebSocket-Protocol: gar-protocol

Encoding

The server supports two formats for message payloads: binary and JSON.

• JSON (opcode 0x01) is recommended for human-readable debugging and platform-agnostic integration.
• Binary (opcode 0x02) is compact and efficient but platform-dependent due to endianness, sizing, and alignment.

If the first byte of the initial Introduction message is {, then JSON mode is assumed regardless of the WebSocket opcode.

Binary Format

• Each message begins with a message_type enumeration value.
• Immediately followed by a binary-encoded struct corresponding to that type.
• Must conform to platform-specific alignment and byte order.
• Should only be used when both ends are aware of the exact schema and platform.

JSON Format

JSON messages are structured with a "message_type" string and a corresponding "value" object.

Example:

{
  "message_type": "Subscribe",
  "value": {
    "subscription_mode": "Snapshot",
    "nagle_interval": 0,
    "name": "S1",
    "key_id": 0,
    "topic_id": 0,
    "class_list": ["Underlier"],
    "key_filter": null,
    "topic_filter": null,
    "exclude_key_filter": null,
    "exclude_topic_filter": null
  }
}

This format is more portable and ideal for most clients.

Framing

Each websocket frame contains a single message - a json {} or binary object.

Session Lifecycle

1. Introduction must always be the first message. It specifies protocol version, heartbeat expectations, and optional settings such as unique_key_and_record_updates.
2. Clients may then Subscribe to topics and keys or Publish messages.
3. Clients must send Heartbeat messages periodically to keep the connection alive. If none is received within the specified timeout, the server will disconnect the client.
4. The first heartbeat has a 10x grace period to allow for slow client setup.
5. All topics and keys must be explicitly introduced before publishing record messages.
6. IDs 0 are considered invalid for both topic_id and key_id.
7. Enumeration of topics and keys is not globally synchronized between client and server. Each side must track their own ID mappings.

Introduction Options

The introduction message supports the following optional fields:

• unique_key_and_record_updates (bool, default false): If true, only one record update will be sent per key/record even if multiple subscriptions apply. This only affects streaming record updates — snapshot data is still sent for each subscription. Note that active_subscription_group will not be reliable when using this option.
• disconnect_on_write_failure (bool, default false): bail-out-first writers. The server closes the connection on the first client message with a failed write — after reporting it, as a non-recoverable Error carrying failed_writes (delivered even when the active_ownership msg_ref is 0, which is otherwise log-only). Every write that succeeded stays landed; this is not transactional — a client that needs atomicity batches its writes so each batch is safe under partial failure. Meant for critical processes that would rather restart than continue past a refused write, and for integration tests; subscriptions on the same connection go down with it, so a process that both watches and writes should keep them on separate connections. Through a proxy, a failure detected upstream reaches the client as a relayed failed_writes Error and closes the client's connection the same way, but only when the proxy learned of it — i.e. with a nonzero msg_ref, since the proxy's own upstream connection does not carry the flag. pygar: GARClient(..., disconnect_on_write_failure=True); jsgar: the disconnectOnWriteFailure constructor argument. Both libraries already treat a non-recoverable Error as fatal (exit_code 1, stop). The REST analogue is ?fail=true (see Write-failure replies).

Record Updates

Record updates flow both ways through the same messages: a client writes with them, and a server streams post-snapshot updates to a subscriber with them too — only the initial snapshot (and nagle-coalesced ticks) travel as batch updates. Anything that must ride with every streamed record therefore has to live on these messages as well as on batch_update.

The GAR protocol supports record updates via:

Binary encoding

• FixedLengthRecordUpdate: record_id directly followed by the fixed-size record data.
• VariableLengthRecordUpdate: serialized binary_record_update struct with embedded length field.

JSON encoding

• Encoded using json_record_update with clear field names and value.
• Non-finite floats. JSON has no literal for them, so every JSON emitter (REST, the JSON wire protocol, --print) spells an infinite float/double as 1e999 / -1e999 — a literal that overflows to ±infinity in every reader (Python, JavaScript, jq, the trs reader itself) — and NaN as null (a NaN result is absence at the record boundary). On input the trs reader accepts 1e999, and the words infinity / -Infinity (case-insensitive), as ±infinity.

Example:

{
  "message_type": "JSONRecordUpdate",
  "value": {
    "record_id": {"key_id": 1, "topic_id": 22},
    "value": 0.3
  }
}

Augment bypass (assume_cached_subscriptions_cover)

A subscription with assume_cached_subscriptions_cover: true asserts that the proxy serving it has cached subscriptions covering every record it asks for. That proxy then skips this subscription's augment differencing entirely and serves it — derived topics included — from its cache, removing the poller-thread set-algebra the snapshot blocks on. The assertion is per subscription: an operator turns it on for the few heavy subscriptions the cache was provisioned around, and every other subscription on the same proxy keeps augmenting normally. No proxy config gates it — the assertion is the caller's own, and a wrong one under-serves only the subscription that made it; a plain (non-proxy) server has no cache to bypass and ignores the flag. Fail-open: anything the cache does not cover is silently absent or null on that subscription. REST equivalent ?assume-cached=true, trsgar --assume-cached. See augment_bypass_spec.

Record tick_time

A subscription with include_tick_time: true receives each record's tick_time: the owning server's wall-clock nanoseconds at the record's last write. It advances on every write, including one whose value did not change, so it answers "is the writer alive", not "did the value change". A value-unchanged write is throttled (no update); a flagged subscription instead receives a RecordTick (record_tick {record_id, tick_time}) — stamp only, no value — which clients that don't track stamps can ignore. Proxies forward the flag upstream and relay the origin's stamp unchanged, never their own receipt time; a relayed tick never triggers recomputation.

• json_record_update / binary_record_update carry it as tick_time (JSON omits the member, binary sends 0, when the subscription didn't opt in).
• batch_update.key_major_update.tick_times is an array parallel to topics; JSON carries a "tick_times" object beside "topics" with the same <topic_id> member names. Empty/absent otherwise.

Off by default — payloads keep their size and shape. See features/server_tick_time.

Record diagnostics

A derived-topic record that has no value carries a structured diagnostic saying why — a derived_diagnostic {reason, missing_inputs[], text}. reason is one of InputMissing, KeyCastMissed, Suppressed (the null class: plain absence with a cause) or ComputeError / InputError (the error class: the function raised or failed — text is the stripped exception message — or one of its inputs is in error). Errors are poison: an InputError consumer stays in error whatever its optional/default argument policy, and ?? coalesces nulls only. missing_inputs names each absent or erroring input — its argument index and declared parameter name, the input topic's path and the key holding it — so a client can chase a null or an error upstream, record by record, to its root cause.

• Every subscription receives error-class diagnostics with no opt-in; the null-class reasons only reach subscriptions that set include_diagnostics: true.
• The wire carries them as RecordDiagnostic (record_diagnostic {record_id, diagnostic}), sent after the batches that introduce the record's key during a snapshot, and streamed on change. A diagnostic replaces the record's value: a value→diagnostic transition arrives as the ordinary DeleteRecord followed by the RecordDiagnostic; a value arriving supersedes the diagnostic.
• A proxy computes derived topics itself, so its diagnostics are its own computation's; a client that takes derived records from a server (trsgar, pygar, jsgar) applies the relayed diagnostic verbatim.

See null_reasons_spec.

Density

The density field controls how records are grouped in JSON output. The default density is KeyMajor.

• KeyMajor (default): { "key": "key1", "topics": { "topic1": <record>, "topic2": ... } }
• TopicMajor: { "topic": "topic1", "keys": { "key1": <record>, "key2": ... } }
• RecordMajor: { "key": "key1", "topic": "topic1", "value": <record> } (one record per object)

Batch updates

Multiple updates can be sent together in a batch. Initial snapshots after subscribing are always sent in one or more batch.

Records are sent in key-major format. Keys are introduced along with their class or class_list. Key names and classes may be omitted when keys have already been introduced.

Binary format uses the batch_update struct.

JSON format is similar, except the records within each "topics" object are JSON objects with topic_id numbers as member names: { "<topic_id>": record_value, ... }

Example:

{
  "message_type": "BatchUpdate",
  "value": {
    "default_class": "A",
    "keys": [
      {
        "key_id": 1,
        "name": "key1",
        "topics": {
          "20": 10
        }
      },
      {
        "key_id": 2,
        "name": "key2",
        "class": "B",
        "topics": {
          "21": 20
        }
      },
      {
        "key_id": 3,
        "name": "key3",
        "classes": ["A", "B", "C"],
        "topics": {
          "20": 30,
          "21": 31,
          "22": 32
        }
      }
    ]
  }
}

Subscriptions

Clients may initiate:

• Snapshot subscriptions (one-time data snapshot),

• Streaming subscriptions (continuous updates),

• UpdatesOnly subscriptions (continuous updates without an initial snapshot),

• DeleteKeys (delete all matching keys and their records)

• DeleteRecords (delete all matching records)

DeleteKeys and DeleteRecords are one-shot: the server deletes the set matching the subscription's filters at subscribe time (resolving key cross-references — e.g. a referenced parent key is removed even while a child still points at it) and does not act on keys/records created later, so a delete subscription left active will not re-delete a future write under the same filter. The deletion is processed incrementally as a snapshot — scheduled async on the server's poller, not applied synchronously when the subscribe is received — and is complete only once the server sends "status": "Finished" (see Subscription Status). Do not send Unsubscribe (or close the connection) before that Finished status arrives: doing so cancels the in-flight snapshot, leaving the delete partial or unstarted. The REST API runs these deletes internally and blocks the HTTP response until the snapshot finishes, so REST DELETE callers get the completed guarantee for free — a direct WebSocket client must wait for Finished itself.

Subscription filters can be applied using:

• Direct key_id or topic_id,

• _class filters to only keys of this class,

• Regular expressions via key_filter or topic_filter,

• Regular expressions via exclude_key_filter or exclude_topic_filter — the subtractive pair. Each is ANDed with its positive counterpart, so key_filter + exclude_key_filter reads as "match this, minus that", and either exclude works on its own against everything the rest of the filter admitted. This is the shape to reach for when the set you want is easier to say by what it is not; pygar's subscribe(exclude_key_filter=…, exclude_topic_filter=…) and jsgar's equivalents pass them straight through, as do REST's exclude-key-filter/exclude-topic-filter,

• referencing_class_list to include back-referenced keys and records (see below),

• join_class_list + join_mode to join a second class list against class_list's key set: Intersect accepts only keys carrying a class from both lists ("keys of X that are also Y"); LeftJoin leaves the key set unchanged while the join classes contribute their topics on accepted keys that carry them. In both modes the join classes' topic closures become servable. A non-empty join_class_list without a join_mode is a subscribe error. Membership follows class changes on streaming subscriptions: a key gaining its join class is introduced when it enters the scope, and a per-class delete of the join class is relayed as usual — inferring exclusion from it is the client's job.

Heavy topics (inferred for comprehensions, array-typed values and class/comprehension-fed derivations, or stated with {heavy_override: true}) and explicit-only topics ({explicit: true}) are left out of every subscription unless include_heavy / include_explicit is set. Sweeps (class, regex, subscribe-all) simply omit them; a topic_id_list that names one without the flag is a subscribe error — the server replies with a non-recoverable Error stating the topic, why it is excluded, and the flag to set, exactly as for an unknown topic id — rather than coming up Streaming minus a topic the client would wait on forever.

Subscriptions may be changed by sending another subscription messages using the same name. After the ProcessingSnapshot subscription status message is returned, all subsequent updates will be for the new subscription settings.

To cancel an active stream, send an Unsubscribe message with the subscription name. Alternatively, unsubscribe by sending another subscription with "subscription_mode": "Unsubscribed".

If a topic filter regex is provided, it is checked against all relative topic paths. So for example, if the working namespace is "root_ns::sub_ns::w_ns", the topic "root_ns::sub_ns::tn" will be matched by any of the topic filters "tn", "sub_ns::tn", or "root_ns::sub_ns::tn". If the working namespace was "root_ns::other_ns", only "sub_ns::tn" and "root_ns::sub_ns::tn" would match.

The working namespace defaults to the working namespace sent in the introduction message. It may be overridden in the subscription request using the working_namespace field.

Clients may assign an arbitrary numeric subscription_group id for the purpose of isolating callbacks. An active_subscription_group message is sent before record or key updates to denote the subscription_group. Many updates may follow a single active_subscription_group message, and the initial active subscription_group is assumed to be 0. subscription_group ids do not need to be unique across subscriptions.

Large subscriptions may overwhelm the client, server, or network. snapshot_size_limit may be set to throttle the initial snapshot response. Each time the limit is hit, the server sends back a SubscriptionStatus message with status = "NeedsContinue" and stops sending additional snapshot data. The snapshot processing will continue once the client sends a SubscribeContinue message, until another limit is hit or the snapshot finishes with status = Streaming or status = Finished.

Clients do not need to continue with the snapshot if snapshot_size_limit is hit. They may choose to continue, pause for some time or indefinitely, or unsubscribe.

The same protocol paces bulk publications: a publisher using snapshot_size_limit pauses at each NeedsContinue until the receiver replies SubscribeContinue. A plain server replies once it has applied the batch; a proxy additionally defers the reply while any publishing server's write backlog is above a pacing threshold, so a bulk publication cannot overflow a slow server's write queue. The pacing knobs are g::deployment::ProxyPerformanceTuning records (a per-proxy override on the proxy's own key, else the global key's defaults); if a backlog never drains, the threshold doubles on a timeout until pacing yields, so the publisher is throttled but never wedged.

Referencing Class List

The referencing_class_list subscription field enables back-reference expansion. When populated:

• Keys and records are scanned and included as usual based on other filter criteria
• Records of included topics that directly reference any included keys through classes in the referencing_class_list are also included
• Keys of referencing records are included, and recursively along with any of their referencing keys and records
• Once included, referencing keys and records are NOT removed from the subscription if references are later dropped
• Referencing topics must be included in the subscription for this to take effect

This is useful for following key references across classes, such as including all subscriptions that reference a given server.

Echo

Clients may send an EchoRequest message containing a msg_ref (message reference number). The server will respond with an EchoResponse containing the same msg_ref. This can be used to confirm that all prior writes have been processed before continuing.

In proxy chains, the echo propagates end-to-end: the proxy forwards the echo to its upstream server and only responds to the client once the upstream echo response arrives. This guarantees that all prior writes — including record updates and deletes — have been fully processed through the entire chain by the time the client receives the echo response. The REST API uses this mechanism internally, so HTTP POST and DELETE responses also carry this guarantee.

Compare-exchange

A CompareExchange message applies a write only if the record's current value is what the client expects, so two writers racing on one record cannot silently overwrite each other. It carries a record_id, an inline msg_ref correlation id, a test (the expected current value) and a value (what to write on a match). The server replies with a CompareExchangeResult echoing that msg_ref and carrying success; on a mismatch it also returns the record's current value, so a client can retry against fresh state without a separate read.

Null means absent, on both sides. A missing record reads as null, so:

• test: null matches a record that is missing — this is create-if-absent. (For a topic whose type declares a default, a missing record instead reads as that default, so the matching test is the default value.)
• value: null is an empty value, so a successful compare deletes the record. This holds for every topic type, including fixed-length numeric ones — a null value is never stored as a zero.

Binary clients express the same two things with a zero-length test / value byte array.

Together those give compare-and-delete: test = the value you saw, value = null. That makes a record a lock-free "consume exactly once" flag — several readers that all observed X each attempt the delete against X, exactly one succeeds, and the losers are told so. Nothing is claimed, so a reader that dies mid-operation strands nothing. The same operation is available over REST via atomic-compare-exchange=true.

Compare-exchange propagates through proxy chains: the proxy forwards it upstream and relays the result back to the originating client. The REST equivalent forwards the same way, so both surfaces are atomic through a proxy.

Subscription Status

A SubscriptionStatus message will be sent to delineate the lifetime of the subscription.

• "status": "ProcessingSnapshot" is sent at the start of processing. Topic, Key, and record updates will follow.

• "status": "NeedsContinue" is sent if snapshot_size_limit is hit; see Subscriptions above.

• "status": "Streaming" is sent for streaming subscriptions once the snapshot has been fully processed. Additional messages are live updates.

• "status": "Finished" is sent once the snapshot has been fully processed and the subscription mode is non-streaming. It is also sent after an unsubscribe has been processed.

Heartbeats

Clients specify a heartbeat timeout interval within their introduction. If the server does not receive a Heartbeat message within this interval, it sends the client an Error message ("Heartbeat timeout") and closes the connection gracefully — a client that still reads (e.g. one whose sending stalled) receives the error before the close; one that doesn't is dropped after one further timeout interval.

The server also specifies its timeout interval, and sends regular heartbeats. Clients may choose to respect missed heartbeats and disconnect.

Heartbeats are expected to be sent at twice the frequency of the timeout interval. e.g. with timeout of 4 seconds, heartbeats are sent every 2 seconds.

The send cadence is fixed-rate (anchored to the previous scheduled instant, not to when the handler last ran), but a missed-tick backlog never replays. The send loop is single-threaded, so a slow client that backs up the send buffer can stall it for many intervals; rather than firing a burst of catch-up heartbeats onto the already-slow client when it resumes, the sender snaps the next beat forward past the current time, collapsing the backlog to a single heartbeat one interval out.

Synchronizing without sleep

Client code (and test harnesses) frequently needs to wait until some condition holds — the server is accepting connections, prior writes have been applied, a subscription is live, a derived value has settled — before taking the next step. Do not approximate these waits with a fixed sleep. A blind delay is simultaneously too short (it races under load) and too long (it wastes time on a fast path), and it converts a missed event into a silent, intermittent failure. The protocol exposes an explicit signal for each case; synchronize on the signal and the timing becomes deterministic.

• Server is ready to accept connections. Launch the server with --server-ready-out FILE; it writes the bound {protocol, url} endpoints to FILE once it is listening. Poll until the file is non-empty, then read the URL — never sleep waiting for a port to bind.

• All prior writes have been processed. Send an EchoRequest and wait for the matching EchoResponse (see Echo). In a proxy chain the echo is end-to-end, so its response guarantees every prior record update/delete has propagated through the whole chain. The REST API uses this internally, so an HTTP POST/DELETE response already carries the guarantee — no follow-up wait is needed.

• A streaming subscription is live. After subscribing, wait for the "status": "Streaming" SubscriptionStatus message (see Subscription Status); it is sent once the initial snapshot has been fully processed and means subsequent messages are live updates. This matters most for live-only topics (history: none): such a record is pushed only to subscribers that are already streaming when it is produced and is never retained, so a subscriber still in ProcessingSnapshot when the event fires misses it permanently. Trigger the event only after the Streaming status arrives.

• A derived value has settled before you act on it. Poll the value (over REST, or via your subscription stream) until it holds the expected value, rather than sleeping a guessed interval.

• A non-streaming snapshot subscription has completed — most importantly a DeleteKeys / DeleteRecords delete. Wait for the "status": "Finished" SubscriptionStatus before treating the result as done or sending Unsubscribe: the snapshot (and any delete it performs) is processed incrementally, so an Unsubscribe issued synchronously after the subscribe cancels the in-flight work. The REST API waits for Finished for you — an HTTP DELETE returns only after the delete has fully processed — so this caveat applies to direct WebSocket clients (e.g. browser/Node subscribe callers), not REST callers.

For ad-hoc polling, use a bounded retry with a small interval and a hard timeout (e.g. check every 100 ms up to a few seconds) so a stuck condition surfaces as a clear timeout rather than a hang. The only place a fixed delay is defensible is a negative check — confirming that something does not happen within a window.

Termination

To gracefully close a session, clients should send a Logoff message. However, clean TCP connection closure is also acceptable.

If the server encounters an issue, it may respond with an Error message prior to disconnection.

The server sends Logoff when it is shutting down (trsgar --send-shutdown), tearing every subscription down at once. That is an orderly end, but at the socket it is indistinguishable from the connection breaking, and both stop the client the same way. GARClient.server_logoff records which it was, so a long-running client can exit 0 through a planned restart instead of reporting a failure on every one — the alternative is a supervisor alert per restart, which buries the real failures. Read it after the client stops; register_logoff_handler is the callback form. jsgar carries the same flag as serverLogoff, reset per connection since jsgar reconnects — a Logoff from an earlier server must not make a later broken connection look orderly.

Both libraries write their log lines as YYYY-MM-DD HH:MM:SS.mmm LEVEL:<name>:<message> in local time — the shape trsexec and the C++ core use — so a service log carrying several of them reads as one stream rather than as interleaved timezones and formats.

Failed writes

Recoverable write failures do not disconnect. Every failed write from processing one client message batches into ONE recoverable Error message whose failed_writes array identifies each failure — a shape-scoped reason enum (record_write/key_write union), the addressed ids and names, malformed (server-classified: a request defect vs a server-state condition), per-failure detail, and origin (the server or proxy that detected the failure; proxies relay upstream failures verbatim). Delivery requires a nonzero msg_ref on the active_ownership marker — with msg_ref 0 the failure is logged server-side only. pygar (register_write_errors_handler) and jsgar (registerWriteErrorsHandler) deliver the array to a callback as (failed_writes, message, msg_ref). A connection introduced with disconnect_on_write_failure instead receives the batch as a non-recoverable Error (msg_ref 0 included) and is then closed — see Introduction Options. Pinned end-to-end by ipc/write_errors_test.bash.

The plain error callback cannot tell the two apart. pygar's register_error_handler and jsgar's registerErrorHandler are text-only conveniences: their wrappers pass message and drop the rest of the Error, including the recoverable flag that says whether the connection survives. Both clients read that flag internally to decide whether to keep going, so the information reaches the library and stops at the wrapper. A client that must distinguish "this write was rejected" from "this connection is finished" registers for the raw message instead — register_handler("Error", …) / registerHandler('Error', …), whose callback receives the whole Error — or, when the interest is specifically which writes failed, uses the write-errors callback above.

Key teardown is per class

All key teardown is delivered as KeyUpdate messages carrying deleted_class — one message per class. When a server deletes a key entirely, it sends one deleted_class KeyUpdate per class the receiver was shown (each with an empty remaining class_list), followed by KeyIdRetired — a connection-scoped bookkeeping message meaning only "the sender's introduction of this key id is gone; drop your mapping for it", never a data teardown. There is no whole-key teardown broadcast: a subscriber's view of a key ends when its last subscribed class is deleted. (Distributed servers can each hold a subset of a key's classes, so "all classes gone" on one server is not global truth — per-class messages are the only view-independent form.)

DeleteKeyCommand is the client→server command requesting whole-key deletion at the receiving server; servers never broadcast it.

Example Session (Client Perspective)

Sent: {"message_type": "Introduction", "value": { "version": 650269, "heartbeat_timeout_interval": 3000, "user": "jonh" }}
Received: {"message_type": "Introduction", "value": { "version": 650269, "heartbeat_timeout_interval": 3000, "user": "jserver" }}
Sent: {"message_type": "Subscribe", "value": { "subscription_mode": "Streaming", ... }}
Received: {"message_type": "ProcessingSnapshot", "value": { "name": "S1" }}
Received: {"message_type": "TopicIntroduction", "value": { "topic_id": 18, "name": "top_bid_price" }}
...
Received: {"message_type": "SnapshotComplete", "value": { "name": "S1" }}
Sent: {"message_type": "Heartbeat", "value": { "u_milliseconds":, 1745425692890 }
Received: {"message_type": "Heartbeat", "value": { "u_milliseconds":, 1745425693895 }
...
Sent: {"message_type": "Logoff"}

Auditing Topics (Control Port)

When running in audit mode (--ws-control-port <port>), GAR will emit internal topic traffic for monitoring

g Schema

• g.gar

Protocol Schema

• proto.gar

Related

• af_unix_transport.md — local connections silently use AF_UNIX sockets rather than TCP loopback, and why a URL may resolve to unix:.
• gar_pipelining.md — streaming records between endpoints in a shell pipeline, and the failure modes to watch for.

Deployment Schema

• deployment.gar

Features in this area

• protocol — The GAR WebSocket protocol.
• client libraries — Connecting to GAR from Python (pygar) and JavaScript (jsgar) over WebSocket.
• subscription class join — Joining a subscription's key filter against another class's key set.
• key name list subscription — Making a subscription with a finite list of key names snapshot by direct lookup instead of scanning the class.
• rpc — Calling a named GAR function and getting a structured reply.