ds-service: Yet Another Data Structure Server

ds-service is a small, in-memory data structure server that is accessible via gRPC.

ds-service runs a single server process that holds shared state in memory and lets many distributed clients and workers coordinate using it.

Presently, it provides six data structures:

• A key-value store -- a shared string -> bytes store for passing data between processes.
• A task queue -- a priority-based work queue that distributes tasks to workers and tracks their state.
• A journal store -- append-only, ordered logs of binary entries.
• A time series store -- append-only series of timestamped floating-point values.
• Named mutexes -- cooperative locks for coordinating exclusive resource access across workers.
• Counters -- named monotonic counters that hand out successive integers.

Each of these is a separate key space with its own set of RPCs.

Architecture

• Server (cpp/ds-service.cpp) -- a C++23 gRPC service. All state lives in memory, with a separate lock guarding each top-level data structure. Operations on one structure are serialized, while operations on different structures may run concurrently. Each RPC touches a single structure, so no request ever holds more than one lock. State is not persisted; that is, when the server stops all data is lost.
• Client (python/ds_service_client/) -- a Python 3.12+ client library that wraps the generated gRPC stubs and translates gRPC status codes into Python exceptions (KeyError, ValueError, TimeoutError, and the task-queue specific NoTaskAvailable and TaskStateError).
• Interface (misc/ds-service.proto) -- the protobuf/gRPC contract shared by both sides.

Additional Information