ds-service: An in-memory data structure server

ds-service is a small, in-memory data structure server that clients reach over the network, through gRPC today.

ds-service runs a single server process that holds shared state in memory and lets many distributed clients and workers coordinate using it. Use it when several processes must coordinate, on one machine or across a cluster. It covers four cases:

• The processes hand work to each other.
• The processes share intermediate results.
• The processes take turns on a resource.
• The processes agree on a number.

The state only has to live as long as the run does.

Presently, it provides six data structures: a key-value store, a task queue, a journal store, a time series store, named mutexes, and counters. Each is a separate key space with its own set of operations, described in the data structure reference.

Installation

The server is a single statically linked binary. Download the latest release, make it executable, and put it somewhere on your PATH:

curl -sSL -o ds-service \
    https://github.com/parantapa/ds-service/releases/latest/download/ds-service
chmod +x ds-service

It links against musl with no dynamic dependencies, so it runs on any x86-64 Linux host.

The Python client needs Python 3.12 or newer. It comes from PyPI:

pip install ds-service-client

PyPI has a prebuilt wheel for x86-64 Linux. On any other platform, pip builds the client from source, which needs a C++23 toolchain and takes several minutes.

The C++ client comes from source only. See the C++ client reference.

To build the server from source instead, see how to build the server.

Usage

Start a server:

ds-service --address 127.0.0.1:5051

Then, from any process that can reach it:

from ds_service_client import DsServiceClient

with DsServiceClient("127.0.0.1:5051") as client:
    client.map_set("greeting", b"hello")
    assert client.map_get("greeting") == b"hello"

    client.task_add("job-1", parent_task_ids=[], queue="work", priority=1.0, function=b"greet", input=b"world")

    task = client.task_get(worker_id="worker-a", queue="work")
    client.task_done(task.task_id, worker_id="worker-a", output=b"hello world")

    assert client.task_get_output("job-1") == b"hello world"

Documentation

User documentation

Developer documentation

License

MIT. See LICENSE.