ds-service: An in-memory data structure server

ds-service is a small, in-memory data structure server that is accessible via gRPC.

ds-service runs a single server process that holds shared state in memory and lets many distributed clients and workers coordinate using it. Use it when several processes must coordinate, on one machine or across a cluster. It covers four cases:

• The processes hand work to each other.
• The processes share intermediate results.
• The processes take turns on a resource.
• The processes agree on a number.

The state only has to live as long as the run does.

Presently, it provides six data structures: a key-value store, a task queue, a journal store, a time series store, named mutexes, and counters. Each is a separate key space with its own set of RPCs, described in the data structure reference.

Installation

The server is a single statically linked binary. Download the latest release, make it executable, and put it somewhere on your PATH:

curl -sSL -o ds-service \
    https://github.com/parantapa/ds-service/releases/latest/download/ds-service
chmod +x ds-service

It links against musl with no dynamic dependencies, so it runs on any x86-64 Linux host.

The Python client needs Python 3.12 or newer. It comes from PyPI:

pip install ds-service-client

To build the server from source instead, see how to build the server.

Usage

Start a server:

ds-service --address 127.0.0.1:5051

Then, from any process that can reach it:

from ds_service_client import DsServiceClient

with DsServiceClient("127.0.0.1:5051") as client:
    client.map_set("greeting", b"hello")
    assert client.map_get("greeting") == b"hello"

    client.task_add("job-1", parent_task_ids=[], queue="work", priority=1.0, function=b"greet", input=b"world")

    task = client.task_get(worker_id="worker-a", queue="work")
    client.task_done(task.task_id, worker_id="worker-a", output=b"hello world")

    assert client.task_get_output("job-1") == b"hello world"

Documentation

User documentation

Developer documentation

License

MIT. See LICENSE.