ds-service: Yet Another Data Structure Server

ds-service is a small, in-memory data structure server that is accessible via gRPC.

ds-service runs a single server process that holds shared state in memory and lets many distributed clients and workers coordinate using it. Use it when several processes -- on one machine or across a cluster -- need to hand work to each other, share intermediate results, take turns on a resource, or agree on a number, and the state only has to live as long as the run does.

Presently, it provides six data structures:

• A key-value store -- a shared string -> bytes store for passing data between processes.
• A task queue -- a priority-based work queue that distributes tasks to workers and tracks their state.
• A journal store -- append-only, ordered logs of binary entries.
• A time series store -- append-only series of timestamped floating-point values.
• Named mutexes -- worker-owned locks for coordinating exclusive resource access across workers.
• Counters -- named monotonic counters that hand out successive integers.

Each of these is a separate key space with its own set of RPCs.

Installation

The server is a single statically linked binary. Grab the latest release, make it executable, and put it somewhere on your PATH:

curl -sSL -o ds-service \
    https://github.com/parantapa/ds-service/releases/latest/download/ds-service
chmod +x ds-service

It is linked against musl with no dynamic dependencies, so it runs on any x86-64 Linux host.

The Python client comes from PyPI:

pip install ds-service-client

To build the server from source instead, see how to build the server.

Usage

Start a server:

ds-service --address 127.0.0.1:5051

Then, from any process that can reach it:

from ds_service_client import DsServiceClient

with DsServiceClient("127.0.0.1:5051") as client:
    client.map_set("greeting", b"hello")
    assert client.map_get("greeting") == b"hello"

    client.task_add("job-1", queue="work", priority=1.0, function=b"greet", input=b"world")

    task = client.task_get(worker_id="worker-a", queue="work")
    client.task_done(task.task_id, worker_id="worker-a", output=b"hello world")

    assert client.task_get_output("job-1") == b"hello world"

Documentation

License

MIT. See LICENSE.