Dagents

Dagents is a framework for running analytics and machine learning across many data sources without moving the data. Each data source runs a local agent, a global agent combines their results, and typed planners decide what each request may see and whether a new model may be released.

It started as shared code from products such as Watchdog, and is meant to be reused by systems such as Datalytics.

Status: alpha. Dagents is in early development. Every version before 1.0.0 is a pre-release, and the Python APIs, settings, routes and package layout can change from one release to the next.

Install

The Python framework is released to PyPI as dagents, starting with 0.1.0a1. It needs Python 3.11 or later.

pip install dagents                # the Ethical Guard, federated rounds, extensions and the planner runner
pip install "dagents[lma,gma]"     # adds FastAPI, uvicorn and httpx, to serve an LMA or a GMA

The other extras are postgres (the Postgres audit chain, job log, model registry and source adapter), charts (the SVG chart renderer), compose (PyYAML, for the appliers that start a service from a Compose file) and all. While every release is a pre-release, pip install dagents installs the newest one. After 1.0.0, a pre-release needs pip install --pre dagents or an exact version.

The wheels for Linux x86_64, Linux aarch64 and macOS arm64 include the planner, dagentsc. On other systems, such as Windows and Intel Macs, pip builds the package from its source distribution, which has no planner. There the Ethical Guard denies every request until DAGENTSC_BIN names a planner binary. Each GitHub Release has the planner for the three platforms, and bindings/ocaml/ shows how to build it. python -m dagents prints the installed version and the planner it found.

dagents-lma and dagents-gma start an agent. An installed package reads its settings only from the process environment; the LMA and GMA READMEs list them. The services in services/, the Spring services and the demos are not in the package.

Links

The stroke triage demo uses real, de-identified records from the International Stroke Trial (IST-1, 1991-1996): 1,014 patients from three hospitals in the United Kingdom, Italy and Switzerland. Each hospital keeps its records in its own store, and the page has a view for each hospital, with a form to enter a new record. Each hospital is served by a Dagents LMA and the study by a Dagents GMA, and the demo calls them only through their HTTP APIs; the page lists the agent calls behind each answer. The published page calls the demo's deployed API, which runs the agents in its own process. The agents make every governance and federation decision with the OCaml planner in its image, and the page shows the data source the API reports. The API sleeps when idle, so the first request takes about 20 seconds, and the page shows a banner while it waits. If, when the site is built, the API does not answer, is not reading Postgres, has a hospital store without records, or refuses requests from the site, the page replays a recorded run instead and says so.

The NL2SQL demo has no deployed backend. It replays responses recorded from the real services and names the commit they came from. A question that was not recorded gets no answer.

apps/healthcare-demo/docs/deployment.md explains how the deployed parts fit together and where each credential is kept.

Learn

The learning guide explains federated learning, data governance, the Dagents architecture and the APIs, with diagrams, videos, papers and exercises. Read it on the framework site or in docs/learn/.

1. Federated learning
2. Privacy and data governance
3. How Dagents is built
4. Using the APIs
5. Run it yourself

The slide decks and reports we used to teach the framework are on the site's Resources page, as PowerPoint files and PDFs. The list is docs/presentation/resources.json.

Build an application on Dagents

The guide to building an application shows, step by step, how to use Dagents in your own code: choose one of three ways to build on it, describe your data, give each source its own store, govern every read, run a federated round, call the framework services, deploy and test. Read it on the framework site or in docs/guide/. Its code comes from a small application in examples/getting-started/, and a test checks that the two match.

What is in the repository

Governance and federated rounds

These are the two parts an application would otherwise build itself:

• Governance (GRAILS). A planner decides the protection each field of a request needs, from three inputs: how sensitive the field is, how much the requester is trusted, and how much data is requested. The Ethical Guard applies the decision and writes it to an audit log. The planner is OCaml, so adding a sensitivity level or a strategy does not compile until every combination is handled. If the planner cannot be reached, the Guard denies the request.
• Federated rounds. Round manifests with a digest, site eligibility, quorum, aggregation readiness and release gates. Dagents decides who may take part and whether a result may be released. The training protocol itself belongs to a dedicated federated learning runtime behind an adapter. Aggregation produces a candidate model, never a release.

Applications use both through dagents/common/extensions, by registering feature contracts, data classifications, condition packs, model kinds, site sources, pipeline steps and model adapters, and call the LMA and GMA through their APIs. apps/healthcare-demo/ is a worked example.

Run it

Build the planner, then run the stroke triage demo locally (no Docker or database needed):

cd bindings/ocaml && opam exec -- dune build ./bin/dagentsc.exe && cd -
export DAGENTSC_BIN=$PWD/bindings/ocaml/_build/default/bin/dagentsc.exe
apps/healthcare-demo/scripts/run_frontend_demo.sh

Step 5 of the guide covers setup, tests and exercises.

To run the whole stack in containers, with the committed configuration:

docker compose --env-file env/.env.compose up --build

Each service reads env/.env.shared plus its own file under env/.

Further reading

• AGENTS.md: the contributor guide
• docs/reference/service-inventory.md: every endpoint, generated from the code
• docs/architecture/ocaml-adoption-plan.md: why the planners are in OCaml
• docs/integrations/datalytics-backend-guide.md: moving the Datalytics backend onto Dagents
• docs/diagrams/puml/ and docs/presentation/puml/: PlantUML sources and rendered diagrams

The stroke triage demo's records come from the International Stroke Trial database (version 2), made available under the ODC Attribution License; see apps/healthcare-demo/data/ist/NOTICE.md. The demo's models are not clinically validated, and it is not a medical device.