ndif 0.1.1

National Deep Inference Fabric

Website | nnsight | Discord | Paper

NDIF is the server backend for nnsight. It runs user-submitted intervention code — hooks, captures, edits, generation — against large models on a shared GPU cluster. Researchers point nnsight at an NDIF endpoint and run experiments on models too big to fit on their own hardware.

With no authentication configured — the default — every request is trusted and its code runs inside the model process. Only when auth is on and a key is not granted trusted does the code run in a separate process, one fresh process per request; that isolation is process-based and still being hardened — see docs/concepts/sandbox-execution.md.

This repo is the server. For the client, see nnsight; it is an ordinary dependency here, and just up bind-mounts a local checkout over it for client-side development.

Quick start

Three ways to stand up your own NDIF. All three need an NVIDIA GPU and a CUDA driver; the container routes also need the NVIDIA container toolkit.

Trust default: with no NDIF_POSTGRES_URL the API is unauthenticated, and an unauthenticated NDIF runs every request trusted — the submitted block executes in-process next to the model weights and models load with trust_remote_code. That is the intended default for running one for yourself. Before anyone else can reach it, work through docs/runbooks/enable-auth.md.

1. docker run — the published image, whole stack in one container

docker run --gpus all --shm-size 4g -p 8001:8001 -p 9000:9000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  ndif/ndif:0.1.0

NDIF_SERVICE defaults to all, so the container starts redis, minio, ray and the API together. 8001 is the API; 9000 is the object store, which the client downloads results from, so publish it too. Tags: 0.1.0-cu126, 0.1.0-cu130, 0.1.0 (= cu126) and latest; pick the CUDA line your driver supports. The entrypoint is the ndif CLI, so any other command works the same way:

docker run --rm ndif/ndif:0.1.0 version     # ndif, nnsight, torch+CUDA, transformers, ray
docker run --rm --gpus all ndif/ndif:0.1.0 doctor

2. Docker Compose — the development stack, built from this checkout

Each service in its own container, next to Postgres and the full telemetry set (Loki, InfluxDB, Prometheus, Grafana). just wraps the compose commands.

just up            # build (first time) + start the whole stack, detached
just logs api      # follow a service's logs
just ta            # down -> rebuild -> up, after a source change
just down          # tear it down

3. From source — the ndif CLI, no Docker

No checkout needed — the package is on PyPI:

pip install torch --index-url https://download.pytorch.org/whl/cu126   # first, or pip picks PyPI's default (CUDA 13) wheel
pip install "ndif[api,ray]"                                              # add metrics,postgres,dashboard as you need them
conda install --override-channels -c conda-forge redis-server minio-server
ndif doctor        # versions, binaries, GPU, connectivity
ndif start         # redis, minio, ray, api — detached
ndif stop          # ...and back down, Ray daemons included

From a checkout, pip install -r requirements.txt first gives you the pinned dependency set the image is built from, then pip install ".[api,ray]". redis-server and minio come from conda-forge because MinIO no longer publishes standalone binaries (--override-channels sidesteps the anaconda terms-of-service prompt a stock miniconda raises); the other option is copying the binary out of the quay.io/minio/minio image — see docs/operating/quickstart.md.

Then run a remote trace

import nnsight
nnsight.CONFIG.API.HOST = "http://localhost:8001"

from nnsight.modeling.transformers import TransformersModel
model = TransformersModel("openai-community/gpt2", task="text-generation")

with model.trace("The Eiffel Tower is in the city of", remote=True):
    hidden = model.transformer.h[-1].output.save()

No API key is needed; the first request for a model deploys it. Deeper detail for all three routes is in docs/operating/quickstart.md.

What's running

just ps lists the compose stack. The NDIF pieces:

Redis, MinIO (object store), Postgres (API-key auth), and Loki/InfluxDB/Grafana (telemetry) round out the compose file; see docker/docker-compose.yml.

Development

just ta              # down -> rebuild -> up (full refresh after a code change)
just ta ray          # ...targeting a single service
just build && just up

The ndif CLI is the image entrypoint (ENTRYPOINT ["ndif"], CMD ["start", "--foreground"]); NDIF_SERVICE selects which service(s) a container runs. It accepts a space/comma list, and all expands in place — "all dashboard" is the core stack plus the admin UI. Configuration is read from the environment (see the environment: blocks in the compose file).

Agent-facing documentation lives in CLAUDE.md and docs/.

Configuration

Everything is configured through NDIF_* environment variables. There is no central config file — each service/provider reads its own vars (with a working single-host default) at startup, so a bare just up runs end-to-end with none of these set. Override them in the compose environment: blocks, a .env file, or the shell. Empty defaults for the optional providers (Postgres/Loki/Influx) mean that provider is off until you set its URL.

Core / service

API

Request queue

Autoscaling

Ray / cluster

Controller / deployments

Redis / caches

Object store (S3 / MinIO)

Auth — Postgres (empty URL ⇒ API runs unauthenticated)

Telemetry — InfluxDB (metrics)

Telemetry — Loki (logs) (empty URL ⇒ console-only logging)

Dashboard

Contributing

PRs welcome. Please read the Code of Conduct.

License

MIT © Northeastern University.

Citation

@article{fiottokaufman2024nnsightndifdemocratizingaccess,
      title={NNsight and NDIF: Democratizing Access to Foundation Model Internals},
      author={Jaden Fiotto-Kaufman and Alexander R Loftus and Eric Todd and Jannik Brinkmann and Caden Juang and Koyena Pal and Can Rager and Aaron Mueller and Samuel Marks and Arnab Sen Sharma and Francesca Lucchetti and Michael Ripa and Adam Belfki and Nikhil Prakash and Sumeet Multani and Carla Brodley and Arjun Guha and Jonathan Bell and Byron Wallace and David Bau},
      year={2024},
      eprint={2407.14561},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2407.14561},
}