python-fedci 0.2.0

A small package for federated independence tests

fedci

Federated conditional independence (CI) testing via likelihood ratio tests. Data never leaves each client — only aggregated sufficient statistics are exchanged with the central server.

---

Installation

uv add python-fedci        # or: pip install python-fedci

Requires Python 3.10+. ZMQ networking requires pyzmq.

---

Core concepts

Class	Role
Client	Holds a local dataset; answers statistical queries from the server
Server	Orchestrates tests across all clients; never sees raw data
TestResult	Result of one CI test (p-value, Bayes factors)
RepeatedTestResult	Aggregated statistics over many bootstrap runs

---

Basic workflow

import polars as pl
from fedci import Client, Server

# Each site wraps its own data
df1 = pl.read_parquet("site1.parquet")
df2 = pl.read_parquet("site2.parquet")

client1 = Client("site1", df1)
client2 = Client("site2", df2)

server = Server([client1, client2])

Test a single conditional independence

# Is X independent of Y given {Z, W}?
result = server.test("X", "Y", {"Z", "W"})

result.p_value          # float
result.log10_bic_bf     # log10 Bayes factor (BIC approximation)
result.log10_bff        # log10 Bayes factor (BFF method)
result.pH0_bic          # P(H0) = 1 / (1 + BF)  via BIC
result.pH0_bff          # P(H0) = 1 / (1 + BF)  via BFF
result.n_samples        # total observations used

Run all pairwise tests

# All pairs up to conditioning set size 2, 4 parallel workers
results = server.run(max_cond_size=2, workers=4, progress_bar=True)

# results is Dict[(x, y, frozenset(s)), TestResult]
for (x, y, s), r in results.items():
    print(f"{x} ⊥ {y} | {s}  p={r.p_value:.4f}  log10BF={r.log10_bic_bf:.2f}")

workers parallelises at the test level — each CI test runs in its own thread. Numpy releases the GIL during computation so multiple cores are used effectively.

Repeated tests with subsampling

Bootstrap stability analysis: run the same test many times on random subsamples to assess reliability.

rtr = server.test_repeatedly(
    "X", "Y", {"Z"},
    num_runs=100,
    sample_fraction=0.80,
    workers=4,           # parallel runs (local clients only; see Networking)
    progress_bar=True,
)

rtr.n_runs              # 100
rtr.mean_p              # mean p-value across runs
rtr.mean_log10_bic_bf   # mean log10 BF
rtr.std_log10_bic_bf    # spread — lower means more stable
rtr.rate_dependence_bic # fraction of runs where BF > 1 (evidence for dependence)
rtr.mean_pH0_bic        # mean P(H0) across runs

Run the full test suite repeatedly:

repeated = server.run_repeatedly(
    num_runs=50,
    sample_fraction=0.80,
    max_cond_size=2,
    workers=4,           # parallel tests within each run
    progress_bar=True,
)
# Dict[(x, y, frozenset(s)), RepeatedTestResult]

---

Interpreting results

Two Bayes factor estimates are available alongside the p-value.

BIC Bayes factor

log10(BF) ≈ ½ (2·ΔLL − Δk·log n) / log(10)

Closed-form, conservative, low variance across subsamples.

BFF — Bayes Factor Function

Maximises a chi² Bayes factor over a prior width grid.

Decision guide

log10(BF)	Interpretation
> 1.0	Strong evidence for dependence
0 to 1.0	Weak / inconclusive
< 0	Evidence for independence

pH0 = 1 / (1 + BF) converts a Bayes factor directly to a probability of the null.

For stability assessment across repeated runs, prefer rate_dependence_bff (fraction of runs where BF > 1) and std_log10_bff_bf (spread on log scale) over a single p-value threshold or its _bic counterparts.

---

Model configuration

Default — GLM

Logistic regression for binary/categorical responses; linear regression for continuous. No extra configuration required.

server = Server([client1, client2])

GAM — nonlinear continuous relationships

Cubic B-spline basis expansion for each continuous predictor. Custom knot placement is supported, though preferably, data is normalised to [0, 1] before being passed to clients so that a fixed knot grid is valid across all sites.

import numpy as np
from fedci import Server, Client, GAMConfiguration, ModelType

# Knot arithmetic:
#   num_knots=8, num_degrees=3  →  n_basis = num_knots + num_degrees - 2 = 9
#   knot vector length = num_knots + 2*num_degrees - 1 = 13  (5 interior knots)
N_KNOTS  = 8
DEGREE   = 3
interior = list(np.linspace(0.0, 1.0, N_KNOTS - DEGREE + 2)[1:-1])
knot_vec = [0.0] * (DEGREE + 1) + interior + [1.0] * (DEGREE + 1)

# Normalise to [0, 1] on the combined dataset before splitting into clients
mins = df_full.min()
maxs = df_full.max()
df_norm = df_full.with_columns([
    ((pl.col(c) - mins[c][0]) / (maxs[c][0] - mins[c][0])).alias(c)
    for c in df_full.columns
])

gam_config = GAMConfiguration(
    type=ModelType.GAM,
    num_knots=N_KNOTS,
    num_degrees=DEGREE,
    knots={v: knot_vec for v in df_norm.columns},  # include all variables
)

server = Server([client1, client2], model_configuration=gam_config)

Including a variable in knots is safe even when it appears as the response in some tests — the spline basis is only applied to predictors, never to the response.

Site heterogeneity

When datasets across sites differ systematically (e.g. batch effects or recruitment differences), enable random effects:

from fedci import HeterogenietyType

# Random intercepts only (recommended — far fewer parameters)
server = Server(
    [client1, client2],
    heterogeniety=HeterogenietyType.GLOBAL,
    random_intercept_only=True,              # one shift per site per model
    local_ridge_coefficient=1.0,             # starting prior strength; adapted via EM
)

# Full random effects on all coefficients (intercept + slopes)
server = Server(
    [client1, client2],
    heterogeniety=HeterogenietyType.GLOBAL,
    random_intercept_only=False,             # site-specific shift on every coefficient
    local_ridge_coefficient=1.0,
)

random_intercept_only=False is heavily overparameterised for most datasets — a warning is raised when used with GAM, where the spline basis already has many coefficients. Prefer True unless you have a strong reason for full random slopes.

HeterogenietyType	Behaviour
NONE (default)	Pooled model, no site effects
GLOBAL	Random effects with shared variance across sites, estimated by EM; calculates effective DoF
LOCAL	Independent random effects per site, not shared; uses regular DoF

---

Additive masking

Additive masking protects intermediate aggregates: each client adds a per-pair random mask to its output; masks cancel exactly in the sum, so the server only ever sees the correct aggregate — never any individual client's contribution.

Local masking

For multiple clients running in the same process:

server = Server([client1, client2], additive_masking=True)
results = server.run(max_cond_size=1)
# Numerically identical to unmasked results

Network masking

When clients run on separate machines, additive_masking=True triggers a peer-to-peer seed exchange: the server passes each client's address to all others, peers connect directly to agree on shared random seeds, and thereafter each masks its own output independently. The server never participates in seed exchange.

from fedci import connect_client, Server

nc1 = connect_client("192.168.1.10", 5555)
nc2 = connect_client("192.168.1.11", 5555)

server = Server([nc1, nc2], additive_masking=True)
results = server.run(max_cond_size=1)

---

Networking

Each client runs serve_client on its own machine. The orchestrating server uses connect_client and receives a NetworkClient that has the same interface as a local Client.

On each client machine

import polars as pl
from fedci import Client, serve_client

df = pl.read_parquet("local_data.parquet")
client = Client("site1", df)

serve_client(client, port=5555)   # blocks; run as a separate process or service

On the server machine

from fedci import Server, connect_client

nc1 = connect_client("192.168.1.10", 5555)
nc2 = connect_client("192.168.1.11", 5555)

server = Server([nc1, nc2])
results = server.run(max_cond_size=2, workers=4)

How it works

Communication uses ZeroMQ (ROUTER/REQ pattern) with pickle serialisation, which handles numpy arrays, dataclasses, and sets without extra schema definitions. Each calling thread gets its own ZMQ socket, so multiple concurrent tests can call the same remote client without conflicts. The remote serve_client loop processes requests sequentially.

Parallelism with network clients

workers in server.run() and server.test_repeatedly() parallelises across threads and works transparently with both local and network clients.

Each parallel run in test_repeatedly registers its subsample on every client under a unique run-ID, then passes that ID with every compute call. The remote serve_client loop processes requests sequentially but each request carries its own run-ID, so concurrent runs never interfere — they simply reference different subsample entries in the client's registry. The registry is cleaned up automatically when each run completes.

---

Variable types

Client infers variable types from the Polars schema automatically:

Polars dtype	Treated as	Model
Float32 / Float64	Continuous	Gaussian GLM / GAM
Boolean	Binary	Logistic regression
Utf8 / Categorical	Categorical	Multinomial logistic
Int*	Ordinal	Proportional-odds

When sites have different category levels for the same variable, the server takes the union across all clients automatically.