somatize 0.2.16

A computational graph runtime for research pipelines, agent orchestration, and data virtualization

Soma

Soma (σῶμα — body) is a computational graph runtime for research pipelines, agent orchestration, and data virtualization. Written in Rust with Python bindings.

Part of the Nous-Soma-Chronos ecosystem:

• Nous: Understands, reasons — research IDE, agent graphs, automation
• Soma (this project): Executes, materializes — graphs, optimization, distributed workers
• ChronosVector: Remembers — temporal vector database

Key Concepts

Concept	Description
Filter	Data transformation with fit() (learn state) and forward() (transform). Independently cacheable.
Graph	Computational DAG of filters. Build with .node()/.connect() or >> / | operators.
Graph.somatize()	"You think it. Soma somatizes it." — Materialize a chain/fork topology into an executable graph.
TrainingStrategy	Graph-level attribute: Local, DataParallel, ModelParallel, Federated, PopulationBased.
Study	Hyperparameter optimization: Grid, Random, or Bayesian (TPE) search with median/percentile pruning.
PBT	Population-Based Training: evolutionary train→evaluate→exploit/explore cycles.
ExecutionPlan	Compiled from graph. Variants: Sequence, Parallel, Execute, Cached, Remote, Loop, Branch.
DataStore	Abstraction for data movement: Local, S3, Zarr (chunked tensors), Cached, Stream.
Worker	Remote execution daemon. Auto-detects hardware, Slurm-style resource limits, token auth.
Coordinator	Lightweight gateway: worker registration, routing, health monitoring.

Workspace (10 crates)

soma-macros     → proc macro (#[derive(SomaFilter)])
soma-core       → types + traits: Filter, Value, Graph, TrainingStrategy, Schema, Event
                  DataStore (Local/S3/Zarr), VirtualValue, StreamCache
soma-compiler   → Graph → ExecutionPlan (caching, parallelism, distribution)
                  Scheduler, plan visualization (Mermaid/Graphviz)
soma-runtime    → GraphSession, executor, FilterLibrary, caches, samplers, pruners
                  StudyRunner, PbtRunner, stream executor
soma-memory     → KnowledgeBase trait + MemoryKB + ChronosKB
soma-worker     → Worker, Coordinator, Protocol, EnvManager, token auth
                  Auto-detect capabilities, resource limits, CLI binary
soma-agent      → Research agent loop (observe → hypothesize → experiment → conclude)
soma-mcp        → MCP server (13 tools for code, execution, knowledge)
soma-python     → PyO3 bindings: Graph, Filter, Study, Lab, Chain/Fork operators

Quick Start

# Run all tests (355 Rust + 29 Python)
cargo test --workspace
cd soma-python && maturin develop && pytest tests/ -v

# With S3/Zarr DataStore
cargo test -p soma-core --features s3
cargo test -p soma-core --features zarr

# With ChronosVector
cargo test -p soma-memory --features chronos

# MCP server
cargo run -p soma-mcp -- /path/to/project

Python Usage

from soma import Filter, Graph, Study, search

class Scaler(Filter):
    _differentiable = True

    def fit(self, x, y=None):
        return {"mean": sum(x) / len(x)}

    def forward(self, x, state):
        return [v - state["mean"] for v in x]

class Model(Filter):
    lr: float = search(0.001, 1.0, scale="log")

    def fit(self, x, y=None):
        return {"weights": [0.5] * len(x)}

    def forward(self, x, state):
        return [v * w for v, w in zip(x, state["weights"])]

# Build with >> (chain) and | (fork)
g = Graph.somatize(Scaler() >> Model())
g.fit(train_data)
result = g.forward(test_data)

# Visualize
print(g.to_mermaid())
print(g.to_text())

# Complex topologies
g = Graph.somatize(
    (LoadA() >> NormA() | LoadB() >> NormB())
    >> Aggregate()
    >> Backbone()
    >> (HeadA() | HeadB())
)

# Events
g.on_event(lambda e: print(e["event_type"], e.get("node_id", "")))

# Distributed training
g.set_strategy(DataParallel(num_replicas=4))
g.set_coordinator("http://coord:9090", token="sk-xxx")

Workers

# Start a worker with auto-detected capabilities
soma-worker --port 8080 --tags gpu,training --token sk-xxx

# With resource limits (Slurm-style)
soma-worker --cpus 4 --memory 8G --gpus 1 --max-concurrent 2

# With coordinator auto-registration
soma-worker --coordinator http://coord:9090 --token sk-xxx --tags gpu

Workers auto-detect CPU cores, RAM, GPUs (nvidia-smi), and Python environments. Each worker creates isolated venv/conda environments per job with incremental dependency updates.

Feature Flags

• soma-core/s3 — S3-compatible DataStore (AWS, Backblaze B2, MinIO)
• soma-core/zarr — Zarr v3 chunked tensor storage with compression
• soma-memory/chronos — ChronosVector-backed KnowledgeBase

License

Elastic License 2.0