organic-clone-machines 0.0.2

OCM: Hodgkin-Huxley spiking neural networks as a Transformer-shaped language model architecture

OCM — Organic Clone Machines

v0.0.2 · Author: Ömür Bera Işık

A Transformer-shaped architecture where every sublayer is a simulated population of biologically realistic Hodgkin-Huxley (HH) spiking neurons — run in parallel on GPU as a single batch tensor.

First language model based on Hodgkin-Huxley spiking neuron dynamics.

---

Install

pip install organic-clone-machines

Or from source:

git clone <your-repo>
cd ocm_project
pip install -e .

---

Quick Start

from ocm import OCMConfig, OCMForCausalLM
import torch

cfg = OCMConfig(
    num_neurons=256,           # HH population per layer
    synapses_per_neuron=16,    # sparse out-degree per neuron
    quality="standard",        # integration fidelity preset
    hidden_size=256,
    num_layers=4,
    vocab_size=50257,
)

model = OCMForCausalLM(cfg)
print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# Forward pass
ids = torch.randint(0, 50257, (1, 32))
out = model(ids, labels=ids)
print(f"Loss: {out.loss.item():.4f}")

# Text generation
tokens = model.generate(ids[:, :8], max_new_tokens=20, temperature=0.8)

Train on any HuggingFace dataset

from ocm import OCMConfig
from ocm.trainer import OCMTrainer

trainer = OCMTrainer.for_causal_lm(
    config=OCMConfig(num_neurons=512, synapses_per_neuron=32, quality="standard"),
    dataset_name="wikitext",
    dataset_config="wikitext-2-raw-v1",
    tokenizer_name="gpt2",
    output_dir="./ocm-gpt",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    learning_rate=3e-4,
)
trainer.train()

CLI

# Text generation (causal LM)
ocm-train causal_lm \
    --num_neurons 512 --synapses_per_neuron 32 --quality standard \
    --dataset_name wikitext --dataset_config wikitext-2-raw-v1 \
    --tokenizer_name gpt2 --output_dir ./ocm-gpt \
    --num_train_epochs 3 --per_device_train_batch_size 8

# Masked LM (BERT-style)
ocm-train masked_lm \
    --num_neurons 256 --synapses_per_neuron 16 \
    --dataset_name wikitext --dataset_config wikitext-2-raw-v1 \
    --tokenizer_name bert-base-uncased --output_dir ./ocm-bert

# Sequence classification
ocm-train seq_clf \
    --num_neurons 128 --synapses_per_neuron 8 --num_labels 2 \
    --dataset_name stanfordnlp/imdb \
    --tokenizer_name gpt2 --output_dir ./ocm-imdb

# Token classification (NER)
ocm-train tok_clf \
    --num_neurons 128 --synapses_per_neuron 8 --num_labels 9 \
    --dataset_name conll2003 \
    --tokenizer_name bert-base-cased --output_dir ./ocm-ner

# See all options
ocm-train --help

---

The Five Model Types

#	Class	Task	HF Analogy
1	OCMModel	Feature extraction backbone	GPT2Model
2	OCMForCausalLM	Text generation ← flagship	GPT2LMHeadModel
3	OCMForMaskedLM	BERT-style masked LM	BertForMaskedLM
4	OCMForSequenceClassification	Sentence classification / regression	BertForSequenceClassification
5	OCMForTokenClassification	NER, POS tagging	BertForTokenClassification

from ocm import (
    OCMModel,
    OCMForCausalLM,
    OCMForMaskedLM,
    OCMForSequenceClassification,
    OCMForTokenClassification,
)

All five are full transformers.PreTrainedModel subclasses — save_pretrained, from_pretrained, push_to_hub all work.

---

Architecture

The Hodgkin-Huxley Core

Each OCM block maintains a population of N neurons, each described by four ODEs (Hodgkin & Huxley, 1952):

C_m dV/dt = I_ext - g_Na·m³·h·(V-E_Na) - g_K·n⁴·(V-E_K) - g_L·(V-E_L)
dm/dt = αm(V)(1-m) - βm(V)·m
dh/dt = αh(V)(1-h) - βh(V)·h
dn/dt = αn(V)(1-n) - βn(V)·n

All N neurons are stored as a single (batch, N, 4) tensor for (V, m, h, n). The derivatives for the entire population are computed in one vectorized PyTorch call — one kernel launch regardless of N. This is the core insight from the source paper: GPU SIMD maps perfectly onto neuron-parallel HH integration.

Cross-Token Mixing (the Attention Analogue)

Each OCMBlock keeps a spike ring buffer of shape (max_delay, batch, N). At token position t, neuron j receives:

I_syn[j](t) = Σᵢ  W_ij · S_i(t − δij)

where δij is a fixed integer axonal delay drawn from [1, max_delay] and W_ij is a learnable synapse weight. This delayed-spike readout is OCM's causal cross-token mixing primitive — the role attention plays in a Transformer.

Recurrence vs. Parallelism

OCM processes token positions sequentially (RNN/SSM-shaped). Parallelism is along the neuron dimension (N neurons per step, all integrated in one batched call). The past_ocm_state cache lets you extend a sequence one token at a time in O(1) work per layer during generation.

Dimension	Transformer	OCM
Token positions	Parallel	Sequential (RNN-like)
Features per position	Sequential through layers	Parallel: N neurons/layer
Cross-position mixing	Attention (all-to-all)	Delayed spike buffer (causal, sparse)

---

Configuration Reference

OCMConfig(
    # ── Biology ──────────────────────────────────────
    num_neurons=256,            # N neurons per layer
    synapses_per_neuron=16,     # out-degree k (must be < num_neurons)
    max_delay=8,                # spike ring buffer depth (token steps)
    v_thresh=0.0,               # spike threshold (mV)
    surrogate_beta=2.0,         # surrogate gradient steepness
    heterogeneous_neurons=False, # per-neuron learnable (g_Na,g_K,g_L)

    # ── Integration ───────────────────────────────────
    quality="standard",         # "draft"|"standard"|"high"|"research"

    # ── Architecture ─────────────────────────────────
    hidden_size=256,            # d_model
    num_layers=4,               # OCM block stack depth
    max_position_embeddings=1024,
    dropout=0.1,

    # ── Standard HF ──────────────────────────────────
    vocab_size=50257,
    pad_token_id=0,
    num_labels=2,               # for classification heads
)

Quality Presets

All presets use dt = 0.01 ms (paper's stability floor — smaller dt → divergence). Only substep count and integrator vary:

Preset	Substeps	Integrator	Cost	Use when
draft	1	Euler	1×	Shape/pipeline debug
standard	4	Euler	4×	Normal training ← default
high	8	Euler	8×	Long sequences
research	8	RK4	32×	Max fidelity

Sizing Guide

Scale	num_neurons	synapses_per_neuron	~Params (4L, d=256)
Micro	64	8	~500 K
Small	256	16	~3 M
Medium	512	32	~10 M
Large	1024	64	~35 M

---

HuggingFace Integration

# Save / load
model.save_pretrained("./my-ocm")
model = OCMForCausalLM.from_pretrained("./my-ocm")

# Push to Hub
model.push_to_hub("username/ocm-wikitext")

# Stream large datasets
trainer = OCMTrainer.for_causal_lm(
    config=cfg,
    dataset_name="c4",
    dataset_config="en",
    streaming=True,
    max_train_samples=100_000,
    tokenizer_name="gpt2",
    output_dir="./ocm-c4",
)

---

Training Dynamics / Cold Start

Early in training, neurons receiving weak stimulus may not fire (s = 0). Synapse weight gradients are d(W·s)/d(W) = s, so un-fired synapses get zero gradient from the synaptic path.

Mitigation strategies:

• Lower surrogate_beta (e.g. 1.0) early on — widens the gradient region around threshold
• Start with quality="draft" for the first few hundred steps, then switch to "standard"
• stim_scale (per-block learnable scalar) is initialized to 10.0; a warm-start with a slightly larger value pushes neurons closer to threshold faster

---

The Biology

Standard squid-axon parameters (Hodgkin & Huxley, 1952; Dayan & Abbott 2001):

C_m = 1.0 μF/cm²    g_Na = 120 mS/cm²   E_Na = +50 mV
                     g_K  =  36 mS/cm²   E_K  = −77 mV
                     g_L  = 0.3 mS/cm²   E_L  = −54.387 mV
V_rest = −65 mV      m_∞ = 0.0529   h_∞ = 0.5961   n_∞ = 0.3177

With heterogeneous_neurons=True, each neuron gets its own learnable (g_Na, g_K, g_L) initialized near standard values with small noise, allowing population diversity to emerge via gradient descent.

---

PyPI

# Install
pip install organic-clone-machines

# Upgrade
pip install --upgrade organic-clone-machines

# Publish a new version (maintainer only)
python -m build
python -m twine upload dist/*

---

Citation

@software{isik2026ocm,
  title   = {OCM: Organic Clone Machines — Hodgkin-Huxley Spiking Neurons as a Language Model Architecture},
  author  = {Işık, Ömür Bera},
  year    = {2026},
  version = {0.0.2},
}

@article{isik2026hh,
  title  = {Massively Parallel Hodgkin-Huxley Neuron Simulation via GPU Batch Parallelism: One Neuron Cost, Billion Neuron Scale},
  author = {Işık, Ömür Bera},
  year   = {2026},
}

---

License

MIT License — Copyright (c) 2026 Ömür Bera Işık