⛵ SAIL (Self-Activating Interference Layer)

The main idea. In standard $\mathbb{R}$-nets, non-linear activation functions (like GELU or SwiGLU) in the feed-forward layer are strictly necessary. However, multiplying two complex numbers is inherently non-linear in itself. SAIL is essentially SwiGLU without the "Swi": It splits the input via a Linear Layer into two parts and multiplies them together. That alone is sufficient to get a fully trainable complex transformer. Combined with residual connections z = z + SAIL(z), my guess is it could also be chaotic somehow (think Mandelbrot set).

🪢 GyRoPE - learnable RoPE, full circle

While implementing experimenting with RoPE in my $\mathbb{R}$-net, I found that a trainable RoPE initialized randomly from $-2\pi$ to $2\pi$ yielded results comparable to classical positive positional encodings. This proved to work nicely with the complex transformers. However the weights are few and they change really slowly during training.

• $\mathbb{C}$-RoPE rotates in the $2\mathcal{D}$ complex plane.
• $\mathbb{H}$-RoPE rotates around a unit 3-sphere ($S^3$).
• $\mathbb{O}$ and $\mathbb{S}$ rotate around their respective hyperspherical counterparts.

👀 Attention

How do you route attention in complex spaces? I experimented with various variants, but turns out: simplicity wins. Attention scores are computed via the Hermitian dot product ($Q K^\dagger$). By extracting only the real component before applying the causal mask and standard Softmax:

# In C-Space:
wei = (q @ k.adjoint()) * (2 * self.head_size) ** -0.5
wei = wei.real
wei = F.softmax(wei.masked_fill(self.tril[:T, :T] == 0, float("-inf")), dim=-1)
# Weighted value aggregation remains hypercomplex:
out = wei.to(v.dtype) @ v

The real part perfectly captures the phase alignment and magnitude between queries and keys after multiplication, acting as a natural scalar affinity score. The resulting real weights are then used to scale the full complex/hypercomplex value vectors ($V$). No other variant yielded a better validation loss so far.

🏋️ Weight Initialization

Attention and SAIL weights are initialized around the unit circle/sphere and scaled by:

$$\sqrt{\frac{S}{d_{in} + d_{out}}}$$

(where $S$ is $\mathbb{C}=1.0$, $\mathbb{H}=0.5$, $\mathbb{O}=0.25$, $\mathbb{S}=0.125$)

🛌 Embedding & LM-Head

To map tokens into hypercomplex spaces, I used separate embedding tables - one for each real component (e.g., 4 tables for $\mathbb{H}$, 16 for $\mathbb{S}$) and initialized them with a scaled normal distribution (Var = 1.0):

# Quaternion Embedding (4D)
z1 = torch.complex(emb_r1(x), emb_i1(x))
z2 = torch.complex(emb_r2(x), emb_i2(x))
---
 nn.init.normal_(emb.weight, std=(1 / math.sqrt(4)))

To project back to vocabulary logits, the LMHead simply flattens all complex components back to $\mathbb{R}$:

features = torch.cat([z1.real, z1.imag, z2.real, z2.imag], dim=-1)
return self.lin(features)

🛡️ RMSNorm - Norming Amplitudes, Preserving Phases

🧨 remove for exploding transformer noises (NaN) 💥

Standard normalization layers can easily distort geometric phase information in complex spaces.

The custom RMSNorm calculates the Root Mean Square solely based on vector magnitudes ($\Vert{}q\Vert{}^2 = \vert{}z_1\vert{}^2 + \vert{}z_2\vert{}^2$):

# Calculate quaternion amplitude squared
ampsq = (z1 * z1.conj()).real + (z2 * z2.conj()).real
rms = torch.rsqrt((ampsq / 2.0).mean(dim=-1, keepdim=True) + self.eps)

return (z1 * rms) * self.weight, (z2 * rms) * self.weight

This scales feature amplitudes cleanly across layers while leaving phase angles completely intact.

🌊 SAIL Interference Patterns ($\mathbb{C}$)

🎬 2D SAIL Interference Evolution

🎬 3D SAIL Phase-Amplitude Topography

Figure: Evolution of selective phase interference across all 8 Transformer layers over training steps. Deep layers (6 & 7) demonstrate extreme destructive phase cancellation alongside razor-sharp resonance peaks (> 10,000 amplitude). That's why the Post-Norm design makes a lot of sense here

📉 Comparison Runs

shared parameters:

$\mathbb{C}$ vs $\mathbb{R}$ / both ~180k params

$\mathbb{R}$ vs the complex rest

Number of parameters scaled. Note: the higher dimensional runs take forever. A "Sedenion" needs 256 multiplications instead of one.

v0.2.0 Bonus - Centumduodevigintunions 🔻

It's ridiculous, pointless and I can't even spell it... but it runs (very slowly). The most interesting usecases may be found on level 1-3 ;)