hrr 0.0.1

Holographic Reduced Representations

Holographic Reduced Representations 🔥

Supports

     

TensorFlow     PyTorch     JAX     Flax

Holographic Reduced Representations (HRR) is a method of representing compositional structures using circular convolution in distributed representations. The HRR operations binding and unbinding allow assigning abstract concepts to arbitrary numerical vectors. Given vectors x and y in a d-dimensional space, both can be combined using binding operation. Likewise, one of the vectors can be retrieved knowing one of the two vectors using unbinding operation.

To import the HRR library with the TensorFlow backend use HRR.with_tensorflow, to import with the JAX backend use HRR.with_jax, and so on. Vectors are sampled from a normal distribution with zero mean and the variance of the inverse of the dimension using normal function, with projection onto the ball of complex unit magnitude, to enforce that the inverse will be numerically stable during unbinding, proposed in Learning with Holographic Reduced Representations.

from HRR.with_pytorch import normal, projection, binding, unbinding, cosine_similarity


batch = 32
features = 256

x = projection(normal(shape=(batch, features), seed=0))
y = projection(normal(shape=(batch, features), seed=1))

b = binding(x, y)
y_prime = unbinding(b, x)

score = cosine_similarity(y, y_prime, dim=-1, keepdim=False)
print('score:', score[0])
# prints score: tensor(1.0000)

What makes HRR more interesting is that multiple vectors can be combined by element-wise addition of the vectors, however, retrieval accuracy will decrease.

x = projection(normal(shape=(batch, features), seed=0))
y = projection(normal(shape=(batch, features), seed=1))
w = projection(normal(shape=(batch, features), seed=2))
z = projection(normal(shape=(batch, features), seed=3))

b = binding(x, y) + binding(w, z)
y_prime = unbinding(b, x)

score = cosine_similarity(y, y_prime, dim=-1, keepdim=False)
print('score:', score[0])
# prints score: tensor(0.7483)

More interestingly, vectors can be combined and retrieved in hierarchical order. 🌳

x    y
 \  /
  \/
b=x#y  z 
   \  /
    \/
 c=(x#y)#z

x = projection(normal(shape=(batch, features), seed=0))
y = projection(normal(shape=(batch, features), seed=1))
z = projection(normal(shape=(batch, features), seed=2))

b = binding(x, y)
c = binding(b, z)

b_ = unbinding(c, z)
y_ = unbinding(b_, x)

score = cosine_similarity(y, y_, dim=-1)
print('score:', score[0])
# prints score: tensor(1.0000)

Flax Module (Fastest) ⚡

HRR library supports vector binding/unbinding as a Flax module. Like any other Flax module, this needs to be initialized first and then execute using the apply method.

from HRR.with_flax import normal, Projection, Binding, Unbinding, CosineSimilarity


x = normal(shape=(batch, features), seed=0)
y = normal(shape=(batch, features), seed=1)

projection = Projection()
binding = Binding()
unbinding = Unbinding()
similarity = CosineSimilarity()

# create empty frozen dict as parameter less variable
var = projection.init(jax.random.PRNGKey(0), np.ones((batch, features)))

x = projection.apply(var, x)
y = projection.apply(var, y)

b = binding.apply(var, x, y)
y_ = unbinding.apply(var, b, x)

score = similarity.apply(var, y, y_)
print(f'score: {score[0]:.2f}')
# prints score: 1.00