keras-matmulless 0.1.0a1

Keras layers without using matrix multiplications.

Keras-MatMulLess (Keras-MML)

We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.

— Noam Shazeer, in GLU Variants Improve Transformer

Keras layers without using matrix multiplications.

This is a Keras based implementation of some layers mentioned in the papers The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits and Scalable MatMul-free Language Modeling. Find the documentation here.

Rationale

Traditional, matrix multiplication based layers suffer from a few issues.

1. They have high inference and computational costs due to the use of matrix multiplications. This hinders the speed at which inference is performed on GPU-less machines.
2. The memory use for storing full precision weights is very high.
3. The energy costs of running matrix multiplications is very high.

Matrix multiplication free layers addresses these pain points by removing the key source of costs — matrix multiplications.

Installation

Requirements

Keras-MML has a few requirements, namely

• Python 3.9 (or above);
• Keras; and
• the Keras backend (either Tensorflow, PyTorch, or Jax).

Instructions on how to install Keras can be found here.

Installation Instructions

Only the installation via nightly versions and via GitHub are supported for now.

PyPi

If you use pip, you can install Keras-MML using the command

pip install keras-matmulless

Pre-Release Versions

To install pre-release versions, use the command

pip install --pre keras-matmulless

Nightly Versions

Nightly releases for Keras-MML are primarily found on the TestPyPi page. To install them, use the command

pip install -i https://test.pypi.org/simple/ keras-matmulless

GitHub

First, clone the repository using

git clone https://github.com/PhotonicGluon/Keras-MatMulLess.git
cd Keras-MatMulLess

We recommend to create a virtual environment to install Poetry and the other dependencies into.

python -m venv venv  # If `python` doesn't work, try `python3`

Activate the virtual environment using

source venv/bin/activate

or, if you are on Windows,

venv/Scripts/activate

Now we install Poetry.

pip install poetry

Finally, install the development dependencies.

poetry install --with dev

If you have not installed a backend (i.e., Tensorflow, PyTorch, or Jax) you can do so here.

poetry install --with dev,BACKEND_NAME

Note that the BACKEND_NAME to be specified here is

• tensorflow for the Tensorflow backend;
• torch for the PyTorch backend; and
• jax for the Jax backend.

That's it! You should now have access to the keras_mml package.

Installation Issues

These are some of the known issues faced when installing Keras-MML.

Development Installation With torch Backend

This is a known issue when installing Keras-MML with dev and torch on AArch64 (i.e., ARM64) systems.

Attempting to install the torch backend using

poetry install --with torch

fails with a message like Unable to find installation candidates for torch (2.2.2+cpu). To remedy, run

pip install torch~=2.2.2 torchvision~=0.17.2 torchaudio~=2.2.2

Quickstart

Read the tutorial.

Using Dev Containers

If you already have VS Code and Docker installed, you can click the badge above or here to get started. Clicking these links will cause VS Code to automatically install the Dev Containers extension if needed, clone the source code into a container volume, and spin up a dev container for use.

CUDA

Keras-MML offers a cuda dev container for working with CUDA.

[!IMPORTANT]
Edit the Dockerfile file to set up the architecture properly. By default it is using amd64. So, if you are on a arm64 system, uncomment the appropriate lines in the file!

License

Keras-MML is licensed under the Apache 2.0 license.