Keras layers without using matrix multiplications.
Project description
Keras-MatMulLess (Keras-MML)
We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.
— Noam Shazeer, in GLU Variants Improve Transformer
Keras layers without using matrix multiplications.
This is a Keras based implementation of some layers mentioned in the papers The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits and Scalable MatMul-free Language Modeling. Find the documentation here.
Rationale
Traditional, matrix multiplication based layers suffer from a few issues.
- They have high inference and computational costs due to the use of matrix multiplications. This hinders the speed at which inference is performed on GPU-less machines.
- The memory use for storing full precision weights is very high.
- The energy costs of running matrix multiplications is very high.
Matrix multiplication free layers addresses these pain points by removing the key source of costs — matrix multiplications.
Installation
Requirements
Keras-MML has a few requirements, namely
- Python 3.9 (or above);
- Keras; and
- the Keras backend (either Tensorflow, PyTorch, or Jax).
Instructions on how to install Keras can be found here.
Installation Instructions
Only the installation via nightly versions and via GitHub are supported for now.
PyPi
If you use pip, you can install Keras-MML using the command
pip install keras-matmulless
Pre-Release Versions
To install pre-release versions, use the command
pip install --pre keras-matmulless
Nightly Versions
Nightly releases for Keras-MML are primarily found on the TestPyPi page. To install them, use the command
pip install -i https://test.pypi.org/simple/ keras-matmulless
GitHub
First, clone the repository using
git clone https://github.com/PhotonicGluon/Keras-MatMulLess.git
cd Keras-MatMulLess
We recommend to create a virtual environment to install Poetry and the other dependencies into.
python -m venv venv # If `python` doesn't work, try `python3`
Activate the virtual environment using
source venv/bin/activate
or, if you are on Windows,
venv/Scripts/activate
Now we install Poetry.
pip install poetry
Finally, install the development dependencies.
poetry install --with dev
If you have not installed a backend (i.e., Tensorflow, PyTorch, or Jax) you can do so here.
poetry install --with dev,BACKEND_NAME
Note that the BACKEND_NAME
to be specified here is
tensorflow
for the Tensorflow backend;torch
for the PyTorch backend; andjax
for the Jax backend.
That's it! You should now have access to the keras_mml
package.
Installation Issues
These are some of the known issues faced when installing Keras-MML.
Development Installation With torch
Backend
This is a known issue when installing Keras-MML with dev
and torch
on AArch64 (i.e., ARM64) systems.
Attempting to install the torch
backend using
poetry install --with torch
fails with a message like Unable to find installation candidates for torch (2.2.2+cpu)
. To remedy, run
pip install torch~=2.2.2 torchvision~=0.17.2 torchaudio~=2.2.2
Quickstart
Read the tutorial.
Using Dev Containers
If you already have VS Code and Docker installed, you can click the badge above or here to get started. Clicking these links will cause VS Code to automatically install the Dev Containers extension if needed, clone the source code into a container volume, and spin up a dev container for use.
CUDA
Keras-MML offers a cuda
dev container for working with CUDA.
[!IMPORTANT]
Edit theDockerfile
file to set up the architecture properly. By default it is usingamd64
. So, if you are on aarm64
system, uncomment the appropriate lines in the file!
License
Keras-MML is licensed under the Apache 2.0 license.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for keras_matmulless-0.1.0a1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7ca7202bfeadb1bd2029361da0ae3d954e565489fd238e99fb9bb1122ba328d |
|
MD5 | 53a7897d4269b88bea82582644f1e621 |
|
BLAKE2b-256 | ff5e8089bc6fd213c94235780d97cd69eba78b9014d7d3916d922af8ddbc9c47 |