BitMoE
1 bit Mixture of Experts utilizing BitNet ++ Mixture of Experts. Also will add distribution amongst GPUs.
install
$ pip3 install bitmoe
usage
import torch
from bitmoe.main import BitMoE
# Set the parameters
dim = 10 # Dimension of the input
hidden_dim = 20 # Dimension of the hidden layer
output_dim = 30 # Dimension of the output
num_experts = 5 # Number of experts in the BitMoE model
# Create the model
model = BitMoE(dim, hidden_dim, output_dim, num_experts)
# Create random inputs
batch_size = 32 # Number of samples in a batch
sequence_length = 100 # Length of the input sequence
x = torch.randn(batch_size, sequence_length, dim) # Random input tensor
# Forward pass
output = model(x) # Perform forward pass using the model
# Print the output shape
print(output) # Print the output tensor
print(output.shape) # Print the shape of the output tensor
License
MIT
Todo
- Implement better gating mechanisms
- Implement better routing algorithm
- Implement better BitFeedForward
- Implement
1 maintainer
Metadata
1 maintainer
Release files for bitmoe 0.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| bitmoe-0.0.2.tar.gz | 4.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| bitmoe-0.0.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 8.4 kB
Release files / bitmoe-0.0.2.tar.gz
| Download URL | bitmoe-0.0.2.tar.gz |
|---|---|
| Size | 4.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dac5c228a9681ced7eb0b17351521a3583fc89aac7275cef338fb1f37532de83
|
|
BLAKE2b-256 checksum How to use checksums |
3923870e948a1d07b9114c05dc2c488f83b2cd355717ec217a4c4c1d087dc8e1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.11.0 Darwin/23.3.0
|
Release files / bitmoe-0.0.2-py3-none-any.whl
| Download URL | bitmoe-0.0.2-py3-none-any.whl |
|---|---|
| Size | 4.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9663a03ce0376d392a6fc58d7ae2c73858914e6c9c47dc3a73696005ccacada3
|
|
BLAKE2b-256 checksum How to use checksums |
ba3e7521eede30ef5288a1a9ba9e07e41d4b2c8d59a8b501f88a83b942915bc6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.2 CPython/3.11.0 Darwin/23.3.0
|
