A python module and experiment manager for deep learning
Project description
Matรฉ ๐ง your friendly AI project and experiment manager
Matรฉ is a python project, package and experiment manager. Whether you are a seasoned deep learning researcher or just starting out, Matรฉ provides you with the tools to easily add source code and dependencies of models, trainers, and data loaders to your projects. With Matรฉ, you can also evaluate, train, and keep track of your experiments with ease ๐
Features ๐
- Seamless integration with any installed python library such as PyTorch/Lightning, TensorFlow/Keras, JAX/Flax, Huggingface/transformers.
- Easy to use interface to add source code of models, trainers, and data loaders to your projects.
- Support for full customizability and reproducibility of results through the inclusion of dependencies in your project.
- Modular project structure that enforces a clean and organized codebase.
- Fully compatible with python.
- Convenient environment management through the Matรฉ Environment API.
- Support for pip and conda for dependency management.
- Works with Colab.
Table of Contents
Installation ๐
pip install yerbamate
Quick Start โก
Initialize a project
mate init deepnet
This will generate the following empty project structure:
/
|-- models/
| |-- __init__.py
|-- experiments/
| |-- __init__.py
|-- trainers/
| |-- __init__.py
|-- data/
| |-- __init__.py
Install an experiment
To install an experiment, you can use mate install
to install a module and its
dependencies from a github repository. See
docs for more details.
# Short version of GitHub URL https://github.com/oalee/big_transfer/tree/master/big_transfer/experiments/bit
mate install oalee/big_transfer/experiments/bit -yo pip
# Short version of GitHub URL https://github.com/oalee/deep-vision/tree/main/deepnet/experiments/resnet
mate install oalee/deep-vision/deepnet/experiments/resnet -yo pip
Install a module
You can install independant modules such as models, trainers, and data loaders from github projects that follow the Independent modular project structure.
mate install oalee/lightweight-gan/lgan/trainers/lgan
mate install oalee/big_transfer/models/bit -yo pip
mate install oalee/deep-vision/deepnet/models/vit_pytorch -yo pip
mate install oalee/deep-vision/deepnet/trainers/classification -yo pip
Setting up environment
Take a look at Environment API and set up your environment before running your experiments.
Train a model
To train a model, you can use the mate train
command. This command will train
the model with the specified experiment. For example, to train the an experiment
called learn
in the bit
module, you can use the following command:
mate train bit learn
# or alternatively use python
python -m deepnet.experiments.bit.learn train
Project Structure ๐
Deep learning projects can be organized into the following structure with modularity and seperation of concerns in mind. This offers a clean and organized codebase that is easy to maintain and is sharable out-of-the-box.
/
|-- models/
| |-- __init__.py
|-- experiments/
| |-- __init__.py
|-- trainers/
| |-- __init__.py
|-- data/
| |-- __init__.py
Modularity
Modularity is a software design principle that focuses on creating self-contained, reusable and interchangeable components. In the context of a deep learning project, modularity means creating three independent standalone modules for models, trainers and data. This allows for a more organized, maintainable and sharable project structure. The forth module, experiments, is not independent, but rather combines the three modules together to create a complete experiment.
Sample Modular Project Structure
This structure highlights modularity and seperation of concerns. The models
,
data
and trainers
modules are independent and can be used in any project.
The experiments
module is not independent, but rather combines the three
modules together to create a complete experiment.
.
โโโ mate.json
โโโ deepnet
โโโ data
โ โโโ bit
โ โ โโโ fewshot.py
โ โ โโโ __init__.py
โ โ โโโ minibatch_fewshot.py
โ โ โโโ requirements.txt
โ โ โโโ transforms.py
โ โโโ __init__.py
โโโ experiments
โ โโโ bit
โ โ โโโ aug.py
โ โ โโโ dependencies.json
โ โ โโโ __init__.py
โ โ โโโ learn.py
โ โ โโโ requirements.txt
โ โโโ __init__.py
โโโ __init__.py
โโโ models
โ โโโ bit_torch
โ โ โโโ downloader
โ โ โ โโโ downloader.py
โ โ โ โโโ __init__.py
โ โ โ โโโ requirements.txt
โ โ โ โโโ utils.py
โ โ โโโ __init__.py
โ โ โโโ models.py
โ โ โโโ requirements.txt
โ โโโ __init__.py
โโโ trainers
โโโ bit_torch
โ โโโ __init__.py
โ โโโ lbtoolbox.py
โ โโโ logger.py
โ โโโ lr_schduler.py
โ โโโ requirements.txt
โ โโโ trainer.py
โโโ __init__.py
Example Projects ๐
Please check out the transfer learning, vision models, and lightweight gan.
Documentation ๐
Please check out the documentation.
Guides ๐
For more information on modularity, please check out this guide.
For running experiments on Google Colab, please check out this example
Contribution ๐ค
We welcome contributions from the community! Please check out our contributing guide for more information on how to get started.
Contact ๐ค
For questions please contact:
yerba.mate.dl(at)proton.me
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.