APIs and tools to work with abstract "models" - files with numpy arrays and metadata. It is possible to publish models, list them. There is a built-in cache. Storage has backends.
Project description
Modelforge
This project is the foundation for sharing machine learning models. It implements a git based
index to maintain the registry, the remote storage where all model files are stored in a
structured, cataloged way. It defines modelforge.Model
, the base class for all the models which
is capable of automatic fetching from the registry. It provides the abstraction over managing
models on disk as well.
Each model receives a UUID and carries other metadata. The underlying file format is ASDF.
Currently, only one registry storage backend is supported: Google Cloud Storage. Our index is stored at src-d/models.
src-d/ml uses modelforge
to make ML on source code accessible
for everybody.
Install
pip3 install modelforge
Usage
The project exposes two interfaces: API and command line.
Configuration
When using Modelforge, it is possible to store default values, whether you are using the package as
an API or with the command line. To do so, simply create a modelforgecfg.py
anywhere in your
project tree or directly in the package
, and modify the following values to suit your needs:
VENDOR = "user" # name of the issuing vendor for models
BACKEND = "gcs" # type of backend to use
BACKEND_ARGS = "bucket='user_bucket.models',credentials='key.json'" # all backend arguments
INDEX_REPO = "https://github.com/user/models" # git repo for the index
CACHE_DIR = "~/.cache/modelforge" # default cache to use for the index
ALWAYS_SIGNOFF = True # whether to add a DCO line on each commit message
Docker image
docker build -t srcd/modelforge .
docker run -it --rm srcd/modelforge --help
Contributions
We use PEP8 with line length 99 and ". All the tests must pass:
python3 -m unittest discover /path/to/modelforge
If you wish to make your model available in src-d/models, please
clone the repository and use the publish
command to upload your model on your fork, then, simply
open a PR. If you are using your own backend, don't forget to add read access to everybody. If you
wish to publish the model our GCS bucket, feel free to open an issue to contact us.
License
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for modelforge-0.8.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6253bd7bcdfac0346e2426b91a14f2576cf233d50b17f0b5c2a22dcf5b3371ed |
|
MD5 | 7f4d2342ba49d8a4c991a678b3632207 |
|
BLAKE2b-256 | 044548ec730f8bc5780b815ff80bcd542493005119f0baa59439ec2811174022 |