A wrapper around huggingface datasets, invoking an IPFS model manager.
Project description
IPFS Model Manager
This is a model manager and wrapper for huggingface, and it maintains an index of models from collections of models store retrieved through local/https/s3/ipfs/orbitdb, then maintains a state of which what models are currently accesible, and it will choose what files should be cached through local/s3/ipfs/ based on configuration settings.
How to use
pip install .
look run python3 example.py
for examples of usage.
this is designed to be a drop in replacement, which requires only 2 lines to be changed
In your python script
from ipfs_kit import ipfs_kit
from orbitdb_kit import orbitdb_kit
from ipfs_model_manager import ipfs_model_manager as model_manager
from ipfs_model_manager import load_config()
from ipfs_model_manager import load_collection()
config = load_config()
collection = load_collection()
models = ModelManager()
ready = models.ready()
models.import_config(config)
models.import_collection(collection)
models.state()
or
from ipfs_kit import ipfs_kit
from orbitdb_kit import orbitdb_kit
from ipfs_model_manager import ipfs_model_manager as model_manager
from ipfs_model_manager import load_config()
from ipfs_model_manager import load_collection()
config = load_config()
config.s3cfg = {
"bucket": "cloud",
"endpoint": "https://storage.googleapis.com",
"secret_key": "",
"access_key": ""
}
collection = load_collection()
models = ModelManager()
ready = models.ready()
models.import_config(config)
models.import_collection(collection)
models.state()
IPFS Huggingface Bridge:
for huggingface transformers python library visit: https://github.com/endomorphosis/ipfs_transformers/
for huggingface datasets python library visit: https://github.com/endomorphosis/ipfs_datasets/
for faiss KNN index python library visit: https://github.com/endomorphosis/ipfs_faiss
for transformers.js visit:
https://github.com/endomorphosis/ipfs_transformers_js
for orbitdb_kit nodejs library visit: https://github.com/endomorphosis/orbitdb_kit/
for fireproof_kit nodejs library visit: https://github.com/endomorphosis/fireproof_kit
for ipfs_kit python library visit: https://github.com/endomorphosis/ipfs_kit/
for ipfs_kit_js nodejs library visit: https://github.com/endomorphosis/ipfs_kit_js
for python model manager library visit: https://github.com/endomorphosis/ipfs_model_manager/
for nodejs model manager library visit: https://github.com/endomorphosis/ipfs_model_manager_js/
for nodejs ipfs huggingface scraper with pinning services visit: https://github.com/endomorphosis/ipfs_huggingface_scraper/
for ipfs agents visit: https://github.com/endomorphosis/ipfs_agents/
for ipfs accelerate visit: https://github.com/endomorphosis/ipfs_accelerate/
Author - Benjamin Barber QA - Kevin De Haan
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ipfs_model_manager-0.0.12.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | b35836a08d6fe3f89fb1a3d012dd9941c63c2d02e0ac39daff4980e92035f6bc |
|
MD5 | 22584e4c6ba62f3102202b0a192a65eb |
|
BLAKE2b-256 | 5fc295ed7f18d83645b4e85a93df13cf2b6fba6fe9a9748a05cef8aa53e449a8 |
Hashes for ipfs_model_manager-0.0.12-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 49e6e43970ce8b5f9e2840209b2e12f76d7cc97007a55cbdee9a7a130dca7e5c |
|
MD5 | 715ddbcdc82b6ce8494b2ec6b67b5097 |
|
BLAKE2b-256 | cc33aa509d9a2a649a3820f890a5cc53dbb43c5404133c1b2d473faaa88ffdcd |