A wrapper around huggingface datasets, invoking an IPFS model manager.
Project description
IPFS Model Manager
This is a model manager and wrapper for huggingface, and it maintains an index of models from collections of models store retrieved through local/https/s3/ipfs/orbitdb, then maintains a state of which what models are currently accesible, and it will choose what files should be cached through local/s3/ipfs/ based on configuration settings.
How to use
pip install .
look run python3 example.py
for examples of usage.
this is designed to be a drop in replacement, which requires only 2 lines to be changed
In your python script
from ipfs_kit import ipfs_kit
from orbitdb_kit import orbitdb_kit
from ipfs_model_manager import ipfs_model_manager as model_manager
from ipfs_model_manager import load_config()
from ipfs_model_manager import load_collection()
config = load_config()
collection = load_collection()
models = ModelManager()
ready = models.ready()
models.import_config(config)
models.import_collection(collection)
models.state()
or
from ipfs_kit import ipfs_kit
from orbitdb_kit import orbitdb_kit
from ipfs_model_manager import ipfs_model_manager as model_manager
from ipfs_model_manager import load_config()
from ipfs_model_manager import load_collection()
config = load_config()
config.s3cfg = {
"bucket": "cloud",
"endpoint": "https://storage.googleapis.com",
"secret_key": "",
"access_key": ""
}
collection = load_collection()
models = ModelManager()
ready = models.ready()
models.import_config(config)
models.import_collection(collection)
models.state()
IPFS Huggingface Bridge:
for huggingface transformers python library visit: https://github.com/endomorphosis/ipfs_transformers/
for huggingface datasets python library visit: https://github.com/endomorphosis/ipfs_datasets/
for faiss KNN index python library visit: https://github.com/endomorphosis/ipfs_faiss
for transformers.js visit:
https://github.com/endomorphosis/ipfs_transformers_js
for orbitdb_kit nodejs library visit: https://github.com/endomorphosis/orbitdb_kit/
for fireproof_kit nodejs library visit: https://github.com/endomorphosis/fireproof_kit
for ipfs_kit python library visit: https://github.com/endomorphosis/ipfs_kit/
for ipfs_kit_js nodejs library visit: https://github.com/endomorphosis/ipfs_kit_js
for python model manager library visit: https://github.com/endomorphosis/ipfs_model_manager/
for nodejs model manager library visit: https://github.com/endomorphosis/ipfs_model_manager_js/
for nodejs ipfs huggingface scraper with pinning services visit: https://github.com/endomorphosis/ipfs_huggingface_scraper/
for ipfs agents visit: https://github.com/endomorphosis/ipfs_agents/
for ipfs accelerate visit: https://github.com/endomorphosis/ipfs_accelerate/
Author - Benjamin Barber QA - Kevin De Haan
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file ipfs_model_manager-0.0.12.tar.gz
.
File metadata
- Download URL: ipfs_model_manager-0.0.12.tar.gz
- Upload date:
- Size: 35.2 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b35836a08d6fe3f89fb1a3d012dd9941c63c2d02e0ac39daff4980e92035f6bc |
|
MD5 | 22584e4c6ba62f3102202b0a192a65eb |
|
BLAKE2b-256 | 5fc295ed7f18d83645b4e85a93df13cf2b6fba6fe9a9748a05cef8aa53e449a8 |
File details
Details for the file ipfs_model_manager-0.0.12-py3-none-any.whl
.
File metadata
- Download URL: ipfs_model_manager-0.0.12-py3-none-any.whl
- Upload date:
- Size: 35.3 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 49e6e43970ce8b5f9e2840209b2e12f76d7cc97007a55cbdee9a7a130dca7e5c |
|
MD5 | 715ddbcdc82b6ce8494b2ec6b67b5097 |
|
BLAKE2b-256 | cc33aa509d9a2a649a3820f890a5cc53dbb43c5404133c1b2d473faaa88ffdcd |