Skip to main content

A wrapper around huggingface datasets, invoking an IPFS model manager.

Project description

IPFS Model Manager

This is a model manager and wrapper for huggingface, and it maintains an index of models from collections of models store retrieved through local/https/s3/ipfs/orbitdb, then maintains a state of which what models are currently accesible, and it will choose what files should be cached through local/s3/ipfs/ based on configuration settings.

How to use

pip install .

look run python3 example.py for examples of usage.

this is designed to be a drop in replacement, which requires only 2 lines to be changed

In your python script

from ipfs_kit import ipfs_kit
from orbitdb_kit import orbitdb_kit 
from ipfs_model_manager import ipfs_model_manager as model_manager
from ipfs_model_manager import load_config()
from ipfs_model_manager import load_collection()
config = load_config()
collection = load_collection()
models = ModelManager()
ready = models.ready()
models.import_config(config)
models.import_collection(collection)
models.state()

or

from ipfs_kit import ipfs_kit
from orbitdb_kit import orbitdb_kit 
from ipfs_model_manager import ipfs_model_manager as model_manager
from ipfs_model_manager import load_config()
from ipfs_model_manager import load_collection()
config = load_config()
config.s3cfg = {
        "bucket": "cloud",
        "endpoint": "https://storage.googleapis.com",
        "secret_key": "",
        "access_key": ""
    }
collection = load_collection()
models = ModelManager()
ready = models.ready()
models.import_config(config)
models.import_collection(collection)
models.state()

IPFS Huggingface Bridge:

for huggingface transformers python library visit: https://github.com/endomorphosis/ipfs_transformers/

for huggingface datasets python library visit: https://github.com/endomorphosis/ipfs_datasets/

for faiss KNN index python library visit: https://github.com/endomorphosis/ipfs_faiss

for transformers.js visit:
https://github.com/endomorphosis/ipfs_transformers_js

for orbitdb_kit nodejs library visit: https://github.com/endomorphosis/orbitdb_kit/

for fireproof_kit nodejs library visit: https://github.com/endomorphosis/fireproof_kit

for ipfs_kit python library visit: https://github.com/endomorphosis/ipfs_kit/

for ipfs_kit_js nodejs library visit: https://github.com/endomorphosis/ipfs_kit_js

for python model manager library visit: https://github.com/endomorphosis/ipfs_model_manager/

for nodejs model manager library visit: https://github.com/endomorphosis/ipfs_model_manager_js/

for nodejs ipfs huggingface scraper with pinning services visit: https://github.com/endomorphosis/ipfs_huggingface_scraper/

for ipfs agents visit: https://github.com/endomorphosis/ipfs_agents/

for ipfs accelerate visit: https://github.com/endomorphosis/ipfs_accelerate/

Author - Benjamin Barber QA - Kevin De Haan

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ipfs_model_manager-0.0.12.tar.gz (35.2 MB view details)

Uploaded Source

Built Distribution

ipfs_model_manager-0.0.12-py3-none-any.whl (35.3 MB view details)

Uploaded Python 3

File details

Details for the file ipfs_model_manager-0.0.12.tar.gz.

File metadata

  • Download URL: ipfs_model_manager-0.0.12.tar.gz
  • Upload date:
  • Size: 35.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for ipfs_model_manager-0.0.12.tar.gz
Algorithm Hash digest
SHA256 b35836a08d6fe3f89fb1a3d012dd9941c63c2d02e0ac39daff4980e92035f6bc
MD5 22584e4c6ba62f3102202b0a192a65eb
BLAKE2b-256 5fc295ed7f18d83645b4e85a93df13cf2b6fba6fe9a9748a05cef8aa53e449a8

See more details on using hashes here.

File details

Details for the file ipfs_model_manager-0.0.12-py3-none-any.whl.

File metadata

File hashes

Hashes for ipfs_model_manager-0.0.12-py3-none-any.whl
Algorithm Hash digest
SHA256 49e6e43970ce8b5f9e2840209b2e12f76d7cc97007a55cbdee9a7a130dca7e5c
MD5 715ddbcdc82b6ce8494b2ec6b67b5097
BLAKE2b-256 cc33aa509d9a2a649a3820f890a5cc53dbb43c5404133c1b2d473faaa88ffdcd

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page