A wrapper around huggingface transformers, invoking an IPFS model manager.
Project description
IPFS Huggingface Bridge
for transformers.js visit:
https://github.com/endomorphosis/ipfs_transformers_js
for huggingface datasets python library visit: https://github.com/endomorphosis/ipfs_datasets
for faiss KNN index python library visit: https://github.com/endomorphosis/ipfs_faiss
for orbitdb_kit nodejs library visit: https://github.com/endomorphosis/orbitdb_kit/
for python model manager library visit: https://github.com/endomorphosis/ipfs_model_manager/
for nodejs model manager library visit: https://github.com/endomorphosis/ipfs_model_manager_js/
for nodejs ipfs huggingface scraper with pinning services visit: https://github.com/endomorphosis/ipfs_huggingface_scraper/
for ipfs huggingface agents visit: https://github.com/endomorphosis/ipfs_agents
for ipfs huggingface accelerate visit: https://github.com/endomorphosis/ipfs_accelerate
for ipfs_kit visit: https://github.com/endomorphosis/ipfs_kit
for ipfs_kit_js visit: https://github.com/endomorphosis/ipfs_kit_js
Author - Benjamin Barber QA - Kevin De Haan
About
This is a model manager and wrapper for huggingface, looks up a index of models from an collection of models, and will download a model from either https/s3/ipfs, depending on which source is the fastest.
How to use
pip install .
look run python3 example.py
for examples of usage.
this is designed to be a drop in replacement, which requires only 2 lines to be changed
In your python script
from transformers import AutoModel
from ipfs_transformers import AutoModel
model = AutoModel.from_auto_download("bge-small-en-v1.5")
or
from transformers import AutoModel
from ipfs_transformers import AutoModel
model = AutoModel.from_ipfs("QmccfbkWLYs9K3yucc6b3eSt8s8fKcyRRt24e3CDaeRhM1")
or to use with with s3 caching
from transformers import AutoModel
from ipfs_transformers import AutoModel
model = T5Model.from_auto_download(
model_name="google-bert/t5_11b_trueteacher_and_anli",
s3cfg={
"bucket": "cloud",
"endpoint": "https://storage.googleapis.com",
"secret_key": "",
"access_key": ""
}
)
To scrape huggingface
with interactive prompt:
node scraper.js [source] [model name]
node scraper.js
import a model already defined:
node scraper.js hf "modelname" (as defined in your .json files)
import all models previously defined:
node scraper.js hf
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ipfs_transformers-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9b8ae44993136744ebc3683c20709a1d0b52e1340fd79498ba3523b413fd2f4 |
|
MD5 | 1eceb9a91781cf1f19f8dd3288c2b62a |
|
BLAKE2b-256 | 71132d04e36ec6a0df993b8b320a27352c7fce0484e855794985bafe84d3d58c |