A wrapper around huggingface datasets, invoking an IPFS model manager.
Project description
IPFS Huggingface Datasets
This is a model manager and wrapper for huggingface, looks up a index of models from an collection of models, and will download a model from either https/s3/ipfs, depending on which source is the fastest.
How to use
pip install .
look run python3 example.py
for examples of usage.
this is designed to be a drop in replacement, which requires only 2 lines to be changed
In your python script
from datasets import load_dataset
from ipfs_datasets import load_dataset
dataset = load_dataset.from_auto_download("bge-small-en-v1.5")
or
from datasets import load_dataset
from ipfs_datasets import load_dataset
dataset = load_dataset.from_ipfs("QmccfbkWLYs9K3yucc6b3eSt8s8fKcyRRt24e3CDaeRhM1")
or to use with with s3 caching
from datasets import load_dataset
from ipfs_datasets import load_dataset
dataset = load_dataset.from_auto_download(
dataset_name="common-crawl",
s3cfg={
"bucket": "cloud",
"endpoint": "https://storage.googleapis.com",
"secret_key": "",
"access_key": ""
}
)
IPFS Huggingface Bridge:
for transformers python library visit: https://github.com/endomorphosis/ipfs_transformers/
for transformers js client visit:
https://github.com/endomorphosis/ipfs_transformers_js/
for orbitdb_kit nodejs library visit: https://github.com/endomorphosis/orbitdb_kit/
for fireproof_kit nodejs library visit: https://github.com/endomorphosis/fireproof_kit
for Faiss KNN index python library visit: https://github.com/endomorphosis/ipfs_faiss/
for python model manager library visit: https://github.com/endomorphosis/ipfs_model_manager/
for nodejs model manager library visit: https://github.com/endomorphosis/ipfs_model_manager_js/
for nodejs ipfs huggingface scraper with pinning services visit: https://github.com/endomorphosis/ipfs_huggingface_scraper/
Author - Benjamin Barber QA - Kevin De Haan
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ipfs_embeddings_py-0.0.12.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 24c9def459c960e9c8dc25e652094a975a01433a88107f723a2a5b2ccb3e7c8e |
|
MD5 | a874c7148b8bebbba0103ef65ad74510 |
|
BLAKE2b-256 | 67d45d511f2d1b5f261f5594b750a44b011be210944e6eeace48b7615504b99d |
Hashes for ipfs_embeddings_py-0.0.12-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 86bfccc7e49942ac3d1afbefa63e183c5d1154c3ddfc0bd7f7361ab72bad5d21 |
|
MD5 | ef8998cc2c22481f97614be0638b6053 |
|
BLAKE2b-256 | 8f0c8bdbe1ce5822d90d120709e37f1b7719b57d8b523f7530cbe6205bd2d56a |