llama-index readers huggingface fs integration
Project description
Hugging Face FS Loader
pip install llama-index-readers-huggingface-fs
This loader uses Hugging Face Hub's Filesystem API (> 0.14) to load datasets.
Besides the existing load_data function, you may also choose to use
load_dicts and load_df.
Usage
To use this loader, you need to pass in a path to a Hugging Face dataset.
from pathlib import Path
from llama_index.readers.huggingface_fs import HuggingFaceFSReader
# load documents
loader = HuggingFaceFSReader()
documents = loader.load_data("datasets/dair-ai/emotion/data/data.jsonl.gz")
# load dicts
dicts = loader.load_dicts("datasets/dair-ai/emotion/data/data.jsonl.gz")
# load df
df = loader.load_df("datasets/dair-ai/emotion/data/data.jsonl.gz")
This loader is designed to be used as a way to load data into LlamaIndex.
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file llama_index_readers_huggingface_fs-0.3.0.tar.gz.
File metadata
- Download URL: llama_index_readers_huggingface_fs-0.3.0.tar.gz
- Upload date:
- Size: 2.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.3 CPython/3.11.10 Darwin/22.3.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8e9d54b413d95c919f0002ea379aefd8c3b1123c1cc60d8e44aa6b6f0887d4b5 |
|
| MD5 | bd05c87e5fe4a871c0e0d6cd108d4849 |
|
| BLAKE2b-256 | b6daad2011f065946a9a187e11f975b789cdae9fad2b5096bb0b30a45eeeb68d |
File details
Details for the file llama_index_readers_huggingface_fs-0.3.0-py3-none-any.whl.
File metadata
- Download URL: llama_index_readers_huggingface_fs-0.3.0-py3-none-any.whl
- Upload date:
- Size: 2.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.8.3 CPython/3.11.10 Darwin/22.3.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | cf1151f52b0f76f73a7c7623ea9162a89a44e3524a2e30152aa3757b5e3cc11b |
|
| MD5 | 0b7c7e9257a5702536b433bf1855739a |
|
| BLAKE2b-256 | c3b8813dd72c4fa921e53a5c3e9014d59374587baeb57071d1bd678c19ca3413 |
