Project description

Hugging Face FS Loader

This loader uses Hugging Face Hub's Filesystem API (> 0.14) to load datasets.

Besides the existing load_data function, you may also choose to use load_dicts and load_df.

Usage

To use this loader, you need to pass in a path to a Hugging Face dataset.

from pathlib import Path
from llama_index import download_loader

HuggingFaceFSReader = download_loader("HuggingFaceFSReader")

# load documents
loader = HuggingFaceFSReader()
documents = loader.load_data("datasets/dair-ai/emotion/data/data.jsonl.gz")

# load dicts
dicts = loader.load_dicts("datasets/dair-ai/emotion/data/data.jsonl.gz")

# load df
df = loader.load_df("datasets/dair-ai/emotion/data/data.jsonl.gz")

This loader is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for examples.

Project details

These details have not been verified by PyPI

Release history Release notifications | RSS feed

0.2.0

Aug 22, 2024

0.1.3

Feb 21, 2024

0.1.2

Feb 13, 2024

0.1.1

Feb 12, 2024

0.1.0

Feb 10, 2024

This version

0.0.1

Feb 4, 2024

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

llama_index_readers_huggingface_fs-0.0.1.tar.gz (2.6 kB view details)

Uploaded Feb 4, 2024 Source

Built Distribution

llama_index_readers_huggingface_fs-0.0.1-py3-none-any.whl (2.8 kB view details)

Uploaded Feb 4, 2024 Python 3

File details

Details for the file llama_index_readers_huggingface_fs-0.0.1.tar.gz.

File metadata

Download URL: llama_index_readers_huggingface_fs-0.0.1.tar.gz
Upload date: Feb 4, 2024
Size: 2.6 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: poetry/1.7.1 CPython/3.10.13 Darwin/23.0.0

File hashes

Hashes for llama_index_readers_huggingface_fs-0.0.1.tar.gz
Algorithm	Hash digest
SHA256	`10a3adf380ab77072e13ab014399b19650c8196d89be82fb08689f304021fa00`
MD5	`bdba72a36e6b20b34cadac5f03179120`
BLAKE2b-256	`dcd2a899ca0ad293b30b134ca51b28849e8ced030066f9c1c9be534bd1996d35`

See more details on using hashes here.

File details

Details for the file llama_index_readers_huggingface_fs-0.0.1-py3-none-any.whl.

File metadata

Download URL: llama_index_readers_huggingface_fs-0.0.1-py3-none-any.whl
Upload date: Feb 4, 2024
Size: 2.8 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: poetry/1.7.1 CPython/3.10.13 Darwin/23.0.0

File hashes

Hashes for llama_index_readers_huggingface_fs-0.0.1-py3-none-any.whl
Algorithm	Hash digest
SHA256	`f9fed45dda34a24939f0653c98e4fb91970123518adc3cc06b0ba81f2d539d81`
MD5	`62a8af07731bd4ad8a51cd3fe20464c4`
BLAKE2b-256	`c8a3879d4db85a32a3e494f89da02de1bc90b6165467698a1ff771bac0d2eeb4`