llama-index-readers-opendal-reader 0.0.1

llama-index readers opendal_reader integration

OpenDAL Loaders

Base OpendalReader

This loader parses any file via Apache OpenDAL.

All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!

Usage

OpendalReader can read data from any supported storage services including s3, azblob, gcs and so on.

from llama_index import download_loader

OpendalReader = download_loader("OpendalReader")

loader = OpendalReader(
    scheme="s3",
    bucket="bucket",
    path="path/to/data/",
)
documents = loader.load_data()

We also provide Opendal[S3|Gcs|Azblob]Reader for convenience.

---

This loader is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for examples.

Azblob Loader

This loader parses any file stored on Azblob.

All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!

Azblob loader is based on OpendalReader.

Usage

from llama_index import download_loader

OpendalAzblobReader = download_loader("OpendalAzblobReader")

loader = OpendalAzblobReader(
    container="container",
    path="path/to/data/",
    endpoint="[endpoint]",
    account_name="[account_name]",
    account_key="[account_key]",
)
documents = loader.load_data()

---

This loader is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for examples.

Gcs Loader

This loader parses any file stored on Gcs.

All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!

Gcs loader is based on OpendalReader.

Usage

from llama_index import download_loader

OpendalGcsReader = download_loader("OpendalGcsReader")

loader = OpendalGcsReader(
    bucket="bucket",
    path="path/to/data/",
    endpoint="[endpoint]",
    credentials="[credentials]",
)
documents = loader.load_data()

Note: if credentials is not provided, this loader to try to load from env.

---

This loader is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for examples.

S3 Loader

This loader parses any file stored on S3. When initializing S3Reader, you may pass in your AWS Access Key. If none are found, the loader assumes they are stored in ~/.aws/credentials.

All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader. Hence, you may also specify a custom file_extractor, relying on any of the loaders in this library (or your own)!

S3 loader is based on OpendalReader.

Usage

from llama_index import download_loader

OpendalS3Reader = download_loader("OpendalS3Reader")

loader = OpendalS3Reader(
    bucket="bucket",
    path="path/to/data/",
    access_key_id="[ACCESS_KEY_ID]",
    secret_access_key="[ACCESS_KEY_SECRET]",
)
documents = loader.load_data()

Note: if access_key_id or secret_access_key is not provided, this loader to try to load from env.

Possible arguments includes:

• endpoint: Specify the endpoint of s3 service.
• region: Specify the region of s3 service.

---

This loader is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for examples.