llama-index readers opendal_reader integration
Project description
OpenDAL Loaders
Base OpendalReader
This loader parses any file via Apache OpenDAL.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader
. Hence, you may also specify a custom file_extractor
, relying on any of the loaders in this library (or your own)!
Usage
OpendalReader
can read data from any supported storage services including s3
, azblob
, gcs
and so on.
from llama_index import download_loader
OpendalReader = download_loader("OpendalReader")
loader = OpendalReader(
scheme="s3",
bucket="bucket",
path="path/to/data/",
)
documents = loader.load_data()
We also provide Opendal[S3|Gcs|Azblob]Reader
for convenience.
This loader is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for examples.
Azblob Loader
This loader parses any file stored on Azblob.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader
. Hence, you may also specify a custom file_extractor
, relying on any of the loaders in this library (or your own)!
Azblob loader is based on
OpendalReader
.
Usage
from llama_index import download_loader
OpendalAzblobReader = download_loader("OpendalAzblobReader")
loader = OpendalAzblobReader(
container="container",
path="path/to/data/",
endpoint="[endpoint]",
account_name="[account_name]",
account_key="[account_key]",
)
documents = loader.load_data()
This loader is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for examples.
Gcs Loader
This loader parses any file stored on Gcs.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader
. Hence, you may also specify a custom file_extractor
, relying on any of the loaders in this library (or your own)!
Gcs loader is based on
OpendalReader
.
Usage
from llama_index import download_loader
OpendalGcsReader = download_loader("OpendalGcsReader")
loader = OpendalGcsReader(
bucket="bucket",
path="path/to/data/",
endpoint="[endpoint]",
credentials="[credentials]",
)
documents = loader.load_data()
Note: if credentials
is not provided, this loader to try to load from env.
This loader is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for examples.
S3 Loader
This loader parses any file stored on S3. When initializing S3Reader
, you may pass in your AWS Access Key. If none are found, the loader assumes they are stored in ~/.aws/credentials
.
All files are temporarily downloaded locally and subsequently parsed with SimpleDirectoryReader
. Hence, you may also specify a custom file_extractor
, relying on any of the loaders in this library (or your own)!
S3 loader is based on
OpendalReader
.
Usage
from llama_index import download_loader
OpendalS3Reader = download_loader("OpendalS3Reader")
loader = OpendalS3Reader(
bucket="bucket",
path="path/to/data/",
access_key_id="[ACCESS_KEY_ID]",
secret_access_key="[ACCESS_KEY_SECRET]",
)
documents = loader.load_data()
Note: if access_key_id
or secret_access_key
is not provided, this loader to try to load from env.
Possible arguments includes:
endpoint
: Specify the endpoint of s3 service.region
: Specify the region of s3 service.
This loader is designed to be used as a way to load data into LlamaIndex and/or subsequently used as a Tool in a LangChain Agent. See here for examples.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for llama_index_readers_opendal_reader-0.1.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9e42b9c310d78b0fcd034b07d49bd2e8e549dd016703cbda11048913df2159b7 |
|
MD5 | 63f2651f26ef9bb5db9d99152646a36a |
|
BLAKE2b-256 | 459b7ff34cf9b1a5168486d8d01176541934af0a04494f5dba347bbd9ff562f2 |
Hashes for llama_index_readers_opendal_reader-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | caac8d099628eeaf1f0dd4f2fcd22e86d2ae367cc0c5a7b03e390ada58a7d469 |
|
MD5 | b84542979ec50534bd313ab72c1b90ac |
|
BLAKE2b-256 | a63a407b7487a48ddaa6608ddbd94a3e96a20945125361eca2ca7dbb4ed940f6 |