PowerScale RAG Connector

The PowerScale RAG Connector is an open-source Python library designed to enhance RAG application performance during data ingestion by skipping files that have already been processed. It leverages PowerScale's unique MetadataIQ capability to identify changed files within the OneFS filesystem and publish this information in an easily consumable format via ElasticSearch.

Developers can integrate the PowerScale RAG Connector directly within a LangChain RAG application as a supported document loader, a LlamaIndex RAG application as a supported reader, or use it independently as a generic Python class.

Workflow

Figure 1: Workflow and integration of how the PowerScale RAG Connector integrates with the LangChain and NVIDIA AI Enterprise Software.

Audience

The intended audience for this document includes software developers, machine learning scientists, and AI developers who will utilize files from PowerScale in the development of a RAG application.

Overview

This guide is divided into two sections: setting up the environment and using the connector. Note that system administration privileges are required for the initial configuration on PowerScale, which may need to be performed by PowerScale administrators.

Terminology

Term	Definition
RAG	Retrieval Augmented Generation. A technique used to take an off the shelf large language model and provide the LLM context to data it has no knowledge of.
LangChain	LangChain is an open-source python and javascript framework used to help developers create RAG applications.
LlamaIndex	LlamaIndex is an open-source Python framework for building RAG applications.
Nvidia NIM Services	Part of Nvidia AI Enterprise, a set of microservices that can optional be used to efficiently chunk and embed files with GPU. The output of this data can be stored in a vector database for a RAG framework to use.
NV-Ingest	An Nvidia NIM microservice that will ingest complex office documents files with tables, and figures, and produce chunks and embedding to be stored in a vector database.
Chunking	The process of splitting the source file into smaller context aware pieces that can be searched and converted into vectors. Example: a chunk could be every paragraph within a large office document
Embedding	Turning a chunk of data into a vector where vector operations such as similarity, can be performed.
MetadataIQ	A new feature in PowerScale OneFS 9.10 that will periodically save filesystem metadata to an external database such as Elasticsearch
lin	Logical inode number. A unique and stable identifier for a file on PowerScale OneFS, used to track file versions across renames and modifications.
PowerScale RAG Connector	An open-source connector that integrates with LangChain or LlamaIndex to improve data ingestion when data resides on PowerScale.

Installation

Basic installation

pip install powerscale-rag-connector

Install with LangChain dependencies

pip install powerscale-rag-connector[langchain]

Install with LlamaIndex dependencies

pip install powerscale-rag-connector[llamaindex]

Full installation (LangChain + LlamaIndex)

pip install powerscale-rag-connector[all]

Installing NVIDIA Ingest Client

The NvIngest examples use the v2 API. For more information refer to the official NV-Ingest documentation.

pip install nv-ingest-client

Usage

The PowerScale RAG Connector can be used in three ways:

1. As a LangChain document loader
2. As a LlamaIndex reader
3. As a standalone Python class

Using as a LangChain Document Loader

from powerscale_rag_connector import PowerScaleDocumentLoader

loader = PowerScaleDocumentLoader(
    es_host_url="https://elasticsearch:9200",
    es_index_name="isi-metadataiq-index.cluster.guid",
    es_api_key="your-encoded-api-key",
    folder_path="/ifs/data"
)

for doc in loader.lazy_load():
    print(doc.metadata["source"], doc.metadata["change_types"])
# Checkpoint is saved automatically when the loop completes.

Each returned Document includes source, snapshot, lin, and change_types in its metadata.

Handling Modified Files

The lin field (logical inode number) is a stable file identifier on OneFS that persists across renames and modifications. Use it to remove stale chunks from your vectorstore before re-ingesting a modified file.

for doc in loader.lazy_load():
    lin = doc.metadata["lin"]
    source = doc.metadata["source"]
    change_types = doc.metadata["change_types"]

    if "ENTRY_MODIFIED" in change_types:
        vectorstore.delete({"lin": lin})

    chunks = process_document(source)  # your chunking/embedding logic
    vectorstore.add(chunks, metadata={"lin": lin, "source": source})
# Checkpoint is saved automatically when the loop completes.

Note on deleted files: MetadataIQ does not emit ENTRY_DELETED events in the current OneFS firmware version. get_deleted_files() raises NotImplementedError accordingly. To handle deletes, you must track lin values in your own application layer and detect when a previously-seen lin stops appearing in results.

Using as a LangChain Unstructured Loader

PowerScaleUnstructuredLoader subclasses langchain-unstructured's UnstructuredLoader, so every partition option of the upstream loader stays available while PowerScale supplies the set of files to parse:

from powerscale_rag_connector import PowerScaleUnstructuredLoader

loader = PowerScaleUnstructuredLoader(
    es_host_url="https://elasticsearch:9200",
    es_index_name="isi-metadataiq-index.cluster.guid",
    es_api_key="your-encoded-api-key",
    folder_path="/ifs/data",
    # chunking_strategy controls how unstructured partitions each file.
    # None (default): each document element is a separate Document object.
    # "basic": merge elements into larger contiguous chunks.
    # "by_title": chunk at section-title boundaries.
    chunking_strategy=None,
)

for doc in loader.lazy_load():
    print(doc.metadata["source"], doc.page_content[:80])
# Checkpoint is saved automatically when the loop completes.

Any additional keyword arguments are forwarded to UnstructuredLoader, so upstream options work unchanged:

loader = PowerScaleUnstructuredLoader(
    es_host_url="https://elasticsearch:9200",
    es_index_name="isi-metadataiq-index.cluster.guid",
    es_api_key="your-encoded-api-key",
    folder_path="/ifs/data",
    chunking_strategy="by_title",
    strategy="hi_res",          # forwarded to unstructured
    languages=["en", "de"],     # forwarded to unstructured
    partition_via_api=False,    # forwarded to UnstructuredLoader
)

Deprecation note: langchain-community's UnstructuredFileLoader (the old loader that accepted a mode= parameter) is deprecated and has been replaced by langchain-unstructured's UnstructuredLoader. PowerScaleUnstructuredLoader uses the new loader. The old mode="single" / mode="elements" parameter does not exist in the new API; use chunking_strategy= instead.

Using as a LlamaIndex Reader

Two LlamaIndex readers are available. PowerScaleSimpleDirectoryReader wraps LlamaIndex's SimpleDirectoryReader filtered to changed files. It supports three mutually exclusive selection scopes: input_dir, input_files, or dataset_name (a MetadataIQ dataset definition in Elasticsearch):

from powerscale_rag_connector import PowerScaleSimpleDirectoryReader

reader = PowerScaleSimpleDirectoryReader(
    es_host_url="https://elasticsearch:9200",
    es_index_name="isi-metadataiq-index.cluster.guid",
    es_api_key="your-encoded-api-key",
    input_dir="/ifs/data",
    # Alternatively use input_files=[...] or dataset_name="my_dataset"
)

for doc in reader.lazy_load_data():
    print(doc.metadata["source"], doc.metadata["change_types"])
# Checkpoint is saved automatically when the loop completes.

PowerScaleUnstructuredReader subclasses LlamaIndex's UnstructuredReader for element-level parsing:

from powerscale_rag_connector import PowerScaleUnstructuredReader

reader = PowerScaleUnstructuredReader(
    es_host_url="https://elasticsearch:9200",
    es_index_name="isi-metadataiq-index.cluster.guid",
    es_api_key="your-encoded-api-key",
    folder_path="/ifs/data",
    mode="elements",   # 'single' keeps the whole file as one Document; 'elements' yields element-level Documents
    languages=["en"],  # optional OCR language hints
)

documents = reader.load_data()
# Checkpoint is saved automatically after load_data() returns.

Because it subclasses UnstructuredReader, the upstream single-file contract still works and bypasses PowerScale entirely:

# PowerScale-driven scan (no `file` argument)
documents = reader.load_data()

# Upstream UnstructuredReader behaviour: parse one explicit file, no MetadataIQ query
documents = reader.load_data(file=Path("/ifs/data/report.pdf"))

Common reader/loader parameters

All loaders/readers that parse file content share a PowerScale-specific raise_on_error switch. The default is False for all of them:

• raise_on_error (default False): If a file fails to parse, the error is logged and processing continues with the next file. The checkpoint advances automatically when the scan completes, meaning failed files are not retried on the next run. Set to True to re-raise the exception and stop the scan; the checkpoint will not advance, so the next run will retry the same set of files.

Using as a Standalone Path Loader

from powerscale_rag_connector import PowerScalePathLoader

# Initialize the loader
loader = PowerScalePathLoader(
    es_host_url="https://elasticsearch:9200",
    es_index_name="isi-metadataiq-index.cluster.guid",
    es_api_key="your-encoded-api-key",
    folder_path="/ifs/data"
)

# Get changed files
for path_info in loader.lazy_load():
    print(path_info)  # (Path, snapshot, lin, change_types)
# Checkpoint is saved automatically when the loop completes.

Examples

Check out the examples directory for complete usage examples:

• Environment Configuration
• PowerScale LangChain Document Loader
• PowerScale LangChain Unstructured Loader
• PowerScale LlamaIndex Simple Directory Reader
• PowerScale LlamaIndex Unstructured Reader
• PowerScale Path Loader
• PowerScale NVIngest LangChain Document Loader
• PowerScale NVIngest LlamaIndex Simple Directory Reader
• PowerScale NVIngest Path Loader

Components

The connector consists of several modules:

• PowerScalePathLoader: Core module for identifying changed files
• PowerScaleDocumentLoader: Custom DocumentLoader for LangChain integration
• PowerScaleUnstructuredLoader: Custom Loader returning Documents processed by LangChain's UnstructuredLoader
• PowerScaleSimpleDirectoryReader: Custom Simple Directory Reader for LlamaIndex integration
• PowerScaleUnstructuredReader: Custom Loader returning Documents processed by LlamaIndex's UnstructuredReader

Requirements

• Python 3.10+
• Elasticsearch client
• PowerScale OneFS 9.10+ with MetadataIQ configured
• LangChain (optional, for LangChain integration)
• LlamaIndex (optional, for LlamaIndex integration)

License

MIT