wurzel 2.11.0

Pick, choose and chain your RAG tools in a pythonic way

wurzel

wurzel is the german word for root. So with this framework we provide a combination of the best out of the whole domain of retrieval and beyond.

wurzel is an open-source Python library built to address advanced Extract, Transform, Load (ETL) needs for Retrieval-Augmented Generation (RAG) systems. It is designed to streamline ETL processes while offering essential features like multi-tenancy, cloud-native deployment support, and job scheduling.

The repository includes initial implementations for widely-used frameworks in the RAG ecosystem, such as Qdrant, Milvus, and Hugging Face, providing users with a strong starting point for building scalable and efficient RAG pipelines.

Features

• Advanced ETL Pipelines: Tailored for the specific needs of RAG systems.
• Multi-Tenancy: Easily manage multiple tenants or projects within a single system.
• Cloud-Native Deployment: Designed for seamless integration with Kubernetes, Docker, and other cloud platforms.
• Scheduling Capabilities: Schedule and manage ETL tasks using built-in or external tools.
• Framework Integrations: Pre-built support for popular tools like Qdrant, Milvus, and Hugging Face.
• Type Security: By leveraging capabilities of pydantic and pandera we ensure type security

Installation

To get started with wurzel, install the library using pip:

pip install wurzel

Run a Step (Two Ways)

1. CLI-based Execution

Run a step using the CLI:

wurzel run <step_file_path> --inputs ./data --output ./out

To inspect the step requirements:

wurzel inspect wurzel.<step_path>

2. Programmatic Execution (Python)

Run a step using the snippet below:

import os
from pathlib import Path

from wurzel.step_executor import BaseStepExecutor
from wurzel.steps.manual_markdown import ManualMarkdownStep

# Create input dir and set folder (required by ManualMarkdownStep)
input_dir = Path("./input")
input_dir.mkdir(exist_ok=True)
abs_input = input_dir.resolve()
os.environ["MANUALMARKDOWNSTEP__FOLDER_PATH"] = str(abs_input)
with BaseStepExecutor() as ex:
    ex(ManualMarkdownStep, {abs_input}, Path("./output"))

Building your one step

For detailed instructions and examples on how to use wurzel, please refer to our official documentation.

Code of Conduct

This project has adopted the Contributor Covenant in version 2.1 as our code of conduct. Please see the details in our CODE_OF_CONDUCT.md. All contributors must abide by the code of conduct.

By participating in this project, you agree to abide by its Code of Conduct at all times.

Licensing

This project follows the REUSE standard for software licensing. Each file contains copyright and license information, and license texts can be found in the ./LICENSES folder. For more information visit https://reuse.software/. You can find a guide for developers at https://telekom.github.io/reuse-template/.