GLLM RAG

Description

A library containing Retrieval-Augmented Generation (RAG) pipeline builders and presets for Gen AI applications.

Installation

Prerequisites

1. Python 3.11+ - Install here
2. Pip (if using Pip) - Install here
3. Poetry 2.1.4+ - Install here
4. Git (if using Git) - Install here
5. For artifact registry installation:
   • Google Cloud CLI - [installed and authenticated]
   • Access to the GDP Labs SDK artifact registry
6. For git installation, access to the GDP Labs SDK github repository

1. Installation from Artifact Registry

Choose one of the following methods to install the package:

Using pip

pip install keyring keyrings.gl-artifactregistry-auth
pip install gllm-rag-binary --index-url https://glsdk.gdplabs.id/gen-ai/simple

Using Poetry

poetry source add --priority=explicit gen-ai https://glsdk.gdplabs.id/gen-ai/simple
poetry config http-basic.gen-ai oauth2accesstoken "$(gcloud auth print-access-token)"
poetry add --source gen-ai gllm-rag-binary

2. Development Installation (Git)

For development purposes, you can install directly from the Git repository:

git clone git@github.com:GDP-ADMIN/gl-sdk.git
cd gl-sdk/libs/gllm-rag

Local Development Setup

Quick Setup (Recommended)

For local development with editable gllm packages, use the provided Makefile:

# Complete setup: installs Poetry, configures auth, installs packages, sets up pre-commit
make setup

The following are the available Makefile targets:

1. make setup - Complete development setup (recommended for new developers)
2. make install-poetry - Install or upgrade Poetry to the latest version
3. make auth - Configure authentication for internal repositories
4. make install - Install all dependencies
5. make install-pre-commit - Set up pre-commit hooks
6. make update - Update dependencies

Manual Development Setup (Legacy)

If you prefer to manage dependencies manually:

1. Go to root folder of gllm-rag module, e.g. cd libs/gllm-rag.
2. Run poetry shell to create a virtual environment.
3. Run poetry lock to create a lock file if you haven't done it yet.
4. Run poetry install to install the gllm-rag requirements for the first time.
5. Run poetry update if you update any dependency module version at pyproject.toml.

Contributing

Please refer to this Python Style Guide to get information about code style, documentation standard, and SCA that you need to use when contributing to this project

Getting Started with Development

1. Clone the repository and navigate to the gllm-rag directory
2. Run make setup to set up your development environment
3. Run which python to get the path to be referenced at Visual Studio Code interpreter path (Ctrl+Shift+P or Cmd+Shift+P)
4. Try running the unit test to see if it's working:

poetry run pytest -s tests/unit_tests/