minio-weaviate-langchain 0.1.0

A FastAPI application with Langchain, Weaviate, and MinIO integrations

This will guide users on how to deploy the application using Docker, which simplifies the setup process by handling dependencies and configurations through Docker containers. Here's an updated version of the README.md with Docker Compose deployment instructions:

README.md

My FastAPI Application with Langchain, Weaviate, and MinIO Integrations

This FastAPI application demonstrates an integrated text processing pipeline using Weaviate and MinIO. It's designed to efficiently process, store, and retrieve documents through a Dockerized environment.

Features

• Process documents from a MinIO bucket.
• Use OpenAI's LangChain for advanced text processing.
• Index and manage documents in Weaviate.
• Easy deployment with Docker Compose.

Requirements

• Docker
• Docker Compose

Installation and Deployment

1. Clone the Repository:

   git clone https://github.com/Cdaprod/minio-weaviate-langchain
   cd minio-weaviate-langchain
   

2. Start the Application with Docker Compose:

   Run the following command to start all services defined in your docker-compose.yaml:

   docker-compose up
   

   This command will build and start containers for your FastAPI app, Weaviate, and MinIO.

Proof of Concept

The application showcases the following workflow:

1. Load Documents from MinIO: The application retrieves documents stored in a MinIO bucket.

2. Document Processing: Documents are processed via LangChain's OpenAI integration.

3. Indexing in Weaviate: Processed documents are indexed in Weaviate for full-text search and data retrieval.

4. API Endpoints:

   • POST /process_documents: Processes and indexes documents from MinIO into Weaviate.
   • GET /: Health check endpoint.
   • POST /index_from_minio: Indexes documents from MinIO into Weaviate.
   • POST /query: Queries documents in Weaviate.
   • POST /update/{uuid}: Updates a document in Weaviate.
   • DELETE /delete/{uuid}: Deletes a document from Weaviate.

Example Usage

• Index Documents from MinIO:

  import requests
  response = requests.post('http://localhost:8000/index_from_minio')
  print(response.json())
  

• Query Documents in Weaviate:

  query = '{"query": "your_query_here"}'
  response = requests.post('http://localhost:8000/query', json=query)
  print(response.json())
  

Contributing

Contributions are welcome! Fork the repository, make your changes, and submit a pull request.

Further Development for Notebook Environment

To adapt your FastAPI application with Weaviate and MinIO integration for use in a Jupyter notebook, you'll need to structure it as a Python package that can be installed and used interactively. The structure of the package should facilitate easy installation and usage within a notebook environment.

Here's a basic outline of steps to restructure your application:

1. Organize Your Application as a Package

Your project directory might look something like this:

minio-weaviate-langchain/
    my_fastapi_lib/
        __init__.py
        main.py
        weaviate_operations.py
        minio_operations.py
        # ... other modules ...
    setup.py
    README.md
    LICENSE

In minio-weaviate-langchain/my_fastapi_lib/__init__.py, you can import the main components:

from .main import app
from .weaviate_operations import WeaviateOperations
from .minio_operations import load_documents_from_minio

2. Setup.py for Installation

Your setup.py allows the package to be installed via pip. Here's an example:

from setuptools import setup, find_packages

setup(
    name='my_fastapi_lib',
    version='0.1.0',
    packages=find_packages(),
    include_package_data=True,
    install_requires=[
        'fastapi',
        'uvicorn',
        'weaviate-client',
        'minio',
        'langchain',
        # ... other dependencies ...
    ],
)

3. Using the Library in a Jupyter Notebook

Once your library is structured and installable, you can create a Jupyter notebook to demonstrate its usage. For example:

Cell 1: Install the library

!pip install git+https://github.com/yourusername/my_fastapi_lib.git

Cell 2: Import and Initialize Components

from my_fastapi_lib import WeaviateOperations, load_documents_from_minio

# Setup Weaviate and MinIO clients
weaviate_ops = WeaviateOperations(WEAVIATE_ENDPOINT)
minio_documents = load_documents_from_minio(MINIO_BUCKET, MINIO_ENDPOINT, MINIO_ACCESS_KEY, MINIO_SECRET_KEY)

Cell 3: Use the Library

# Example operation
for doc in minio_documents:
    processed_doc = process_document(doc)  # Define this function as per your processing logic
    weaviate_ops.index_document(MINIO_BUCKET, "DocumentName", processed_doc)

4. Documentation and Examples

Provide clear instructions and examples in your README.md and include Jupyter notebooks that demonstrate the use of your library.

5. Running the FastAPI App

Since the FastAPI app is designed to be run as a server, it might not be directly applicable in a Jupyter notebook for interactive use. Instead, focus on demonstrating how to interact with the server (which could be running locally or remotely) using requests or similar in the notebook.

Note

Adapting a FastAPI application for use as a library in a Jupyter notebook involves a shift in focus. The interactive notebook environment is more suited for demonstrating API calls to the running FastAPI server, rather than running the server itself within the notebook.