wizit_context_ingestor

A powerful document processing and ingestion system that leverages AI services for document transcription, analysis, and semantic chunking.

Features

Document transcription using AWS and Google Cloud AI services
Semantic chunking of documents for better context understanding
Vector storage integration with PostgreSQL
Support for both local and cloud storage (S3)
Synthetic data generation capabilities
RAG (Retrieval-Augmented Generation) implementation

Prerequisites

Python 3.11 or higher
Poetry for dependency management
AWS credentials (for AWS services)
Google Cloud credentials (for GCP services)
PostgreSQL database (for vector storage)
Supabase account (for data storage)

Installation

Clone the repository:

git clone https://github.com/yourusername/mega-ingestor.git
cd mega-ingestor

Install dependencies using Poetry:

poetry install

Set up your environment variables by copying the example.env file:

cp example.env .env

Fill in your environment variables in the .env file with your credentials and configuration.

Usage

The project provides several main functionalities:

Document Transcription

from main import transcribe_document

# Transcribe a document using AWS services
transcribe_document("your-document.pdf")

# Transcribe a document using Google Cloud services
cloud_transcribe_document("your-document.pdf")

Context Chunking

from main import context_chunks_in_document

# Get semantic chunks from a document
context_chunks_in_document("your-document.pdf")

Running Memory Profiler

To run the memory profiler, use the following command:

python -m memray run test_redis.py

Project Structure

mega-ingestor/
├── src/
│   ├── application/
│   ├── infra/
│   └── ...
├── data/
├── credentials/
├── main.py
├── app.py
└── pyproject.toml

Dependencies

llama-parse
langchain-experimental
langchain-google-vertexai
pymupdf
supabase
vecs
langchain-postgres
boto3
langchain-aws

GENERATE THE PACKAGE WITH POETRY

    poetry build

PUBLISH PACKAGE

    poetry config repositories.tbbcmegaingestor https://aws:$CODEARTIFACT_AUTH_TOKEN@tbbc-mega-ingestor-411728455297.d.codeartifact.us-east-1.amazonaws.com/pypi/tbbc-mega-ingestor-lib/

    export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain tbbc-mega-ingestor --domain-owner 411728455297 --region us-east-1 --query authorizationToken --output text --profile <your-profile>`

Finally

    poetry publish -r tbbcmegaingestor

USAGE

For transcriptions

----- TODO --- You can provide number of retries and a transcription quality threshold

License

This project is licensed under the Apache License - see the LICENSE file for details.

TODO

Do not transcribe logos
Support for more cloud providers

Authors

(Daniel Quesada)[https://github.com/daquesada] (Jeison Patiño)[https://github.com/jeison-patino] (Javier Fernandez)[https://github.com/javimaufermu] (Esteban Cerón)[https://github.com/estebance]

Metadata

Release files for wizit-context-ingestor 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for wizit-context-ingestor 0.5.0
File	Size	Uploaded
wizit_context_ingestor-0.5.0.tar.gz	24.4 kB	Feb 26, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for wizit-context-ingestor 0.5.0
File	Interpreter	ABI	Platform	Reset
wizit_context_ingestor-0.5.0-py3-none-any.whl 40.8 kB Feb 26, 2026	Python 3	none	any	Details

Total release size: 65.2 kB

Release files / wizit_context_ingestor-0.5.0.tar.gz

Download URL	wizit_context_ingestor-0.5.0.tar.gz
Size	24.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`f37d06197384d96cfa4df7bab829d70e07ab1d5ffa8b03fff90c5058a0574b95`
BLAKE2b-256 checksum How to use checksums	`9bef12d28cae7ec7f82603476d0d474250e673049d14ac7bc582aba6433a8269`
Upload date	Feb 26, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.7.9`

Release files / wizit_context_ingestor-0.5.0-py3-none-any.whl

Download URL	wizit_context_ingestor-0.5.0-py3-none-any.whl
Size	40.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`d9ad44c1471e3bb6163b24203a9af001728607d2841a090558437a9c1b0dadfa`
BLAKE2b-256 checksum How to use checksums	`17b90cf8a58531a1fd178850e9a798ec97d9d8d4a0b59766cc9e20fa595c0e7d`
Upload date	Feb 26, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.7.9`