caominhtien-tools 0.1.4

Add your description here

Tools API - Document Processing & AI Services

A comprehensive FastAPI application providing document conversion, translation, email AI classification, paper tracking, and web search capabilities. Built with modern Python tooling and deployed on Google Cloud Platform.

🚀 Features

Document Processing

• Document Conversion: Convert PDFs, DOCs, EPUBs to HTML, JSON, and Markdown formats
• Document Translation: Translate documents between multiple languages with AI
  • Horizontal Bilingual Format: Side-by-side comparison view (default since v0.1.4)
  • Vertical Bilingual Format: Traditional stacked translation format
  • Non-ASCII Filename Support: Proper RFC 5987 encoding for international characters
• OCR Support: Extract text from images within documents
• Chunking: Process large documents by splitting into manageable parts
• Async Processing: Handle large files with background job processing

AI Services

• Email Classification: AI-powered email categorization and analysis
• Paper Tracking: Automated academic paper discovery and summarization
• Web Search: Multi-engine web search capabilities
• Prompt Improvement: AI-assisted translation prompt optimization

Web Interface

• Flutter Web UI: Modern web interface for document processing
• Background Pattern: Enhanced visual design
• Terraform Deployment: Infrastructure as Code for UI deployment

Infrastructure

• Cloud Storage Integration: Google Cloud Storage for file management
• Background Jobs: Cloud Run Jobs for long-running tasks
• Database Tracking: MySQL with Alembic migrations for job and paper tracking
• Email Notifications: Gmail API integration for job completion alerts
• Terraform Configuration: Infrastructure as Code for all services

📁 Project Structure

app/
├── __init__.py
├── main.py                     # FastAPI application entry point
├── api/                        # API layer
│   ├── __init__.py
│   └── v1/                     # API version 1
│       ├── __init__.py
│       ├── doc_convert.py      # Document conversion endpoints
│       ├── doc_translation.py  # Document translation endpoints
│       ├── emai_ai.py         # Email AI classification endpoints
│       ├── health.py          # Health check endpoints
│       ├── paper_tracking.py  # Paper tracking endpoints
│       └── general_tool/       # General tools (web search)
│           └── web_search.py
├── core/                       # Core application logic
│   ├── __init__.py
│   ├── config/                 # Configuration
│   │   ├── __init__.py
│   │   └── settings.py         # Application settings
│   ├── database.py             # Database connection
│   ├── logging.py              # Logging configuration
│   └── security/               # Security utilities
│       ├── __init__.py
│       └── auth.py             # API key authentication
├── jobs/                       # Background job definitions
│   ├── __init__.py
│   ├── doc_convert_job.py
│   ├── doc_translation_job.py
│   └── summary_papers_job.py
├── models/                     # Data models
│   ├── __init__.py
│   ├── entities/               # Database entities
│   │   ├── __init__.py
│   │   ├── job_execution.py    # Job tracking model
│   │   └── paper_tracking.py   # Paper tracking model
│   ├── enums/                  # Enumeration definitions
│   │   ├── __init__.py
│   │   ├── email_ai.py
│   │   └── general_tool/
│   ├── parses/                 # Response parsers
│   │   ├── __init__.py
│   │   ├── doc_convert.py
│   │   ├── doc_translation.py
│   │   ├── email_ai.py
│   │   └── paper_tracking.py
│   └── schemas/                # Pydantic schemas
│       ├── __init__.py
│       ├── doc_convert.py
│       ├── doc_translation.py
│       ├── email_ai.py
│       ├── general_tool/
│       ├── health.py
│       └── paper_tracking.py
├── prompts/                    # AI prompt templates
│   ├── __init__.py
│   ├── doc_convert.py
│   ├── doc_translation.py
│   ├── email_ai.py
│   └── paper_tracking.py
├── repositories/                # Data access layer
│   ├── __init__.py
│   └── paper_tracking_repository.py
├── services/                    # Business logic layer
│   ├── __init__.py
│   ├── doc_convert/            # Document conversion service
│   │   ├── __init__.py
│   │   ├── core_convert_mixin.py
│   │   ├── doc_convert_service.py
│   │   └── post_processing_mixin.py
│   ├── doc_translation/        # Document translation service
│   │   ├── __init__.py
│   │   ├── core_translation_mixin.py
│   │   ├── doc_translation_service.py
│   │   └── improve_prompt_mixin.py
│   ├── email_ai/               # Email AI service
│   │   ├── __init__.py
│   │   └── email_ai_service.py
│   ├── general_tool/           # General tools
│   │   ├── __init__.py
│   │   └── web_search/
│   ├── health/                 # Health service
│   │   ├── __init__.py
│   │   └── health_service.py
│   ├── paper_tracking/         # Paper tracking service
│   │   ├── __init__.py
│   │   ├── email_template.py
│   │   ├── paper_tracking_service.py
│   │   └── summary_papers_mixin.py
│   └── tracking/               # Job tracking service
├── tests/                      # Test suite
│   ├── __init__.py
│   ├── unit/                   # Unit tests
│   │   └── utils/
│   │       └── test_file_helper.py
│   ├── integration/            # Integration tests
│   ├── api/                    # API tests
│   ├── fixtures/               # Test fixtures
│   └── e2e/                    # End-to-end tests
└── utils/                      # Utility functions
    ├── __init__.py
    ├── concurrency.py
    ├── email.py
    ├── file_helper.py
    ├── gcs_helper.py
    ├── languages.py
    ├── pdf_helper.py
    └── time_helper.py

ui/                             # Flutter Web UI (separate deployment)
├── lib/                        # Flutter source code
├── web/                        # Web-specific files
├── pubspec.yaml                # Flutter dependencies
└── terraform/                  # UI Terraform configuration
    ├── main.tf
    ├── variables.tf
    └── outputs.tf

terraform/                      # Backend Terraform configuration
├── main.tf                     # Main infrastructure
├── variables.tf                # Configuration variables
├── outputs.tf                  # Output values
├── cloud_run.tf                # Cloud Run services and jobs
├── iam.tf                      # IAM roles and permissions
├── secrets.tf                  # Secret Manager configuration
└── build.tf                    # Cloud Build configuration

🛠 Technology Stack

• Framework: FastAPI with Hypercorn ASGI server
• Language: Python 3.12+
• Package Manager: uv (Astral)
• Database: MySQL with SQLAlchemy ORM
• Migrations: Alembic
• AI/ML: Google Gemini AI, LangChain, LangGraph
• Document Processing: Docling, PyPDF, Pandoc, Playwright
• Cloud: Google Cloud Platform (Cloud Run, Cloud Storage, Secret Manager)
• Containerization: Docker with multi-stage builds
• Infrastructure: Terraform for Infrastructure as Code
• Deployment: Cloud Build, GKE, Cloud Run Jobs
• Web UI: Flutter Web with Nginx

🚀 Quick Start

Prerequisites

• Python 3.12+
• uv package manager
• MySQL database
• Google Cloud Platform account
• Docker (for containerized deployment)
• Terraform (for infrastructure management)

Local Development

1. Clone and setup:

git clone <repository-url>
cd tools
uv sync

2. Environment Configuration: Create a .env file with required variables:

# API Configuration
API_TITLE="Tools API"
API_DESCRIPTION="API for document conversion and processing tools"
API_VERSION="v51"
HOST="0.0.0.0"
PORT=8080
DEBUG=false
CORS_ORIGINS=["*"]

# Authentication
API_KEYS=["your-api-key-here"]

# Database Configuration
DB_HOST=localhost
DB_PORT=3306
DB_USER=root
DB_PASSWORD=your_password
DB_NAME=tools_db

# Google Cloud Configuration
GOOGLE_API_KEY=your_google_api_key
UPLOAD_BUCKET=your_gcs_bucket
GOOGLE_APPLICATION_CREDENTIALS=path/to/service-account.json

# Gmail API Configuration
GOOGLE_CREDENTIALS=path/to/google-credentials.json
GOOGLE_TOKEN=path/to/token.json

# Search APIs
GOOGLE_CUSTOM_SEARCH_KEY=your_search_key
GOOGLE_CUSTOM_SEARCH_CSE_ID=your_cse_id
BRAVE_API_KEY=your_brave_key
TAVILY_API_KEY=your_tavily_key

3. Database Setup:

# Start MySQL service
brew services start mysql  # macOS
# or
sudo systemctl start mysql  # Linux

# Create database
mysql -u root -p
CREATE DATABASE tools_db;

4. Run Database Migrations:

alembic upgrade head

5. Start the Server:

python -m app.main

6. Access the API:

• API Documentation: http://localhost:8080/docs
• Health Check: http://localhost:8080/health/
• Simple Health: http://localhost:8080/health/simple

📚 API Endpoints

Health Check

• GET /health/ - Comprehensive health check with system metrics
• GET /health/simple - Simple health check
• GET / - Root endpoint with API information

Document Conversion (/doc-convert)

• POST / - Convert uploaded document to HTML, JSON, and Markdown
• POST /download-zip - Convert document and return as ZIP archive
• POST /convert-epub - Convert EPUB files to HTML or PDF
• POST /post-processing-html - Post-process HTML with formatting options
• POST /generate-upload-url - Generate GCS signed URL for direct upload
• GET /convert-from-storage - Convert document from GCS storage
• GET /convert-from-storage/download-zip - Convert from GCS and return ZIP
• POST /trigger-large-file-conversion - Trigger async conversion for large files

Document Translation (/doc-translation)

• POST / - Translate uploaded document with bilingual format options
  • bilingual_format: Choose between horizontal (default) or vertical layout
• POST /translate-converted-zip - Translate documents from conversion ZIP
• POST /translate-pdf - Translate PDF by converting then translating
• POST /translate-epub - Translate EPUB files
• POST /trigger-large-file-translation - Trigger async translation for large files
• POST /improve-prompt - Get improved translation prompt suggestions

Email AI (/email-ai)

• POST /classify-email - Classify single email using AI
• POST /classify-email/batch - Classify multiple emails in batch
• GET /classify-email/categories - Get available classification categories

Paper Tracking (/paper-tracking)

• POST /fetch-papers - Fetch academic papers by date and type
• GET /summary-papers - Summarize papers with AI
• POST /trigger-summary-job - Trigger background job for paper summarization

Web Search (/search-web)

• GET / - Search the web using multiple search engines

🗄 Database Schema

Job Executions Table

Tracks background job executions with token usage and timing information:

• Job types: DOC_TRANSLATION, DOC_CONVERT, DOC_TRANSLATION_V2, IMPROVED_TRANSLATION_PROMPT, EMAIL_CLASSIFICATION
• Status tracking: PENDING, RUNNING, COMPLETED, FAILED
• Token usage monitoring for AI services
• Email notifications and error handling

Paper Tracking Tables

• paper_tracking: Stores academic paper metadata
• paper_tracking_types: Links papers to tracking types (daily, weekly, monthly)

🔧 Database Migrations

Create Migration

# Auto-generate from model changes
alembic revision --autogenerate -m "Description of changes" --rev-id="N"

# Create empty migration
alembic revision -m "Description of changes" --rev-id="N"

Apply Migrations

# Apply all pending migrations
alembic upgrade head

# Apply specific migration
alembic upgrade N

# Rollback to previous migration
alembic downgrade -1

# Rollback to specific migration
alembic downgrade N

Migration Management

# Check current migration status
alembic current

# Check migration history
alembic history

# Show SQL for migration (without applying)
alembic upgrade head --sql

🚀 Deployment

Terraform (Recommended)

The project uses Terraform for Infrastructure as Code to manage all cloud resources.

Backend Services

1. Initialize Terraform:

cd terraform
terraform init

2. Configure Variables: Create terraform.tfvars:

project_id = "your-gcp-project-id"
region = "us-east1"
api_keys = ["your-api-key"]

3. Deploy Infrastructure:

# Plan deployment
terraform plan

# Apply changes
terraform apply

# Deploy with auto-approve
terraform apply -auto-approve

4. Access Services:

• Backend API: Output from terraform output tools_api_url
• Cloud Run Jobs: Configured for document conversion and translation

Web UI

1. Build and Deploy UI:

cd ui/terraform
terraform init
terraform apply

2. Access UI:

• Web Interface: Output from terraform output ui_url

Google Cloud Run (Legacy)

1. Build and Deploy:

# Deploy API service
./deploy.sh

# Deploy all services (API + Cloud Run Jobs)
./deploy-all.sh

# Deploy with tests
./deploy-all.sh --test

2. Configuration:

• Update version.env for version management
• Configure env.sh for environment variables and secrets
• Set up Google Cloud Secret Manager for sensitive data

Google Kubernetes Engine (GKE)

1. Prerequisites:

gcloud auth login
gcloud config set project your-project-id
gcloud services enable artifactregistry.googleapis.com container.googleapis.com cloudbuild.googleapis.com

2. Deploy:

# Build and deploy to GKE
gcloud builds submit --region=us-east1 --config=cloudbuild.yaml --substitutions=_TAG=v1

3. Access:

gcloud container clusters get-credentials tools-cluster --region=us-east1
kubectl get svc tools-api -w

Docker

# Build image
docker build -t tools-api .

# Run container
docker run -p 8080:8080 --env-file .env tools-api

🔐 Security

• API Key Authentication: All endpoints require valid API keys
• CORS Configuration: Configurable cross-origin resource sharing
• Secret Management: Google Cloud Secret Manager integration
• Input Validation: Pydantic schemas for request/response validation
• Error Handling: Comprehensive error handling and logging
• RFC 5987 Encoding: Proper handling of international characters in filenames

📊 Monitoring & Logging

• Health Checks: Comprehensive system health monitoring
• Structured Logging: Loguru-based logging with JSON output
• Job Tracking: Database-backed job execution monitoring
• Performance Metrics: CPU, memory, and disk usage tracking
• Error Tracking: Detailed error logging with stack traces

🧪 Testing

# Run unit tests
pytest tests/unit

# Run all tests
pytest

# Manual API testing
curl -X GET "http://localhost:8080/health/" \
  -H "X-API-Key: your-api-key"

📖 Documentation

• API Documentation: Auto-generated OpenAPI docs at /docs
• Deployment Guides:
  • GKE Deployment
  • Secrets Management
  • Cloud SQL Setup
  • CI/CD Setup
• Code Documentation: Comprehensive docstrings and type hints

🆕 What's New

Version 0.1.4 (November 2025)

• ✨ Horizontal Bilingual Translation: New side-by-side comparison format (now default)
• 🌐 Non-ASCII Filename Support: RFC 5987 encoding for international characters
• 🎨 Flutter Web UI: Modern web interface with Terraform deployment
• 📋 Enhanced Changelog: Improved release notes generation
• 🐛 Bug Fixes: Fixed backtick preservation in release notes

Version 0.1.3 (October 2025)

• 🔧 RFC 5987 Encoding: Utility for Content-Disposition headers
• 🧪 Test Organization: Restructured test suite (unit/integration/api/e2e)
• 🔄 EPUB Support: Convert and translate EPUB files
• ⚡ Performance: Improved translation speed and reliability

🤝 Contributing

1. Follow the established layer architecture pattern
2. Add new features by extending the appropriate service layer
3. Update API documentation and schemas
4. Add database migrations for schema changes
5. Write tests for new functionality
6. Update CHANGELOG.md for significant changes
7. Test thoroughly before deployment

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Apache License 2.0 is a permissive open-source license that allows you to:

• ✅ Use the software for any purpose
• ✅ Modify and distribute the software
• ✅ Include in proprietary software
• ✅ Patent use and distribution
• ✅ Commercial use

Requirements:

• 📋 Include the original license and copyright notice
• 📋 State any changes made to the original software
• 📋 Include the NOTICE file if present

---

Version: v54.1
Last Updated: 2025-11-08
Maintainer: Cao Minh Tien ([e-mail])