evolvishub-minio-adapter 0.1.0

A professional MinIO adapter library with configuration management

Evolvishub MinIO Adapter

A production-ready, enterprise-grade Python library for interacting with MinIO object storage. Designed for modern cloud-native environments with comprehensive configuration management, monitoring, health checks, and deployment support.

Developed by Evolvis.ai - Advancing AI-driven solutions for modern enterprises.

✨ Key Features

🚀 Core Functionality

• Synchronous & Asynchronous clients - Full async/await support for high-performance applications
• Complete MinIO operations - Upload, download, list, delete, presigned URLs, bucket management
• Robust error handling - Comprehensive exception management with detailed error messages
• Type safety - Full type hints and mypy compatibility
• Memory efficient - Streaming support for large files

⚙️ Configuration Management

• Multiple formats - YAML, INI, and environment variables
• Multi-environment support - Development, staging, production configurations
• Secure credential handling - Environment variable overrides for sensitive data
• Section-based configs - INI files with multiple environment sections
• Validation - Built-in configuration validation and error reporting

🏗️ Production Ready

• Docker support - Multi-stage Dockerfile with security hardening
• Kubernetes ready - Complete K8s manifests with best practices
• Health checks - Built-in liveness and readiness probes
• Structured logging - JSON logging for production environments
• Security hardened - Non-root containers, RBAC, network policies
• Auto-scaling - Horizontal Pod Autoscaler configuration
• Monitoring - Prometheus-compatible metrics endpoints

🧪 Developer Experience

• Comprehensive testing - 100% test coverage with pytest
• CI/CD ready - GitHub Actions, tox, and quality tools
• Documentation - Complete examples and deployment guides
• Development tools - Black, isort, mypy, flake8 configurations

📦 Installation

From PyPI (Recommended)

pip install evolvishub-minio-adapter

From Source

git clone https://github.com/yourusername/evolvishub-minio-adapter.git
cd evolvishub-minio-adapter
pip install -r requirements.txt
pip install -e .

Development Installation

pip install -r requirements-dev.txt
pip install -e .

⚙️ Configuration

The library supports multiple configuration methods for maximum flexibility across different environments.

📋 YAML Configuration

Create a YAML configuration file (recommended for development):

# config/minio_config.yaml
endpoint: "localhost:9000"
access_key: "minioadmin"
secret_key: "minioadmin"
secure: true
region: "us-east-1"
bucket_name: "my-bucket"

📄 INI Configuration

Use INI files for multi-environment configurations:

# config/minio_config.ini
[minio]
endpoint = localhost:9000
access_key = minioadmin
secret_key = minioadmin
secure = true
region = us-east-1
bucket_name = my-bucket

# Production environment
[minio_production]
endpoint = minio.example.com:9000
access_key = ${MINIO_ACCESS_KEY}
secret_key = ${MINIO_SECRET_KEY}
secure = true
region = us-west-2
bucket_name = production-bucket

# Development environment
[minio_development]
endpoint = localhost:9000
access_key = dev_access_key
secret_key = dev_secret_key
secure = false
region = us-east-1
bucket_name = dev-bucket

🔐 Environment Variables

Override any configuration value with environment variables (recommended for production):

export MINIO_ENDPOINT="minio.production.com:9000"
export MINIO_ACCESS_KEY="your-access-key"
export MINIO_SECRET_KEY="your-secret-key"
export MINIO_SECURE="true"
export MINIO_REGION="us-west-2"
export MINIO_BUCKET_NAME="production-bucket"

🔧 Configuration Priority

Configuration values are resolved in the following order (highest to lowest priority):

1. Environment variables (highest priority)
2. Configuration file values (YAML/INI)
3. Default values (lowest priority)

🚀 Quick Start

Basic Synchronous Usage

from minio_adapter import MinioClient, MinioConfig

# Load configuration from YAML
config = MinioConfig.from_yaml("config/minio_config.yaml")

# Or load from INI with environment-specific section
config = MinioConfig.from_ini("config/minio_config.ini", section="minio_production")

# Initialize client
client = MinioClient(config)

# Upload a file
result = client.upload_file(
    file_path="path/to/file.txt",
    object_name="file.txt",
    content_type="text/plain",
    metadata={"author": "John Doe", "version": "1.0"}
)
print(f"Uploaded: {result}")

# Download a file
client.download_file(
    object_name="file.txt",
    file_path="downloaded_file.txt"
)

# List objects with filtering
objects = client.list_objects(prefix="documents/", recursive=True)
for obj in objects:
    print(f"📄 {obj['object_name']} ({obj['size']} bytes)")

# Generate presigned URL for sharing
url = client.get_presigned_url(
    object_name="file.txt",
    expires=3600,  # 1 hour
    method="GET"
)
print(f"🔗 Share URL: {url}")

Asynchronous Usage

import asyncio
from minio_adapter import AsyncMinioClient, MinioConfig

async def main():
    # Load configuration
    config = MinioConfig.from_yaml("config/minio_config.yaml")

    # Initialize async client
    async_client = AsyncMinioClient(config)

    # Upload file asynchronously
    await async_client.upload_file("large_file.zip", "backups/large_file.zip")

    # List objects asynchronously
    objects = await async_client.list_objects(prefix="backups/")

    # Process multiple operations concurrently
    tasks = [
        async_client.upload_file(f"file_{i}.txt", f"batch/file_{i}.txt")
        for i in range(10)
    ]
    results = await asyncio.gather(*tasks)
    print(f"✅ Uploaded {len(results)} files")

# Run async code
asyncio.run(main())

Advanced Operations

from minio_adapter import MinioClient, MinioConfig

client = MinioClient(config)

# Upload data directly from memory
data = b"Hello, World! This is in-memory data."
client.upload_data(
    data=data,
    object_name="memory/hello.txt",
    content_type="text/plain",
    metadata={"source": "memory", "author": "system"}
)

# Batch operations with error handling
files_to_upload = ["doc1.pdf", "doc2.pdf", "doc3.pdf"]
for file_path in files_to_upload:
    try:
        result = client.upload_file(file_path, f"documents/{file_path}")
        print(f"✅ Uploaded: {result}")
    except Exception as e:
        print(f"❌ Failed to upload {file_path}: {e}")

# Bucket policy management
try:
    # Get current bucket policy
    current_policy = client.get_bucket_policy()
    print("Current policy:", current_policy)

    # Set new bucket policy for public read access
    public_read_policy = {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {"AWS": "*"},
                "Action": ["s3:GetObject"],
                "Resource": ["arn:aws:s3:::my-bucket/*"]
            }
        ]
    }
    client.set_bucket_policy(public_read_policy)
    print("✅ Bucket policy updated")

except Exception as e:
    print(f"❌ Policy operation failed: {e}")

🏥 Health Checks & Monitoring

The library includes built-in health check capabilities for production deployments:

from minio_adapter import create_health_checker, setup_logging

# Setup structured logging
setup_logging(log_level="INFO", log_format="json")

# Create health checker
health_checker = create_health_checker("config/minio_config.yaml")

# Check overall health
health_status = health_checker.check_health()
print(f"Healthy: {health_status.healthy}")
print(f"Details: {health_status.details}")

# Kubernetes-style health checks
liveness = health_checker.check_liveness()    # Basic functionality
readiness = health_checker.check_readiness()  # Ready to serve requests

# Use in web frameworks (FastAPI example)
from fastapi import FastAPI, HTTPException

app = FastAPI()

@app.get("/health/live")
async def liveness_check():
    status = health_checker.check_liveness()
    if not status.healthy:
        raise HTTPException(status_code=503, detail=status.error)
    return {"status": "alive", "timestamp": status.timestamp}

@app.get("/health/ready")
async def readiness_check():
    status = health_checker.check_readiness()
    if not status.healthy:
        raise HTTPException(status_code=503, detail=status.error)
    return {"status": "ready", "details": status.details}

🐳 Docker Deployment

Quick Start with Docker Compose

# Clone the repository
git clone https://github.com/yourusername/evolvishub-minio-adapter.git
cd evolvishub-minio-adapter

# Start MinIO and the adapter
docker-compose up -d

# View logs
docker-compose logs -f minio-adapter

# Run example application
docker-compose --profile example up example-app

Custom Docker Build

# Build the image
docker build -t evolvishub/minio-adapter:latest .

# Run with custom configuration
docker run -d \
  --name minio-adapter \
  -e MINIO_ENDPOINT="your-minio:9000" \
  -e MINIO_ACCESS_KEY="your-access-key" \
  -e MINIO_SECRET_KEY="your-secret-key" \
  -v $(pwd)/config:/app/config:ro \
  evolvishub/minio-adapter:latest

☸️ Kubernetes Deployment

Deploy to Kubernetes with production-ready configurations:

# Deploy complete stack
kubectl apply -f k8s/

# Check deployment status
kubectl get pods -n minio-adapter

# View logs
kubectl logs -f deployment/minio-adapter -n minio-adapter

# Check health
kubectl exec -it deployment/minio-adapter -n minio-adapter -- \
  python -c "from minio_adapter import create_health_checker; print(create_health_checker().check_health())"

The Kubernetes deployment includes:

• 🔒 Security: RBAC, NetworkPolicies, non-root containers
• 📈 Scaling: HorizontalPodAutoscaler, PodDisruptionBudget
• 🏥 Health: Liveness and readiness probes
• 📊 Monitoring: Prometheus-compatible metrics
• 🔧 Configuration: ConfigMaps and Secrets management

🛠️ Development

Setting Up Development Environment

# Clone repository
git clone https://github.com/yourusername/evolvishub-minio-adapter.git
cd evolvishub-minio-adapter

# Install development dependencies
pip install -r requirements-dev.txt
pip install -e .

# Run tests
pytest tests/ --cov=minio_adapter

# Run quality checks
black minio_adapter tests examples
isort minio_adapter tests examples
flake8 minio_adapter tests examples
mypy minio_adapter

# Run all checks with tox
tox

Testing

# Run all tests
pytest tests/ -v

# Run with coverage
pytest tests/ --cov=minio_adapter --cov-report=html

# Run specific test
pytest tests/test_minio_client.py::TestMinioClient::test_upload_file -v

# Run async tests
pytest tests/test_async_minio_client.py -v

📚 Documentation

• Deployment Guide - Complete deployment instructions
• API Reference - Detailed API documentation
• Examples - Code examples and use cases
• Configuration Guide - Configuration options
• Troubleshooting - Common issues and solutions

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Quick Contribution Steps

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Add tests for new functionality
5. Run the test suite (pytest)
6. Run quality checks (tox)
7. Commit your changes (git commit -m 'Add amazing feature')
8. Push to the branch (git push origin feature/amazing-feature)
9. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

�‍💻 Author

Alban Maxhuni, PhD AI Engineer 📧 [e-mail] 🌐 Evolvis.ai

Dr. Maxhuni is a leading expert in AI-driven enterprise solutions with extensive experience in cloud-native architectures, machine learning systems, and scalable data infrastructure. He specializes in developing production-ready AI tools that bridge the gap between research and enterprise deployment.

🏢 About Evolvis.ai

Evolvis.ai is at the forefront of AI innovation, developing cutting-edge solutions that empower enterprises to harness the full potential of artificial intelligence. Our mission is to create robust, scalable, and production-ready AI tools that solve real-world business challenges.

Our Focus Areas:

• 🤖 AI-Powered Enterprise Solutions - Custom AI systems for business automation
• ☁️ Cloud-Native AI Infrastructure - Scalable AI deployment platforms
• 📊 Data Engineering & MLOps - End-to-end ML pipeline solutions
• 🔧 Open Source Tools - Contributing to the AI/ML community

Why Choose Evolvis.ai Solutions:

• ✅ Production-Ready - Battle-tested in enterprise environments
• ✅ Scalable Architecture - Designed for cloud-native deployment
• ✅ Expert Support - Backed by PhD-level AI expertise
• ✅ Open Source Commitment - Contributing to the community

�🙏 Acknowledgments

• MinIO for the excellent object storage server
• Python MinIO SDK for the underlying client library
• The open-source community for inspiration and best practices
• The AI/ML community for driving innovation in data infrastructure

---

Made with ❤️ by Evolvis.ai
Advancing AI-driven solutions for modern enterprises