django-container-manager 1.0.6

Django app for container orchestration with multi-executor support (Docker, Google Cloud Run, AWS Fargate)

Django Docker Container Manager

The most comprehensive Docker container management system for Django. Execute Django commands inside Docker containers with complete lifecycle tracking, resource monitoring, and multi-platform support. Eliminate the complexity of Celery/RQ while gaining powerful containerized job execution.

Transform your Django application into a container orchestration platform with full observability, automatic resource management, and production-ready security controls.

Why Choose This Over Celery?

• 🎯 Django-native design: Built for Django from the ground up, not adapted from generic queuing
• 🐳 Container-first architecture: Docker containers as first-class execution environments with full isolation
• 📊 Complete observability: Track every aspect of job execution - logs, metrics, resource usage, timings
• 🔧 Zero infrastructure dependencies: No Redis, RabbitMQ, or external queue management required
• 🌐 Multi-cloud ready: Single codebase supports local Docker, remote hosts, Google Cloud Run, AWS Fargate
• ⚡ Production hardened: Built-in security controls, resource limits, error handling, and monitoring
• 🔍 Real-time visibility: Django admin integration with live job status and log viewing

✨ Key Features

• 🐳 Multi-executor support: Docker local/remote, Google Cloud Run, AWS Fargate, Mock executor
• 📊 Complete lifecycle tracking: Status monitoring, logs, metrics, and resource usage
• 🎛️ Django admin integration: Beautiful web interface with real-time job monitoring
• 🔧 Background job processor: Automatic container job execution and management
• 📦 Environment management: Template-based environment variables with job-level overrides
• 🔒 Production security: TLS support, resource limits, container isolation, and safe execution
• 📈 Resource monitoring: Memory and CPU usage tracking with configurable limits
• 🌐 Cloud-native: Built for containerized environments and cloud deployment

🏗️ Architecture Overview

graph TD
    A[Django Management Command] --> B[ContainerJob Model]
    B --> C[ExecutorHost Selection]
    C --> D[Docker/Cloud Executor]
    D --> E[Container Launch]
    E --> F[Real-time Monitoring]
    F --> G[Log Collection]
    G --> H[Resource Tracking]
    H --> I[Cleanup & Results]
    
    subgraph "Django Application"
        B --> J[Admin Interface]
        B --> K[Django ORM]
        K --> L[Database Storage]
    end
    
    subgraph "Execution Environment"
        D --> M[Local Docker]
        D --> N[Remote Docker]
        D --> O[Cloud Run]
        D --> P[AWS Fargate]
    end

Core Components:

• ContainerJob: Django model representing a containerized task
• ExecutorHost: Defines where containers run (local, remote, cloud)
• Executors: Pluggable backends for different container platforms
• Job Processor: Background service that manages the job lifecycle
• Admin Interface: Web UI for job creation, monitoring, and management

🚀 Quick Start

Prerequisites

• Docker: Install Docker and ensure it's running
• Python 3.8+: Compatible with Python 3.8, 3.9, 3.10, 3.11, 3.12
• Django 4.2+: Compatible with Django 4.2, 5.0, 5.1, 5.2

# Verify Docker is running
docker --version
docker ps

Installation

# Install from PyPI
pip install django-container-manager

# Or with uv (recommended)
uv add django-container-manager

5-Minute Setup

1. Add to Django settings:

# settings.py
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    # ... your other apps
    'container_manager',  # Add this line
]

2. Run migrations:

python manage.py migrate

3. Create superuser and start admin:

python manage.py createsuperuser
python manage.py runserver  # Visit http://localhost:8000/admin/

4. Start the job processor (required for job execution):

# In a separate terminal
python manage.py process_container_jobs

Your First Container Job

Create and run your first containerized task in under 2 minutes:

1. Set up Docker executor (via Django shell):

python manage.py shell

from container_manager.models import ExecutorHost

# Create local Docker executor
docker_host = ExecutorHost.objects.create(
    name="local-docker",
    executor_type="docker",
    connection_string="unix:///var/run/docker.sock",
    is_active=True
)

2. Create a simple job:

from container_manager.models import ContainerJob

# Create a "Hello World" container job
job = ContainerJob.objects.create_job(
    image="hello-world",
    name="My First Container Job",
    memory_limit=128,  # 128MB
    cpu_limit=0.5,     # 0.5 CPU cores
    timeout_seconds=300
)

print(f"Created job: {job.id}")

3. Monitor execution (the job processor will automatically pick it up):

# Refresh the job to see updated status
job.refresh_from_db()
print(f"Status: {job.status}")
print(f"Output: {job.clean_output}")

That's it! Your first container job is running. Visit the Django admin to see real-time status updates and logs.

📖 Real-World Usage Examples

Example 1: Data Processing Pipeline

Run a Python data processing script in an isolated container:

from container_manager.models import ContainerJob

# Process CSV data with pandas
job = ContainerJob.objects.create_job(
    image="python:3.11-slim",
    command="pip install pandas && python -c \"import pandas as pd; print('Processing complete')\"",
    name="Data Processing Job",
    memory_limit=1024,  # 1GB for data processing
    cpu_limit=2.0,      # 2 CPU cores
    timeout_seconds=1800  # 30 minutes
)

Example 2: Automated Testing

Run tests in a clean container environment:

# Run pytest in isolated environment
test_job = ContainerJob.objects.create_job(
    image="python:3.11",
    command="/bin/bash -c 'pip install pytest && pytest --verbose'",
    name="Test Suite Execution",
    environment_vars={
        "PYTHONPATH": "/app",
        "TEST_ENV": "container",
        "PYTEST_ARGS": "--tb=short"
    },
    memory_limit=512,
    cpu_limit=1.0
)

Example 3: Image Processing

Process images with ImageMagick in a containerized environment:

# Image processing pipeline
image_job = ContainerJob.objects.create_job(
    image="dpokidov/imagemagick:latest",
    command="convert -version && echo 'ImageMagick ready for processing'",
    name="Image Processing Pipeline",
    memory_limit=2048,  # 2GB for image processing
    cpu_limit=1.5,
    timeout_seconds=3600  # 1 hour
)

Example 4: Database Operations

Run database migrations or data imports:

# Database migration in container
db_job = ContainerJob.objects.create_job(
    image="postgres:15",
    command="psql --version && echo 'Database operations ready'",
    name="Database Migration",
    environment_vars={
        "POSTGRES_HOST": "db.example.com",
        "POSTGRES_DB": "myapp",
        "POSTGRES_USER": "admin"
        # Note: Use secrets management for passwords in production
    },
    memory_limit=256,
    cpu_limit=0.5
)

Environment Variables

Environment variables can be easily added to templates using a simple text format:

env_template.environment_variables_text = """
# Database configuration
DATABASE_URL=postgresql://user:pass@host:5432/db
DB_POOL_SIZE=10

# API settings  
API_KEY=your-secret-key
API_TIMEOUT=30

# Feature flags
DEBUG=false
ENABLE_CACHE=true
"""

Format rules:

• One variable per line: KEY=value
• Comments start with # and are ignored
• Values can contain spaces and = characters
• Empty lines are ignored

Advanced Configuration

# settings.py
CONTAINER_MANAGER = {
    "AUTO_PULL_IMAGES": True,
    "IMMEDIATE_CLEANUP": True,
    "MAX_CONCURRENT_JOBS": 10,
    "POLL_INTERVAL": 5,
    "DEFAULT_MEMORY_LIMIT": 512,
    "DEFAULT_CPU_LIMIT": 1.0,
    "CLEANUP_HOURS": 24,
}

# Enable executor factory for multi-cloud support
USE_EXECUTOR_FACTORY = True

🌥️ Multi-Cloud Support

Google Cloud Run

# Configure Cloud Run executor
host = ExecutorHost.objects.create(
    name="gcp-cloud-run",
    executor_type="cloudrun",
    weight=150,
    executor_config={
        "project": "my-gcp-project",
        "region": "us-central1",
        "cpu_limit": "2",
        "memory_limit": "2Gi",
    }
)

AWS Fargate

# Configure Fargate executor
host = ExecutorHost.objects.create(
    name="aws-fargate",
    executor_type="fargate",
    weight=120,
    executor_config={
        "cluster": "my-ecs-cluster",
        "subnets": ["subnet-12345", "subnet-67890"],
        "security_groups": ["sg-abcdef"],
    }
)

🔧 Job Processing & Monitoring

Starting the Job Processor

The background job processor is essential for job execution:

# Production: Keep running continuously
python manage.py process_container_jobs

# Development: Run once and exit
python manage.py process_container_jobs --once

# Custom polling: Check for jobs every 10 seconds
python manage.py process_container_jobs --poll-interval=10

Critical: Without the job processor running, no containers will be executed. This service manages the complete job lifecycle from container creation to cleanup.

Real-time Monitoring

Monitor job execution through multiple interfaces:

Django Admin Interface

• Live job status updates with automatic refresh
• Real-time log viewing directly in the browser
• Resource usage tracking (memory, CPU, execution time)
• Bulk operations for managing multiple jobs
• Executor host management and health monitoring

Command Line Monitoring

# Check job status programmatically
from container_manager.models import ContainerJob

# View recent jobs
recent_jobs = ContainerJob.objects.filter(
    created_at__gte=timezone.now() - timedelta(hours=24)
)

for job in recent_jobs:
    print(f"{job.name}: {job.status} ({job.duration})")

Production Monitoring

# Health check endpoint example
def health_check():
    from container_manager.models import ExecutorHost
    
    unhealthy_hosts = []
    for host in ExecutorHost.objects.filter(is_active=True):
        # Check host health (implement based on your needs)
        if not host.is_available():
            unhealthy_hosts.append(host.name)
    
    return {
        'healthy': len(unhealthy_hosts) == 0,
        'unhealthy_hosts': unhealthy_hosts
    }

🔒 Production Security

Built-in security features for production deployment:

• Resource limits: Strict memory (max 2GB) and CPU (max 2 cores) constraints
• Container isolation: Non-privileged containers with security contexts
• Network isolation: Configurable network policies and container networking
• TLS support: Secure connections to remote Docker hosts and registries
• Secret management: Environment variable masking and secure credential handling
• Access controls: Role-based permissions through Django's auth system

Security Best Practices

# Secure job creation with resource limits
secure_job = ContainerJob.objects.create_job(
    image="trusted-registry.company.com/app:latest",
    command="python secure_task.py",
    memory_limit=512,    # Always set memory limits
    cpu_limit=1.0,       # Always set CPU limits
    timeout_seconds=900, # 15-minute timeout
    environment_vars={
        "LOG_LEVEL": "INFO",
        # Never put secrets in environment_vars in code
        # Use Django secrets management instead
    }
)

See the Docker Integration Guide for comprehensive security configuration.

🛠️ Development

Setup Development Environment

# Clone and setup
git clone https://github.com/samtexas/django-container-manager.git
cd django-container-manager

# Install with uv (recommended)
uv sync --extra dev

# Or with pip
pip install -e ".[dev]"

Running Tests

# Run all tests
python manage.py test

# Run with coverage
pytest --cov=container_manager

# Run specific test categories
pytest -m unit  # Unit tests only
pytest -m integration  # Integration tests only

Code Quality

# Format and lint with ruff
uv run ruff format .
uv run ruff check .

# Type checking
uv run mypy container_manager

📚 Documentation

Core Documentation

• Docker Integration Guide - Comprehensive Docker setup, configuration, and best practices
• Installation Guide - Detailed installation instructions for all platforms
• API Reference - Complete API documentation and examples
• Troubleshooting Guide - Common issues and solutions

Development Documentation

• Contributing Guide - Development setup, testing, and contribution guidelines
• Internal Documentation - Technical implementation details and development notes

External Resources

• PyPI Package: django-container-manager
• Source Code: GitHub Repository
• Issue Tracker: GitHub Issues

🤖 LLM Agent Guidelines

Behavioral Constraints

• DO: Test every code example in a clean environment before including
• DO: Verify all installation commands work on the target system
• DO: Keep examples simple and immediately runnable
• DO: Link to detailed documentation for complex topics
• DO NOT: Include misleading performance claims or unsupported features
• DO NOT: Add examples that require external services without clear setup
• DO NOT: Promise functionality that doesn't exist or is experimental
• LIMITS: README should be scannable in under 5 minutes

Security Requirements

• Never include: Real credentials, API keys, or production URLs in examples
• Always use: Placeholder values like your-registry.com or your-project-id
• Validate examples: Ensure no security anti-patterns in code samples
• Credential guidance: Direct users to proper credential management documentation

Safe Operation Patterns

• Example validation process:
  1. Create clean test environment
  2. Follow README instructions exactly as written
  3. Verify each step produces expected output
  4. Document any prerequisites or assumptions
• Link validation: Test all internal and external links for accuracy
• Version consistency: Ensure examples match current codebase capabilities

Error Handling

• If examples fail: Update examples, don't ignore failures
• If installation doesn't work: Revise instructions, add troubleshooting notes
• If technical accuracy questioned: Cross-reference with CLAUDE.md and codebase
• When unsure: Ask for clarification rather than guessing

Validation Requirements

• All code examples tested in clean environment
• Installation steps verified on target platforms
• No credentials or sensitive information in examples
• All links functional and point to correct resources
• Technical claims verified against actual codebase capabilities
• Examples use only documented, stable features

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide for details on:

• Setting up development environment
• Code style and testing requirements
• Submitting pull requests
• Reporting bugs and feature requests

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔗 Links

• PyPI: https://pypi.org/project/django-container-manager/
• Source Code: https://github.com/samtexas/django-container-manager
• Issue Tracker: https://github.com/samtexas/django-container-manager/issues
• Documentation: https://django-container-manager.readthedocs.io/

---

For the Django community