django-query-log 1.0.1

A Django package for logging database query execution time and performance analysis for all Django requests

Django Query Log

A comprehensive Django package for logging database query execution time and performance analysis for all Django requests. Monitor your application's database performance with detailed insights, N+1 query detection, and customizable logging formats. Works with regular Django views, class-based views, and Django REST Framework APIs.

Features

• 🚀 Real-time Query Monitoring: Track database queries for all Django requests
• 📊 Performance Analysis: Automatic detection of slow queries and performance bottlenecks
• 🔍 N+1 Query Detection: Identify and warn about N+1 query problems
• 🔄 Duplicate Query Detection: Find and report duplicate queries
• 📝 Multiple Log Formats: Support for JSON, compact, detailed, and custom formatters
• ⚙️ Highly Configurable: Extensive configuration options for different use cases
• 🎯 Universal Django Support: Works with function views, class-based views, and Django REST Framework
• 🔐 Security: Automatic sanitization of sensitive data in logs
• 📈 Stack Trace Support: Optional stack trace logging for query sources
• 🎨 Custom Formatters: Create your own log formatting logic

Installation

Install using pip:

pip install django-query-log

For Django REST Framework support:

pip install django-query-log[drf]

Or install with development dependencies:

pip install django-query-log[dev]

Quick Start

1. Add to Django Settings:

# settings.py
INSTALLED_APPS = [
    # ... your other apps
    'django_query_log',
]

MIDDLEWARE = [
    # ... your other middleware
    'django_query_log.middleware.QueryLogMiddleware',
    # ... rest of your middleware
]

# Optional: Configure django-query-log
DJANGO_QUERY_LOG = {
    'ENABLED': True,
    'LOG_LEVEL': 'INFO',
    'LOGGER_NAME': 'django_query_log',
}

2. Configure Logging (optional):

# settings.py
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'query_log': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'django_query_log': {
            'handlers': ['query_log'],
            'level': 'INFO',
            'propagate': False,
        },
    },
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {message}',
            'style': '{',
        },
    },
}

3. Start your Django server and make requests to any Django view. Query logs will appear in your console!

Usage Examples

Basic Usage

Once installed and configured, django-query-log automatically logs database queries for all Django requests (views, APIs, admin, etc.):

[2024-01-15T10:30:45.123456] GET /api/users/ - Queries: 3, Total Time: 0.0157s
  User: admin (ID: 1)
  Performance Analysis:
    Average Query Time: 0.0052s
    Duplicate Queries: 0 queries in 0 groups
  Queries:
    1. [0.0031s] SELECT "auth_user"."id", "auth_user"."username", "auth_user"."email" FROM "auth_user" WHERE "auth_user"."is_active" = TRUE
    2. [0.0089s] SELECT "user_profile"."id", "user_profile"."user_id", "user_profile"."avatar" FROM "user_profile" WHERE "user_profile"."user_id" IN (1, 2, 3)
    3. [0.0037s] SELECT COUNT(*) AS "__count" FROM "auth_user" WHERE "auth_user"."is_active" = TRUE
  Response: 200 (1024 bytes)

Using Different Formatters

JSON Formatter (great for structured logging):

# settings.py
DJANGO_QUERY_LOG = {
    'FORMATTER_CLASS': 'django_query_log.formatters.JSONFormatter',
}

Compact Formatter (minimal output):

# settings.py
DJANGO_QUERY_LOG = {
    'FORMATTER_CLASS': 'django_query_log.formatters.CompactFormatter',
}

Output:

[2024-01-15T10:30:45.123456] GET /api/users/ | queries=3 | time=0.016s | user=admin

Detailed Formatter (comprehensive information):

# settings.py
DJANGO_QUERY_LOG = {
    'FORMATTER_CLASS': 'django_query_log.formatters.DetailedFormatter',
}

Works with All Django Views

Django Query Log works seamlessly with all types of Django views:

Function-based views:

def my_view(request):
    users = User.objects.all()  # Logged automatically
    return render(request, 'users.html', {'users': users})

Class-based views:

class UserListView(ListView):
    model = User  # All queries logged automatically
    template_name = 'users.html'

Django REST Framework views:

from rest_framework.viewsets import ModelViewSet

class UserViewSet(ModelViewSet):
    queryset = User.objects.all()  # All queries logged automatically
    serializer_class = UserSerializer

Context Manager

Temporarily disable or enable logging for specific code blocks:

from django_query_log.middleware import DjangoQueryLogContext

# Disable logging for this block
with DjangoQueryLogContext(enabled=False):
    users = User.objects.all()  # This won't be logged

# Enable logging (if disabled globally)
with DjangoQueryLogContext(enabled=True):
    users = User.objects.all()  # This will be logged

Configuration

Complete Configuration Options

# settings.py
DJANGO_QUERY_LOG = {
    # Enable/disable query logging
    'ENABLED': True,

    # Minimum query execution time to log (in seconds)
    'MIN_QUERY_TIME': 0.001,

    # Maximum number of queries to log per request
    'MAX_QUERIES_PER_REQUEST': 100,

    # Log level for query information
    'LOG_LEVEL': 'INFO',  # DEBUG, INFO, WARNING, ERROR, CRITICAL

    # Logger name to use
    'LOGGER_NAME': 'django_query_log',

    # Include request information in logs
    'INCLUDE_REQUEST_INFO': True,

    # Include response information in logs
    'INCLUDE_RESPONSE_INFO': True,

    # Include user information in logs
    'INCLUDE_USER_INFO': True,

    # Include SQL query text in logs
    'INCLUDE_SQL': True,

    # Include query execution stack trace
    'INCLUDE_STACK_TRACE': False,

    # Sensitive parameters to exclude from logs
    'SENSITIVE_PARAMS': ['password', 'token', 'secret', 'key'],

    # Only log queries for specific paths (empty list means all paths)
    'INCLUDE_PATHS': [],

    # Exclude queries for specific paths
    'EXCLUDE_PATHS': ['/admin/', '/static/', '/media/'],

    # Custom formatter class
    'FORMATTER_CLASS': 'django_query_log.formatters.DefaultFormatter',

    # Include duplicate query detection
    'DETECT_DUPLICATE_QUERIES': True,

    # Warn about N+1 query problems
    'DETECT_N_PLUS_ONE': True,

    # Include query performance analysis
    'INCLUDE_PERFORMANCE_ANALYSIS': True,
}

Configuration Examples

Production Environment (minimal logging):

DJANGO_QUERY_LOG = {
    'ENABLED': True,
    'MIN_QUERY_TIME': 0.01,  # Only log slow queries
    'MAX_QUERIES_PER_REQUEST': 50,
    'LOG_LEVEL': 'WARNING',
    'INCLUDE_SQL': False,  # Don't log SQL in production
    'FORMATTER_CLASS': 'django_query_log.formatters.CompactFormatter',
    'EXCLUDE_PATHS': ['/admin/', '/static/', '/media/', '/health/'],
}

Development Environment (detailed logging):

DJANGO_QUERY_LOG = {
    'ENABLED': True,
    'MIN_QUERY_TIME': 0.001,
    'LOG_LEVEL': 'DEBUG',
    'INCLUDE_STACK_TRACE': True,
    'FORMATTER_CLASS': 'django_query_log.formatters.DetailedFormatter',
    'DETECT_N_PLUS_ONE': True,
    'DETECT_DUPLICATE_QUERIES': True,
}

API-Only Logging:

DJANGO_QUERY_LOG = {
    'ENABLED': True,
    'INCLUDE_PATHS': ['/api/'],  # Only log API requests
    'EXCLUDE_PATHS': ['/api/health/', '/api/metrics/'],
}

Performance Features

N+1 Query Detection

Django Query Log automatically detects potential N+1 query problems:

N+1 Query Issues: 1 detected
  - Pattern executed 25 times (0.1234s total)

Duplicate Query Detection

Identifies repeated identical queries:

Duplicate Queries: 15 queries in 3 groups

Slow Query Identification

Highlights the slowest queries in each request:

Slowest Queries:
  1. [0.0892s] SELECT * FROM "products" WHERE "products"."category_id" = 1
  2. [0.0445s] SELECT COUNT(*) FROM "orders" WHERE "orders"."user_id" = 123

Custom Formatters

Create your own custom formatter:

# myapp/formatters.py
from django_query_log.formatters import BaseFormatter

class MyCustomFormatter(BaseFormatter):
    def format_log_entry(self, log_data):
        request_info = log_data.get('request', {})
        performance = log_data.get('performance_analysis', {})

        return f"API {request_info.get('method')} {request_info.get('path')} " \
               f"executed {performance.get('total_queries', 0)} queries " \
               f"in {performance.get('total_time', 0):.3f}s"

# settings.py
DJANGO_QUERY_LOG = {
    'FORMATTER_CLASS': 'myapp.formatters.MyCustomFormatter',
}

Integration with Logging Systems

Structured Logging with JSON

import json
import logging

class JSONLogHandler(logging.StreamHandler):
    def emit(self, record):
        try:
            # Parse the log message as JSON if possible
            log_data = json.loads(record.getMessage())
            # Send to your logging service (e.g., ELK, Splunk, etc.)
            super().emit(record)
        except json.JSONDecodeError:
            super().emit(record)

# settings.py
LOGGING = {
    'handlers': {
        'json_query_log': {
            'level': 'INFO',
            '()': 'myapp.logging.JSONLogHandler',
        },
    },
    'loggers': {
        'django_query_log': {
            'handlers': ['json_query_log'],
            'level': 'INFO',
        },
    },
}

DJANGO_QUERY_LOG = {
    'FORMATTER_CLASS': 'django_query_log.formatters.JSONFormatter',
}

Requirements

• Python 3.8+
• Django 3.2+
• Django REST Framework 3.12.0+ (optional)

Compatibility

• Django: 3.2, 4.0, 4.1, 4.2, 5.0
• Python: 3.8, 3.9, 3.10, 3.11, 3.12
• Django REST Framework: 3.12.0+ (optional for API logging)

Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

Development Setup

1. Clone the repository:

git clone https://github.com/dipee/django-query-log.git
cd django-query-log

2. Install development dependencies:

pip install -e .[dev]

3. Run tests:

pytest

4. Run linting:

black .
flake8 .
isort .
mypy .

License

This project is licensed under the MIT License - see the LICENSE file for details.

Changelog

See CHANGELOG.md for release history.

Support

• Documentation: https://django-query-log.readthedocs.io/
• Issues: https://github.com/dipee/django-query-log/issues
• Discussions: https://github.com/dipee/django-query-log/discussions

Acknowledgments

• Inspired by Django Debug Toolbar's query logging capabilities
• Built for Django developers who need production-ready query monitoring for all types of views
• Thanks to all contributors who help improve this package

---