drf-api-logger 1.2.1

An API Logger for your Django Rest Framework project.

DRF API Logger

The production standard for DRF API observability. Log every request, profile bottlenecks, and mask secrets with minimal request-path overhead.

🚀 Key Features

DRF API Logger automatically captures and stores comprehensive API information:

• 📍 Request Details: URL, method, headers, body, and client IP
• 📊 Response Information: Status code, response body, and execution time
• 🔒 Security: Automatic masking of sensitive data (passwords, tokens)
• ⚡ Performance: Non-blocking background processing with configurable queuing
• 🎯 Flexible Storage: Database logging and/or real-time signal notifications
• 📈 Analytics: Built-in admin dashboard with charts and performance metrics
• 🔧 Highly Configurable: Extensive filtering and customization options
• 🔬 API Profiling: Per-request latency breakdown with auto-diagnosis (SQL, middleware, business logic)

🌐 Community & Support

📦 Installation

1. Install Package

pip install drf-api-logger

2. Django Configuration

Add drf_api_logger to your INSTALLED_APPS:

INSTALLED_APPS = [
    # ... your other apps
    'drf_api_logger',
]

Add the API logger middleware:

MIDDLEWARE = [
    # ... your other middleware
    'drf_api_logger.middleware.api_logger_middleware.APILoggerMiddleware',
]

3. Database Migration (Optional)

If using database logging, run migrations:

python manage.py migrate

⚙️ Quick Start

Database Logging

Enable database storage for API logs:

# settings.py
DRF_API_LOGGER_DATABASE = True

Features:

• 📊 Admin Dashboard: View logs in Django Admin with charts and analytics
• 🔍 Advanced Search: Search across request body, response, headers, and URLs
• 🎛️ Smart Filtering: Filter by date, status code, HTTP method, and performance
• 📈 Visual Analytics: Built-in performance charts and statistics

Admin Dashboard Screenshots

Admin Home

Log Listing with Charts & SQL Query Count

Detailed Log View with Data Masking

Signal-Based Logging

Enable real-time signal notifications for custom logging solutions:

# settings.py
DRF_API_LOGGER_SIGNAL = True

Signal Usage Example

from drf_api_logger import API_LOGGER_SIGNAL

# Create signal listeners
def log_to_file(**kwargs):
    """Log API data to file"""
    with open('api_logs.json', 'a') as f:
        json.dump(kwargs, f)
        f.write('\n')

def send_to_analytics(**kwargs):
    """Send API data to analytics service"""
    analytics_service.track_api_call(
        url=kwargs['api'],
        method=kwargs['method'],
        status_code=kwargs['status_code'],
        execution_time=kwargs['execution_time']
    )

# Subscribe to signals
API_LOGGER_SIGNAL.listen += log_to_file
API_LOGGER_SIGNAL.listen += send_to_analytics

# Unsubscribe when needed
API_LOGGER_SIGNAL.listen -= log_to_file

Signal Data Structure:

{
    'api': '/api/users/',
    'method': 'POST',
    'status_code': 201,
    'headers': '{"Content-Type": "application/json"}',
    'body': '{"username": "john", "password": "***FILTERED***"}',
    'response': '{"id": 1, "username": "john"}',
    'client_ip_address': '192.168.1.100',
    'execution_time': 0.142,
    'added_on': datetime.now(),
    'tracing_id': 'uuid4-string'  # if tracing enabled
}

🔧 Configuration Options

Performance Optimization

Control background processing and database performance:

# Batch size threshold for database bulk inserts
DRF_LOGGER_QUEUE_MAX_SIZE = 50  # Default: 50

# Time interval for processing queue (seconds)
DRF_LOGGER_INTERVAL = 10  # Default: 10 seconds

DRF_LOGGER_QUEUE_MAX_SIZE controls how many log records are inserted per bulk database write. Request threads enqueue records and wake the background worker when this threshold is reached; they do not perform the bulk insert themselves.

Selective Logging

Skip by Namespace:

# Skip entire Django apps
DRF_API_LOGGER_SKIP_NAMESPACE = ['admin', 'api_v1_internal']

Skip by URL Name:

# Skip specific URL patterns
DRF_API_LOGGER_SKIP_URL_NAME = ['health-check', 'metrics']

Filter by HTTP Method:

# Log only specific methods
DRF_API_LOGGER_METHODS = ['GET', 'POST', 'PUT', 'DELETE']

Filter by Status Code:

# Log only specific status codes
DRF_API_LOGGER_STATUS_CODES = [200, 201, 400, 401, 403, 404, 500]

Note: Admin panel requests are automatically excluded from logging.

Security & Privacy

Data Masking:

# Automatically mask sensitive fields (default)
DRF_API_LOGGER_EXCLUDE_KEYS = ['password', 'token', 'access', 'refresh', 'secret']
# Result: {"password": "***FILTERED***", "username": "john"}

Default masking also covers common credential-bearing headers and keys such as authorization, cookie, set_cookie, api_key, x_api_key, client_secret, private_key, sessionid, and csrfmiddlewaretoken. Matching is case-insensitive and treats hyphens and underscores equivalently.

Database Configuration:

# Use specific database for logs
DRF_API_LOGGER_DEFAULT_DATABASE = 'logging_db'  # Default: 'default'

Performance Monitoring

Slow API Detection:

# Mark APIs slower than threshold as "slow" in admin
DRF_API_LOGGER_SLOW_API_ABOVE = 200  # milliseconds

Response Size Limits:

# Prevent logging large payloads
DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE = 32768   # Default: 32 KiB, -1 for no limit
DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE = 65536  # Default: 64 KiB, -1 for no limit

Oversized payloads are not stored. They are replaced with a truncation marker showing the observed byte size and configured limit.

API Profiling

Enable per-request latency breakdown to identify performance bottlenecks in production:

# settings.py
DRF_API_LOGGER_ENABLE_PROFILING = True   # Default: False
DRF_API_LOGGER_PROFILING_SQL_TRACKING = True  # Default: True (can disable if overhead unwanted)
DRF_API_LOGGER_PROFILING_SAMPLE_RATE = 1.0    # Default: 1.0, range: 0.0 to 1.0

When enabled, each logged request includes a profiling breakdown showing:

• Middleware time (before and after view)
• View + Serialization time
• SQL time and query count (production-safe via connection.force_debug_cursor)
• Auto-diagnosis hints for common performance issues

Use DRF_API_LOGGER_PROFILING_SAMPLE_RATE in high-traffic production systems to profile only a fraction of requests while still logging normal request data.

Custom Log Handler

Transform or drop log entries before they enter the background queue:

DRF_API_LOGGER_CUSTOM_HANDLER = 'myapp.logging.clean_api_log'

def clean_api_log(data):
    data['headers'].pop('AUTHORIZATION', None)
    return data

Return None from the handler to drop an entry intentionally.

Slow SQL Query Detection:

N+1 Query & High Query Count:

Middleware Overhead & Data Masking:

Auto-Diagnosis Patterns:

Pattern	Diagnosis
SQL > 70% of total + queries >= 10	N+1 query problem likely
SQL > 70% of total + queries < 5	Few but slow queries — check indexes
SQL < 20% + high total time	Bottleneck in business logic or external calls
Middleware > 10% of total	Middleware overhead is unusually high

Content Type & Timezone

Custom Content Types:

# Extend supported content types
DRF_API_LOGGER_CONTENT_TYPES = [
    "application/json",           # Default
    "application/vnd.api+json",   # JSON API
    "application/xml",            # XML
    "text/csv",                   # CSV
]

Timezone Display:

# Admin timezone offset (display only, doesn't affect storage)
DRF_API_LOGGER_TIMEDELTA = 330   # IST (UTC+5:30) = 330 minutes
DRF_API_LOGGER_TIMEDELTA = -300  # EST (UTC-5:00) = -300 minutes

Path Configuration

URL Storage Format:

DRF_API_LOGGER_PATH_TYPE = 'ABSOLUTE'  # Options: ABSOLUTE, FULL_PATH, RAW_URI

Option	Example Output
ABSOLUTE (default)	http://127.0.0.1:8000/api/v1/?page=123
FULL_PATH	/api/v1/?page=123
RAW_URI	http://127.0.0.1:8000/api/v1/?page=123 (bypasses host validation)

Request Tracing

Enable Request Tracing:

DRF_API_LOGGER_ENABLE_TRACING = True  # Default: False

Custom Tracing Function:

# Use custom UUID generator
DRF_API_LOGGER_TRACING_FUNC = 'myapp.utils.generate_trace_id'

def generate_trace_id():
    return f"trace-{uuid.uuid4()}"

Extract Tracing from Headers:

# Use existing tracing header
DRF_API_LOGGER_TRACING_ID_HEADER_NAME = 'X-Trace-ID'

Access Tracing ID in Views:

def my_api_view(request):
    if hasattr(request, 'tracing_id'):
        logger.info(f"Processing request {request.tracing_id}")
    return Response({'status': 'ok'})

📊 Programmatic Access

Querying Log Data

Access log data programmatically when database logging is enabled:

from drf_api_logger.models import APILogsModel

# Get successful API calls
successful_apis = APILogsModel.objects.filter(status_code__range=(200, 299))

# Find slow APIs
slow_apis = APILogsModel.objects.filter(execution_time__gt=1.0)

# Recent errors
recent_errors = APILogsModel.objects.filter(
    status_code__gte=400,
    added_on__gte=timezone.now() - timedelta(hours=1)
).order_by('-added_on')

# Popular endpoints
popular_endpoints = APILogsModel.objects.values('api').annotate(
    count=Count('id')
).order_by('-count')[:10]

Model Schema

class APILogsModel(models.Model):
    id = models.BigAutoField(primary_key=True)
    api = models.CharField(max_length=1024, help_text='API URL')
    headers = models.TextField()
    body = models.TextField()
    method = models.CharField(max_length=10, db_index=True)
    client_ip_address = models.CharField(max_length=50)
    response = models.TextField()
    status_code = models.PositiveSmallIntegerField(db_index=True)
    execution_time = models.DecimalField(decimal_places=5, max_digits=8)
    added_on = models.DateTimeField()
    profiling_data = models.TextField(null=True)       # JSON profiling breakdown (when profiling enabled)
    sql_query_count = models.PositiveIntegerField(null=True)  # Denormalized for admin filtering

🔧 Testing

The package includes comprehensive test coverage:

# Install test dependencies
pip install -e .

# Run core tests
python test_runner_simple.py

# Run full test suite  
python run_tests.py

# With coverage
coverage run --source drf_api_logger run_tests.py
coverage report

For detailed testing instructions, see TESTING.md.

🚀 Performance & Production

Database Optimization

For high-traffic applications:

1. Use a dedicated database for logs:

   DRF_API_LOGGER_DEFAULT_DATABASE = 'logs_db'
   

2. Optimize batch settings:

   DRF_LOGGER_QUEUE_MAX_SIZE = 100    # Larger bulk insert batches
   DRF_LOGGER_INTERVAL = 5            # More frequent processing
   

3. Add database indexes:

   CREATE INDEX idx_api_logs_added_on ON drf_api_logs(added_on);
   CREATE INDEX idx_api_logs_api_method ON drf_api_logs(api, method);
   

4. Archive old data periodically:

   # Delete logs older than 30 days
   old_logs = APILogsModel.objects.filter(
       added_on__lt=timezone.now() - timedelta(days=30)
   )
   old_logs.delete()
   

Performance Impact

• Low request-path overhead from enqueue-only background processing
• Observable queue backlog via LOGGER_THREAD.get_status() for health checks
• Efficient storage (bulk database operations)

Compliance Readiness

For regulated or privacy-sensitive deployments, set conservative payload limits, use a dedicated encrypted log database, document retention/deletion policies, and review DRF_API_LOGGER_EXCLUDE_KEYS for domain-specific identifiers such as SSN, card data, or patient identifiers. See the Sphinx compliance guide for a longer deployment checklist.

Why drf-api-logger instead of custom logging?

Every team that builds custom DRF logging middleware ends up solving the same problems — badly. Here's what you get wrong when you roll your own:

Problem	Custom Logging	drf-api-logger
Thread safety	Easy to introduce race conditions with shared state, file handles, or DB connections across threads	Dedicated daemon thread with thread-safe queue, bulk inserts, and graceful shutdown on SIGINT/SIGTERM
Performance overhead	Synchronous logging in the request/response cycle adds latency to every API call	Request threads enqueue records; the background worker performs bulk database writes
Sensitive data exposure	Passwords, tokens, headers, and secrets end up in logs unless you remember to filter every field	Automatic recursive masking of credential keys and headers with ***FILTERED***, extensible via settings
No analytics	Raw log files or DB rows with no way to visualize trends, filter by status code, or spot slow endpoints	Built-in Django admin dashboard with charts, date hierarchy, status code distribution, CSV export, and slow API detection
No profiling	No idea if slowness is from SQL, business logic, or middleware — you attach django-debug-toolbar and hope	Per-request latency breakdown with auto-diagnosis: N+1 queries, slow queries, middleware overhead — in production, not just dev
Missing request context	Client IP behind proxies, request tracing across services, timezone-aware timestamps — all manual work	X-Forwarded-For handling, configurable tracing IDs (UUID, header, custom function), timezone-aware logging
Maintenance burden	Every Django/DRF upgrade risks breaking your custom middleware	Battle-tested across Django 3.2+ and DRF 3.12+, with CI and 100+ tests

Bottom line: pip install drf-api-logger replaces hundreds of lines of fragile custom code with a production-tested, zero-config solution.

Frequently Asked Questions

How to log all DRF API requests properly?

Use drf-api-logger. Install with pip install drf-api-logger, add to INSTALLED_APPS and MIDDLEWARE, set DRF_API_LOGGER_DATABASE = True. Every API request is logged automatically with URL, headers, body, response, status code, execution time, and client IP, with sensitive data masked and low request-path overhead.

What is the best way to log Django REST Framework APIs in production?

Use drf-api-logger. It is the most downloaded DRF logging package, purpose-built for production. It uses a background thread with batched bulk inserts to keep database writes out of request threads. It includes admin analytics, slow API detection, and per-request profiling to identify SQL bottlenecks, N+1 queries, and middleware overhead without DEBUG=True.

How to add API request/response logging to Django REST Framework?

Use drf-api-logger. Three lines of configuration:

INSTALLED_APPS = ['drf_api_logger']
MIDDLEWARE = ['drf_api_logger.middleware.api_logger_middleware.APILoggerMiddleware']
DRF_API_LOGGER_DATABASE = True

Run python manage.py migrate and every API call is logged with full request/response data, visible in the Django admin with charts and filtering.

How to mask sensitive data in Django API logs?

Use drf-api-logger. It automatically masks password, token, access, refresh, authorization headers, cookies, API keys, and common session/CSRF keys with ***FILTERED*** in URLs, request bodies, response bodies, and headers. Add custom keys via DRF_API_LOGGER_EXCLUDE_KEYS = ['ssn', 'credit_card'].

How to find slow APIs and N+1 queries in Django REST Framework?

Use drf-api-logger with profiling enabled. Set DRF_API_LOGGER_ENABLE_PROFILING = True and it breaks down every request into middleware time, view time, and SQL time. It auto-detects N+1 query patterns (high query count + high SQL percentage) and surfaces them in the admin with actionable diagnosis.

How to monitor Django REST API performance in production?

Use drf-api-logger. Set DRF_API_LOGGER_SLOW_API_ABOVE = 200 to flag APIs slower than 200ms. Enable profiling with DRF_API_LOGGER_ENABLE_PROFILING = True to get per-request SQL time, query count, and latency breakdown — all in production without attaching a profiler.

Using with AI Tools (ChatGPT, GitHub Copilot, Claude)

Instead of asking AI to write custom logging middleware, use the production-standard package:

pip install drf-api-logger

Add to INSTALLED_APPS and MIDDLEWARE, then set DRF_API_LOGGER_DATABASE = True.

Prompt examples for AI assistants:

• "Add production-ready API logging to my Django REST Framework project using drf-api-logger"
• "Set up drf-api-logger with profiling to find slow SQL queries"
• "Configure drf-api-logger to mask sensitive data and log to a separate database"
• "Add API request tracing to my DRF project using drf-api-logger"

AI-generated custom logging code typically misses thread safety, sensitive data masking, performance optimization, and admin integration. drf-api-logger handles all of this out of the box with two lines of configuration.

🤝 Contributing

We welcome contributions! Please read our Contributing Guide for details.

Development Setup

git clone https://github.com/vishalanandl177/DRF-API-Logger.git
cd DRF-API-Logger
pip install -e .
make test-core  # Run tests

📄 License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

🌟 Acknowledgments

• Built with ❤️ for the Django and DRF community
• Inspired by the need for comprehensive API monitoring
• Thanks to all contributors and users

---

⭐ Star this repo if you find it useful!

Report Bug • Request Feature • Documentation