setu-trafficmonitor 2.0.1

A comprehensive Django app for logging HTTP requests with analytics dashboard

Setu TrafficMonitor

A comprehensive Django application for logging HTTP requests with a beautiful analytics dashboard. Monitor your API traffic, analyze performance metrics, and gain insights into your application's usage patterns.

Features

Request Logging

• Comprehensive Logging: Captures all HTTP requests with detailed information
• Performance Metrics: Track response times and database query counts
• User Tracking: Associate requests with authenticated users
• IP Address Tracking: Monitor requests by IP address with X-Forwarded-For support
• Request/Response Bodies: Store request and response payloads (with size limits)
• Exception Tracking: Automatic logging of exceptions with full tracebacks
• Configurable Exclusions: Exclude specific paths (static files, health checks, etc.)

Analytics Dashboard

• Real-time Visualizations: Interactive charts powered by Chart.js
• Time-based Analysis: View metrics by today, last 7 days, last 30 days, or custom ranges
• Status Code Distribution: Track success rates and error patterns
• HTTP Method Breakdown: Analyze request types (GET, POST, PUT, DELETE, etc.)
• Endpoint Performance: Identify slowest endpoints and optimization opportunities
• Hourly Heatmap: Discover traffic patterns throughout the day
• Top IPs & Users: Monitor most active clients and users
• Error Trending: Track 4xx and 5xx errors over time
• Advanced Filtering: Filter by method, status code, path, and user

REST API Endpoints

• Analytics Overview API: Get comprehensive analytics data in JSON format
• Chart-specific APIs: Fetch individual chart data for custom integrations
• Admin-only Access: Secure endpoints with Django permissions

Installation

Using pip

pip install <path-to-your-private-setu-trafficmonitor-package>

From source

git clone <internal-repo-url>
cd setu-trafficmonitor
pip install -e .

Quick Start

1. Add to Installed Apps

Add trafficmonitor to your INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    # ... other apps
    'rest_framework',  # Required dependency
    'trafficmonitor',
]

2. Add Middleware

Add the middleware to your MIDDLEWARE setting (preferably near the top):

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'trafficmonitor.middleware.RequestLoggingMiddleware',  # Add here
    # ... other middleware
]

3. Configure URLs

Include the TrafficMonitor URLs in your project's urls.py:

from django.urls import path, include

urlpatterns = [
    # ... your other URLs
    path('', include('trafficmonitor.analytics.urls')),
]

4. Run Migrations

python manage.py migrate trafficmonitor

5. Access the Dashboard

Start your development server and navigate to:

• Dashboard: http://localhost:8000/analytics/dashboard/
• API Overview: http://localhost:8000/api/analytics/overview/

Note: Dashboard access is restricted to staff users. Make sure you have a staff user:

python manage.py createsuperuser

Configuration

Middleware Settings

You can customize the middleware behavior by subclassing and overriding:

from trafficmonitor.middleware import RequestLoggingMiddleware

class CustomRequestLoggingMiddleware(RequestLoggingMiddleware):
    # Maximum body size to log (in characters)
    MAX_BODY_LENGTH = 20000

    # Paths to exclude from logging
    EXCLUDED_PATHS = [
        '/admin/jsi18n/',
        '/static/',
        '/media/',
        '/__debug__/',
        '/favicon.ico',
        '/health/',  # Add custom paths
    ]

Then use your custom middleware in settings.py:

MIDDLEWARE = [
    # ...
    'myapp.middleware.CustomRequestLoggingMiddleware',
    # ...
]

Database Optimization

For high-traffic applications, consider:

1. Database Indexing: The package includes optimized indexes for common queries
2. Partitioning: Consider partitioning the request_log table by date
3. Archiving: Implement a cleanup strategy for old logs:

from django.utils import timezone
from datetime import timedelta
from trafficmonitor.models import RequestLog

# Delete logs older than 90 days
cutoff_date = timezone.now() - timedelta(days=90)
RequestLog.objects.filter(timestamp__lt=cutoff_date).delete()

4. Read Replicas: Route analytics queries to read replicas if available

Usage Examples

Accessing Analytics Data Programmatically

from trafficmonitor.analytics.views import AnalyticsQueryHelper
from django.utils import timezone
from datetime import timedelta

# Get date range
end_date = timezone.now()
start_date = end_date - timedelta(days=7)

# Get total requests
total = AnalyticsQueryHelper.get_total_requests(start_date, end_date)

# Get top endpoints
top_endpoints = AnalyticsQueryHelper.get_top_endpoints(
    start_date, end_date, limit=10
)

# Get slowest endpoints
slow_endpoints = AnalyticsQueryHelper.get_slowest_endpoints(
    start_date, end_date, limit=10
)

# Get comprehensive analytics
analytics = AnalyticsQueryHelper.get_comprehensive_analytics(
    start_date, end_date
)

Filtering Analytics

# Filter by HTTP method
get_requests = AnalyticsQueryHelper.get_total_requests(
    start_date, end_date, method='GET'
)

# Filter by status code range
errors = AnalyticsQueryHelper.get_total_requests(
    start_date, end_date, status_code_range=(400, 599)
)

# Filter by path
api_requests = AnalyticsQueryHelper.get_total_requests(
    start_date, end_date, path_contains='/api/'
)

# Filter by user
user_requests = AnalyticsQueryHelper.get_total_requests(
    start_date, end_date, user_id=user.id
)

Custom Celery Task for Log Cleanup

from celery import shared_task
from django.utils import timezone
from datetime import timedelta
from trafficmonitor.models import RequestLog

@shared_task
def cleanup_old_request_logs():
    """Delete request logs older than 90 days"""
    cutoff_date = timezone.now() - timedelta(days=90)
    deleted_count, _ = RequestLog.objects.filter(
        timestamp__lt=cutoff_date
    ).delete()
    return f"Deleted {deleted_count} old request logs"

API Reference

REST API Endpoints

Analytics Overview

GET /api/analytics/overview/

Query Parameters:

• range: today, yesterday, last_7_days, last_30_days, custom
• start_date: ISO format date (YYYY-MM-DD) for custom range
• end_date: ISO format date (YYYY-MM-DD) for custom range
• method: Filter by HTTP method (GET, POST, etc.)
• status: Filter by status code
• path: Filter by path contains
• user: Filter by user ID

Chart Data

GET /api/analytics/chart/<chart_type>/

Chart Types:

• time-series: Requests over time
• status-codes: Status code distribution
• methods: HTTP methods breakdown
• endpoints: Top endpoints
• performance: Slowest endpoints
• heatmap: Hourly request heatmap
• errors: Error trend over time

Database Schema

RequestLog Model

Field	Type	Description
id	UUID	Primary key
method	CharField	HTTP method (GET, POST, etc.)
path	CharField	Request path/URL
full_url	TextField	Complete URL with query params
status_code	IntegerField	HTTP response status code
user	ForeignKey	Authenticated user (nullable)
ip_address	GenericIPAddressField	Client IP address
user_agent	TextField	User agent string
request_headers	JSONField	Request headers (sensitive ones excluded)
request_body	TextField	Request body (truncated)
response_body	TextField	Response body (truncated)
response_time_ms	FloatField	Response time in milliseconds
query_count	IntegerField	Database queries executed
exception	TextField	Exception traceback if failed
content_length	IntegerField	Response size in bytes
timestamp	DateTimeField	Request timestamp

Performance Considerations

• Indexes: The package includes optimized database indexes for common queries
• Query Optimization: Analytics queries use select_related() and annotate() for efficiency
• Async Logging: Consider implementing asynchronous logging for high-traffic applications
• Body Truncation: Request/response bodies are automatically truncated to prevent database bloat
• Excluded Paths: Static files and health checks are excluded by default

Security

• Sensitive Headers: Authorization headers and cookies are automatically excluded from logging
• Admin Access: Dashboard and API endpoints require staff/admin permissions
• User Authentication: Built-in Django authentication and authorization
• CSRF Protection: Standard Django CSRF protection applies

Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Versioning

This project uses Semantic Versioning:

• MAJOR version for incompatible API changes
• MINOR version for backwards-compatible functionality additions
• PATCH version for backwards-compatible bug fixes

See CHANGELOG.md for version history.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

• Issues: GitHub Issues
• Documentation: GitHub Wiki

Credits

Developed by FarmSetu Team

Changelog

See CHANGELOG.md for a detailed version history.