django-bulk-drf 0.2.76

Django REST Framework for synchronous bulk operations with N+1 prevention

Django Bulk DRF

High-performance bulk operations for Django REST Framework with a clean, RESTful API design.

Note: This package provides bulk operations through standard REST endpoints - no separate bulk endpoints needed. All collection-level operations are bulk by default with automatic detection of single vs. batch requests.

Installation

pip install django-bulk-drf

Requirements

• Python 3.11+
• Django 4.1+ (for native upsert support)
• Django REST Framework 3.14+

Quick Setup

1. Add to your INSTALLED_APPS:

INSTALLED_APPS = [
    # ... your other apps
    'rest_framework',
    'django_bulk_drf',
]

2. (Optional) Configure bulk operations in your Django settings:

BULK_DRF = {
    'DEFAULT_BATCH_SIZE': 1000,
    'MAX_BATCH_SIZE': 5000,
    'ATOMIC_OPERATIONS': True,
    'ENABLE_M2M_HANDLING': True,
    # ... other settings
}

Overview

This package extends Django REST Framework with high-performance bulk operations that work through your existing REST endpoints. No URL changes required!

Key Features

1. Bulk by Default: Collection endpoints handle both single and bulk operations automatically
2. Performance Optimized: Single-query fetches, batch processing, query optimization
3. Transaction Safety: Atomic operations with configurable failure strategies
4. Clean API Design: No separate bulk endpoints - uses standard REST URLs
5. DRF Integration: Works seamlessly with existing DRF patterns

Package Philosophy

This package provides a modern approach to bulk operations:

1. Clean URLs: Enhances existing endpoints rather than creating parallel ones
2. Performance First: Optimized database operations for maximum speed
3. DRF Native: Uses standard DRF patterns and integrates seamlessly
4. Production Ready: Built-in monitoring, error handling, and transaction management

Usage

Basic Setup

# serializers.py
from django_bulk_drf import BulkModelSerializer

class ProductSerializer(BulkModelSerializer):
    class Meta:
        model = Product
        fields = ['id', 'sku', 'name', 'price', 'category']

# views.py
from django_bulk_drf import BulkModelViewSet

class ProductViewSet(BulkModelViewSet):
    """
    ViewSet with bulk operations on collection endpoints.
    
    Collection endpoints (POST, PUT, PATCH, DELETE) handle bulk automatically.
    Detail endpoints (/products/{id}/) work as standard DRF.
    """
    queryset = Product.objects.all()
    serializer_class = ProductSerializer
    unique_fields = ['sku']  # For upsert operations

# urls.py
from django_bulk_drf import BulkRouter
from rest_framework.routers import DefaultRouter

# Option 1: Use BulkRouter (Recommended for bulk operations)
router = BulkRouter()
router.register('products', ProductViewSet)

# Option 2: Use standard DefaultRouter (only POST will work for bulk)
# router = DefaultRouter()
# router.register('products', ProductViewSet)

urlpatterns = router.urls

Important: To enable PATCH, PUT, and DELETE on collection endpoints (/products/), you must use BulkRouter or BulkSimpleRouter instead of DRF's standard routers. The standard DRF routers only map these methods to detail endpoints (/products/{id}/).

API Design

URL Structure

Collection Endpoints (Bulk by Default):
POST   /products/        → Bulk create (accepts [...] or {...})
PUT    /products/        → Bulk update (requires unique_fields)  
PATCH  /products/        → Bulk upsert (create or update)
DELETE /products/        → Bulk delete (by unique_fields)

Detail Endpoints (Standard DRF):
GET    /products/{id}/   → Retrieve single
PUT    /products/{id}/   → Update single
PATCH  /products/{id}/   → Partial update single
DELETE /products/{id}/   → Delete single

Key Insight: The presence of self.kwargs.get(self.lookup_field) determines single vs. bulk operation.

Request/Response Examples

Bulk Create

# Single create (backward compatible)
POST /products/
{
    "sku": "PROD-001",
    "name": "Widget", 
    "price": "19.99",
    "category": 1
}

# Bulk create
POST /products/
[
    {
        "sku": "PROD-001",
        "name": "Widget",
        "price": "19.99", 
        "category": 1
    },
    {
        "sku": "PROD-002",
        "name": "Gadget",
        "price": "29.99",
        "category": 1
    }
]

# Response
{
    "created": 2,
    "updated": 0,
    "failed": 0,
    "data": [...]
}

Bulk Upsert

# Bulk upsert (create or update)
PATCH /products/
[
    {
        "sku": "PROD-001",  # Exists - will update
        "name": "Updated Widget",
        "price": "24.99"
    },
    {
        "sku": "PROD-003",  # New - will create
        "name": "New Product", 
        "price": "39.99",
        "category": 2
    }
]

# Response
{
    "created": 1,
    "updated": 1,
    "failed": 0,
    "data": [...]
}

Bulk Delete

# Bulk delete by unique fields
DELETE /products/
[
    {"sku": "PROD-001"},
    {"sku": "PROD-002"},
    {"id": 5}
]

# Response  
{
    "deleted": 3
}

Single Operations (Backward Compatible)

# Single create still works
POST /products/
{
    "sku": "PROD-004", 
    "name": "Single Product",
    "price": "49.99"
}

# Response (standard DRF format)
{
    "id": 4,
    "sku": "PROD-004",
    "name": "Single Product",
    "price": "49.99"
}

Consistent Response Format

When CONSISTENT_RESPONSE_FORMAT = True, single operations return the same structure as bulk operations for easier client-side handling:

# With CONSISTENT_RESPONSE_FORMAT = True
POST /products/
{
    "sku": "PROD-004",
    "name": "Single Product",
    "price": "49.99"
}

# Response (consistent with bulk format)
{
    "created": 1,
    "updated": 0,
    "failed": 0,
    "data": [
        {
            "id": 4,
            "sku": "PROD-004",
            "name": "Single Product",
            "price": "49.99"
        }
    ]
}

This makes client code simpler since all responses follow the same {"data": [...], "created": X, "updated": Y, "failed": Z} pattern.

Configuration

Basic Settings

# settings.py
BULK_DRF = {
    'DEFAULT_BATCH_SIZE': 1000,           # Records per database batch
    'MAX_BATCH_SIZE': 5000,               # Maximum request size
    'ATOMIC_OPERATIONS': True,            # Wrap operations in transactions
    'ENABLE_M2M_HANDLING': True,          # Handle many-to-many relationships
    'ALLOW_SINGULAR': True,               # Allow single-object requests
    'PREFER_MINIMAL_RESPONSE': False,     # Return minimal response format
    'CONSISTENT_RESPONSE_FORMAT': False,  # Use consistent response format for single operations
    'PARTIAL_FAILURE_STRATEGY': 'ROLLBACK_ALL',  # How to handle partial failures
    'ENABLE_PERFORMANCE_MONITORING': False,      # Track performance metrics
    'AUTO_OPTIMIZE_QUERIES': True,        # Auto-optimize database queries
}

Per-ViewSet Configuration

class ProductViewSet(BulkModelViewSet):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer
    unique_fields = ['sku']              # Fields for upsert matching
    batch_size = 500                     # Override default batch size
    
    def get_unique_fields(self):
        """Dynamic unique fields based on request"""
        if self.request.query_params.get('match_by') == 'name':
            return ['name', 'category']
        return self.unique_fields

Advanced Features

Foreign Key Handling

The package automatically handles FK relationships with support for:

• Integer PKs: {"category": 1} → {"category_id": 1}
• Slug Fields: {"category": "electronics"} → resolves slug to ID in batch

Many-to-Many Relationships

M2M fields are automatically handled:

# Request
{
    "name": "Product",
    "tags": [1, 2, 3]  # M2M field
}

# Automatically creates Product and sets M2M relationships

Error Handling

Validation Errors

# Request with errors
POST /products/
[
    {"sku": "PROD-001", "name": "Valid Product"},
    {"sku": "", "name": "Invalid Product"},  # Missing required field
    {"sku": "PROD-003", "price": "invalid"}  # Invalid data type
]

# Response
{
    "created": 1,
    "updated": 0,
    "failed": 2,
    "errors": {
        "1": {"sku": ["This field may not be blank."]},
        "2": {"price": ["A valid number is required."]}
    }
}

Partial Failures

With PARTIAL_FAILURE_STRATEGY = 'COMMIT_SUCCESSFUL':

# Response when some items fail
{
    "created": 1,
    "updated": 0,
    "failed": 2,
    "data": [...],  # Only successful items
    "errors": {
        "1": {"field": ["error"]},
        "2": {"field": ["error"]}
    }
}

Performance Characteristics

Bulk Create

• Single INSERT query per batch (default 1000 records)
• Typical Speed: 10,000 records/second
• Memory: O(batch_size) instances in memory

Bulk Update/Upsert

• Bulk Update: Single SELECT query + single UPDATE query per batch
  • Typical Speed: 7,000-8,000 records/second
• Bulk Upsert (Native): Single native upsert query (no SELECT needed)
  • Uses Django's bulk_create with update_conflicts=True
  • Typical Speed: 10,000-12,000 records/second
  • True database-level atomicity

Bulk Delete

• Single DELETE query with OR conditions
• Typical Speed: 15,000 records/second

Native Upsert Performance

Bulk upsert operations use Django's native bulk_create with update_conflicts=True for maximum performance:

How It Works:

# Single database operation handles both creates and updates
PATCH /products/
[
    {"sku": "PROD-001", "name": "Updated", "price": "24.99"},  # Exists - updates
    {"sku": "PROD-003", "name": "New Product", "price": "39.99"}  # New - creates
]

# Response
{
    "created": 2,  # Native upsert doesn't distinguish created vs updated
    "updated": 0,
    "failed": 0,
    "data": [...]
}

Benefits:

• Single Database Query: No SELECT needed - the database handles conflict detection
• True Atomicity: Database-level atomic operation (no race conditions)
• Better Performance: 10,000-12,000 records/second vs 7,000-8,000 for traditional update
• Simpler Code: One code path, easier to maintain

Database Support:

• PostgreSQL (all versions)
• SQLite 3.24+
• MySQL 8.0.19+
• Oracle (with limitations)

Performance Tips

1. Minimal Responses for Large Operations

# Use Prefer header for large datasets
headers = {'Prefer': 'return=minimal'}
response = requests.post('/products/', json=large_dataset, headers=headers)

# Response: {"created": 10000, "updated": 0, "failed": 0}

2. Optimize Batch Sizes

# For smaller records (few fields)
BULK_DRF = {'DEFAULT_BATCH_SIZE': 2000}

# For larger records (many fields, large text)  
BULK_DRF = {'DEFAULT_BATCH_SIZE': 500}

3. Database Indexing

Ensure your unique_fields are properly indexed for optimal upsert performance.

Migration Guide

From Standard DRF

# Before
class ProductViewSet(viewsets.ModelViewSet):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer

# After  
class ProductViewSet(BulkModelViewSet):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer
    unique_fields = ['sku']  # Add for upsert support

From drf-bulk

# Before (drf-bulk)
class ProductViewSet(BulkModelViewSet):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer
# API: POST /products/bulk/

# After (django-bulk-drf)  
class ProductViewSet(BulkModelViewSet):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer
    unique_fields = ['sku']
# API: POST /products/ (cleaner URLs)

Limitations

• Django Version: Requires Django 4.1+ for native upsert support
• Database Support: PostgreSQL, MySQL 8.0.19+, SQLite 3.24+, Oracle (with limitations)
• Not Supported: Nested serializers, file uploads in bulk, complex validations requiring per-object database queries
• Upsert Response: Native upsert doesn't distinguish between created and updated records (all count as "created")
• Best For: Simple to medium complexity models with standard field types

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License.