Zibanu Django Repository Package

zibanu.django.repository — A comprehensive file repository manager for Django with versioning, metadata extraction, and thumbnail generation capabilities.

Overview

The Zibanu Django Repository package provides a complete solution for managing file storage in Django projects. It handles file uploads, versioning, structured storage with multi-level directories, automatic metadata extraction, thumbnail generation, and file categorization. Files are stored in a structured, date-partitioned directory hierarchy and can be retrieved through a unified API.

Key Features

• File Versioning: Track and manage multiple versions of uploaded files
• Structured Storage: Multi-level directory organization with date-based partitioning (year/month)
• Metadata Extraction: Automatic extraction from PDFs, images, videos, and Office documents
• Thumbnail Generation: Create thumbnails for image files
• File Categorization: Organize files using a hierarchical category system
• File Validation: Enforce file type restrictions and security policies
• Content Hashing: Multiple hash algorithm support (MD5, SHA1, SHA256, SHA384, SHA512)
• MIME Type Detection: Automatic MIME type resolution
• Base64 Serialization: JSON-compatible file transport via Base64 encoding
• REST API: Built-in endpoints for file retrieval and download

Installation

1. Add to Django Project

Ensure the package is available in your Python environment, then add it to your Django project's INSTALLED_APPS:

# settings.py
INSTALLED_APPS = [
    # ...
    "zibanu.django.repository",
]

2. Configure Django Settings

Set the required media directories and repository-specific settings (see Configuration section).

3. Run Migrations

python manage.py migrate zibanu.django.repository

Quick Start

Basic File Upload and Storage

from django.core.files.uploadedfile import UploadedFile
from zibanu.django.repository.lib.controllers import FileController
from zibanu.django.repository.models import Category, File

# Get or create a category
category = Category.objects.first()  # or create one

# Create a controller with an uploaded file
controller = FileController(
    file_handler=uploaded_file,
    category=category,
    request=request
)

# Create the file record
file_instance = controller.create_file()

print(f"File created: {file_instance.uuid}")
print(f"File URL: {file_instance.file_url}")

Retrieving and Downloading Files

from zibanu.django.repository.lib.controllers import FileController

# Load an existing file
controller = FileController(uuid="your-file-uuid", request=request)

# Access file properties
print(f"File size: {controller.file_size} bytes")
print(f"File hash: {controller.file_hash}")
print(f"MIME type: {controller.mime_type}")

# Get file content as Base64 (for JSON transport)
base64_content = controller.file_content_base64

# Get file content as bytes
content = controller.file_content

Working with File Versions

When versioning is enabled, each file update creates a new version:

# Load a specific file version
controller = FileController(
    uuid="your-file-uuid",
    file_version=1,  # Load version 1
    request=request
)

file_content = controller.file_content

Extracting Metadata

Metadata is automatically extracted for supported file types when the category has extract_metadata=True:

controller = FileController(uuid="your-file-uuid", request=request)

# Access extracted metadata
metadata = controller.metadata
print(metadata)  # e.g., {'pages': 10, 'author': 'John Doe', ...}

Generating Thumbnails

Thumbnails are created for image files when the category has gen_thumb=True:

controller = FileController(uuid="your-file-uuid", request=request)

# Get thumbnail as bytes
thumbnail_content = controller.thumbnail_content

# Or get the file handler
thumbnail_handler = controller.thumbnail_handler

Features

File Versioning

File versioning allows tracking changes over time. When enabled via ZB_REPOSITORY_VERSIONING=True, each new file upload or update creates a version entry:

• Access historical versions using FileController(file_version=N)
• Version numbers start at 1
• All versions are stored in the FileVersion model with their own UUIDs

Multi-Level Directory Structure

Files are organized in a hierarchical structure:

MEDIA_ROOT/
└── ZbRepository/              # ZB_REPOSITORY_ROOT_DIR
    └── ZbFiles/               # ZB_REPOSITORY_FILES_DIR
        ├── 2024/
        │   ├── 1/             # Month 1 (January)
        │   │   ├── file-uuid-1.pdf
        │   │   └── file-uuid-2.jpg
        │   └── 2/             # Month 2 (February)
        │       └── file-uuid-3.docx
        └── 2025/
            └── 3/
                └── file-uuid-4.xlsx

File Categories

Categories organize files hierarchically:

from zibanu.django.repository.models import Category

# Create root category
root = Category.objects.create(name="Documents", parent=None)

# Create subcategory
invoices = Category.objects.create(
    name="Invoices",
    parent=root,
    gen_thumb=False,
    extract_metadata=True,
    file_types="pdf,jpg,png"
)

# Upload file to category
controller = FileController(file_handler=file, request=request)
file_instance = controller.create_file()
file_instance.file_extended.category = invoices
file_instance.file_extended.save()

File Categorization Rules

Categories support several configuration options:

• gen_thumb: Generate thumbnails for image files
• extract_metadata: Automatically extract file metadata
• convert_to_mp4: Convert compatible video formats to MP4
• file_types: Comma-separated list of allowed file extensions (e.g., "pdf,jpg,png")
• is_public: Make category accessible to unauthenticated users
• published: Visibility flag for the category

Upload Rejection Rules

When a file is uploaded, the controller (FileController.__validate_file) rejects it — raising ValidationError — under any of these conditions:

• The extension is in the global ZB_REPOSITORY_INVALID_EXTENSIONS denylist.
• The target category is_root — root categories never store files directly.
• The category's files_allowed is False. This depends on the two flags below: when ZB_REPOSITORY_MULTILEVEL_FILES_ALLOWED is False, only the deepest allowed level may hold files; when ZB_REPOSITORY_MIX_FILES_CATS_ALLOWED is False, a category that already has child categories cannot hold files.
• category.validate_file_type(extension) fails. When file_types is set, the file extension must match one of the comma-separated entries exactly (case-insensitive) — not as a substring.

The same category checks are re-applied by change_category() when moving a file to another category.

Metadata Extraction

Supported file types for automatic metadata extraction:

File Type	Metadata Extracted
PDF	Pages, author, title, creation date, producer
Images (JPG, PNG, etc.)	EXIF data, dimensions, color space, camera info
Videos (MP4, MOV, etc.)	Duration, resolution, codec, bit rate, frame rate
Office (DOCX, XLSX, PPTX)	Author, creation date, title, subject, number of sheets/pages

Thumbnail Generation

Thumbnails are automatically generated for supported image formats:

• Supported formats: JPG, PNG, GIF, BMP, TIFF
• Configurable width: Set via ZB_REPOSITORY_THUMBNAIL_WIDTH (default: 150px)
• Configurable format: Set via ZB_REPOSITORY_THUMBNAIL_EXTENSION (default: "png")

API Reference

FileController

Path: zibanu.django.repository.lib.controllers.FileController

The main controller for managing repository files.

Initialization

# Load existing file by UUID
controller = FileController(
    uuid="file-uuid-string",
    request=request  # Optional: for resolving owner
)

# Load file with custom parameters
controller = FileController(
    uuid="file-uuid-string",
    file_version=2,  # Optional: load specific version
    thumbnail_width=200,  # Optional: thumbnail dimensions
    request=request
)

# Create new file (adding mode)
from io import BytesIO
file_like = BytesIO(b"file content")
file_like.name = "document.pdf"

controller = FileController(
    file_handler=file_like,
    category=category,
    request=request
)

Properties

Property	Type	Description
is_valid	bool	Whether file passed validation
file_size	int	File size in bytes
extension	str	File extension (lowercase)
file_name	str	Storage filename (UUID-based)
file_hash	str	Computed file checksum
mime_type	str	Detected MIME type
metadata	dict	Extracted metadata
file_content	bytes	Full file content (memory-intensive for large files)
file_content_base64	str	Base64-encoded content for JSON transport
thumbnail_content	bytes	Thumbnail bytes (if available)
thumbnail_handler	File	File handler for thumbnail
full_filename	str	Full storage path (relative)
full_path_filename	str	Absolute filesystem path
random_file_name	str	Generate random filename with current extension
random_thumbnail_name	str	Generate random thumbnail filename

Methods

create_file() -> File

Creates a new file record from the currently loaded file handler.

Returns: File model instance

Example:

controller = FileController(file_handler=uploaded_file, request=request)
file_instance = controller.create_file()

save_from_file(file: Any, attrs: dict[str, Any]) -> None

Saves or updates a file from a handler and attribute dictionary.

Parameters:

• file: File-like object (Django uploaded file or similar)
• attrs: Dictionary containing file metadata and extended properties

Owner required: save_from_file resolves the owner from the request user (controller.request.user) or, when no request is available, from the owner= kwarg passed to FileController. If neither yields a user it raises ValidationError("Owner not found.").

Example:

attrs = {
    "description": "Invoice for April 2024",
    "file_extended": {
        "category": invoice_category,
        "title": "Invoice_2024_04.pdf",
        "published": True
    }
}

controller.save_from_file(file=uploaded_file, attrs=attrs)

delete_file_from_storage(artifact_paths: list[str] = None) -> None

Deletes the current file binary and, when artifact_paths is provided, the listed historical version files. Those version paths are normally collected automatically by the pre_delete signal receiver, so calling this method directly is rarely needed. Thumbnails are generated on the fly and never persisted, so there is nothing to clean up for them.

Parameters:

• artifact_paths (optional): List of absolute filesystem paths to version files to remove. Deletion is best-effort; paths that do not exist are skipped.

Example:

controller = FileController(uuid="file-uuid", request=request)
controller.delete_file_from_storage()

change_category(new_category_uuid: str) -> None

Changes the file's category.

Parameters:

• new_category_uuid: UUID of the new category

Example:

controller = FileController(uuid="file-uuid", request=request)
controller.change_category(new_category_uuid="new-category-uuid")

File Model

Path: zibanu.django.repository.models.File

Core model for storing file metadata.

Fields

Field	Type	Description
code	CharField	Unique validation code
uuid	UUIDField	Unique file identifier
file_uuid	UUIDField	Physical file identifier (changes with versioning)
owner	ForeignKey	User who created the file
generated_at	DateTimeField	File creation timestamp
description	TextField	File description
file_type	CharField	File extension
checksum	CharField	File hash value
hash_method	CharField	Algorithm used for hashing
mime_type	CharField	Detected MIME type
version	IntegerField	Current version number

Properties

file = File.objects.get(uuid=uuid)

# Access computed paths
print(file.file_name)        # "uuid-hex.ext"
print(file.thumbnail_name)   # "uuid-hex.png"
print(file.file_url)         # "zbrepo/download/uuid-hex/" (ROOT_URL + DOWNLOAD_URL)
print(file.full_file_name)   # "ZbRepository/ZbFiles/2024/3/uuid.ext"
print(file.icon_class)       # Font Awesome class (e.g., "fa-regular fa-file-pdf")

Related Objects

# Access extended properties
extended = file.file_extended
print(extended.title)       # User-friendly title
print(extended.category)    # Associated category
print(extended.metadata)    # Extracted metadata
print(extended.published)   # Publication status

# Access version history
for version in file.file_versions.all():
    print(f"Version {version.version}: {version.generated_at}")

FileVersion Model

Path: zibanu.django.repository.models.FileVersion

Stores historical versions of files.

Fields

Field	Type	Description
file	ForeignKey	Reference to the main File
uuid	UUIDField	Physical file UUID for this version
version	IntegerField	Version number
owner	ForeignKey	User who created this version
generated_at	DateTimeField	Version creation timestamp

Properties

# Access version information
version = file.file_versions.get(version=1)
print(version.file_name)      # "version-uuid.ext"
print(version.version)        # 1
print(version.generated_at)   # DateTime

Category Model

Path: zibanu.django.repository.models.Category

Hierarchical file organization.

Fields

Field	Type	Description
name	CharField	Category name
level	IntegerField	Hierarchy level (read-only)
gen_thumb	BooleanField	Generate thumbnails for this category
extract_metadata	BooleanField	Extract metadata for this category
convert_to_mp4	BooleanField	Convert videos to MP4 format
file_types	CharField	Comma-separated allowed extensions
is_public	BooleanField	Public category (no auth required)
published	BooleanField	Category visibility
parent	ForeignKey	Parent category (null for root)
uuid	UUIDField	Unique identifier
sort_by	IntegerField	Field to sort by (name, date, size)
sort_type	IntegerField	Sort order (ascending/descending)

Properties

category = Category.objects.first()

print(category.is_root)          # True if no parent
print(category.has_children)     # True if has subcategories
print(category.has_files)        # True if contains files
print(category.root_category)    # Get root ancestor
print(category.files_allowed)    # Check upload permissions
print(category.level_changed)    # Check if level was modified

REST API Endpoints

Both endpoints are served by FileViewSet (api/services/file.py), routed in urls.py under app_name = "zb_repository", and are GET-only with AllowAny permission (per-file access is enforced from the stored FileExtended/Category flags). There are no POST/PUT/DELETE endpoints — the public surface is read/download only.

Get File (JSON/Base64)

Endpoint: GET /zbrepo/get/{file_uuid}/ · action get_file · URL name zb_repository:file-get

Query Parameters:

• feature (optional, default 0): Selects the payload — 0 (FILE) returns the file content, 1 (THUMBNAIL) returns content plus its thumbnail, 3 (METADATA) returns the metadata dictionary.
• version (optional): Historical file version number. Only honored together with feature=0.

Response (feature=0):

{
    "file": "base64_encoded_content",
    "hash": "hash_value"
}

With feature=1 the payload also carries a thumbnail (Base64) key; with feature=3 the response body is the raw metadata dictionary.

Download File (Streaming)

Endpoint: GET /zbrepo/download/{file_uuid}/ · action download_file · URL name zb_repository:file-download

The download/ path segment is configurable via ZB_REPOSITORY_DOWNLOAD_URL.

Query Parameters:

• feature (optional, default 0): 0 (FILE) streams the file, 1 (THUMBNAIL) streams the thumbnail.
• type (optional, default 0): 0 (FILE) streams the raw bytes, 1 (BASE64) returns the same body Base64-encoded as text/plain (named with a .b64 suffix).
• version (optional): Historical file version number (only with feature=0).
• video_format (optional): Target container (e.g. mp4) for stored videos; the file is converted once and the result cached on disk.
• inline (optional): Serve inline (true) or as an attachment (false, the default).

Response: File stream (HTTP streaming)

Configuration

All repository settings are prefixed with ZB_REPOSITORY_ and defined in Django settings.

Directory Settings

Setting	Type	Default	Description
ZB_REPOSITORY_ROOT_DIR	str	"ZbRepository"	Base directory for all repository files
ZB_REPOSITORY_FILES_DIR	str	"ZbFiles"	Subdirectory for physical files
ZB_REPOSITORY_CACHE_DIR	str	"ZbCache"	Subdirectory for cached artifacts
ZB_REPOSITORY_ML_DIR	str	"ZbML"	Subdirectory for ML artifacts
ZB_REPOSITORY_ROOT_URL	str	"zbrepo/"	Base URL path for repository endpoints

Example Configuration

# settings.py
ZB_REPOSITORY_ROOT_DIR = "MyRepository"
ZB_REPOSITORY_FILES_DIR = "StoredFiles"
ZB_REPOSITORY_CACHE_DIR = "CachedContent"
ZB_REPOSITORY_ML_DIR = "MachineLearning"
ZB_REPOSITORY_ROOT_URL = "files/"

Cache maintenance: The ZB_REPOSITORY_CACHE_DIR directory stores converted-video artifacts keyed by source hash and target format. It has no automatic invalidation or size limit — entries accumulate as new files/formats are requested. Purge it with an external scheduled job (e.g. remove files older than N days) to bound disk usage.

Hashing and Security Settings

Setting	Type	Default	Description
ZB_REPOSITORY_HASH_METHOD	str	"sha384"	Primary hash algorithm
ZB_REPOSITORY_ALLOWED_HASH_METHODS	list	["md5", "sha1", "sha224", "sha256", "sha384", "sha512"]	Allowed hash algorithms
ZB_REPOSITORY_INVALID_EXTENSIONS	list	Conservative denylist	Blocked file extensions

Example Configuration

# settings.py
ZB_REPOSITORY_HASH_METHOD = "sha256"
ZB_REPOSITORY_ALLOWED_HASH_METHODS = ["sha256", "sha512", "sha384"]
ZB_REPOSITORY_INVALID_EXTENSIONS = [
    "php", "js", "py", "exe", "bat", "sh", "pl", "ps1",
    "cmd", "ts", "htm", "html", "css", "scr", "vbs", "vbe",
    "wsf", "msi", "msc", "jar", "com", "hta", "cpl", "pif",
    "reg", "inf"
]

Feature Flags

Setting	Type	Default	Description
ZB_REPOSITORY_VERSIONING	bool	True	Enable file versioning
ZB_REPOSITORY_MULTILEVEL_FILES_ALLOWED	bool	True	Allow nested subdirectories
ZB_REPOSITORY_MIX_FILES_CATS_ALLOWED	bool	False	Allow mixing categories in same directory
ZB_REPOSITORY_MAX_LEVEL_ALLOWED	int	3	Maximum directory nesting level

Example Configuration

# settings.py
ZB_REPOSITORY_VERSIONING = True
ZB_REPOSITORY_MULTILEVEL_FILES_ALLOWED = True
ZB_REPOSITORY_MIX_FILES_CATS_ALLOWED = False
ZB_REPOSITORY_MAX_LEVEL_ALLOWED = 3

Thumbnail Settings

Setting	Type	Default	Description
ZB_REPOSITORY_THUMBNAIL_EXTENSION	str	"png"	Thumbnail file extension
ZB_REPOSITORY_THUMBNAIL_WIDTH	int	150	Thumbnail width in pixels

Example Configuration

# settings.py
ZB_REPOSITORY_THUMBNAIL_EXTENSION = "webp"
ZB_REPOSITORY_THUMBNAIL_WIDTH = 200

Download and API Settings

Setting	Type	Default	Description
ZB_REPOSITORY_DOWNLOAD_URL	str	"download/"	Base URL path for downloads

Example Configuration

# settings.py
ZB_REPOSITORY_DOWNLOAD_URL = "files/download/"

Icon Mapping Settings

Setting	Type	Default	Description
ZB_REPOSITORY_ICON_CLASSES	dict	FontAwesome mapping	Icon classes for file types

Example Configuration

# settings.py
ZB_REPOSITORY_ICON_CLASSES = {
    "default": "fa-regular fa-file",
    "pdf": "fa-regular fa-file-pdf",
    "word": "fa-regular fa-file-word",
    "excel": "fa-regular fa-file-excel",
    "powerpoint": "fa-regular fa-file-powerpoint",
    "image": "fa-regular fa-file-image",
}

Usage Examples

Example 1: File Upload in a Django View

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods
from django.contrib.auth.decorators import login_required
from zibanu.django.repository.lib.controllers import FileController
from zibanu.django.repository.models import Category

@login_required
@require_http_methods(["POST"])
def upload_file(request):
    if "file" not in request.FILES:
        return JsonResponse({"error": "No file provided"}, status=400)
    
    uploaded_file = request.FILES["file"]
    category_uuid = request.POST.get("category_uuid")
    
    try:
        category = Category.objects.get(uuid=category_uuid)
        
        controller = FileController(
            file_handler=uploaded_file,
            category=category,
            request=request
        )
        
        file_instance = controller.create_file()
        
        return JsonResponse({
            "success": True,
            "file_uuid": str(file_instance.uuid),
            "file_url": file_instance.file_url,
            "file_size": controller.file_size
        })
    
    except Exception as e:
        return JsonResponse({"error": str(e)}, status=400)

Example 2: File Versioning and History

from zibanu.django.repository.lib.controllers import FileController

# Load current version
current = FileController(uuid="file-uuid", request=request)
print(f"Current version: {current._instance.version}")

# Load specific older version
version_1 = FileController(
    uuid="file-uuid",
    file_version=1,
    request=request
)
print(f"Version 1 hash: {version_1.file_hash}")

# List all versions
file_instance = current._instance
for version in file_instance.file_versions.all():
    print(f"Version {version.version}: {version.generated_at}")

Example 3: Batch Metadata Extraction

from zibanu.django.repository.models import File

# Extract metadata from all files in a category
files = File.objects.filter(file_extended__category__name="Invoices")

for file in files:
    controller = FileController(uuid=str(file.uuid))
    metadata = controller.metadata
    
    print(f"{file.file_extended.title}: {metadata}")

Example 4: Thumbnail Generation

from django.http import FileResponse
from zibanu.django.repository.lib.controllers import FileController

def download_thumbnail(request, file_uuid):
    try:
        controller = FileController(uuid=file_uuid, request=request)
        
        if not controller._instance.file_extended.category.gen_thumb:
            return JsonResponse({"error": "Thumbnails not allowed"}, status=403)
        
        thumbnail_content = controller.thumbnail_content
        return FileResponse(thumbnail_content, content_type="image/png")
    
    except FileNotFoundError:
        return JsonResponse({"error": "Thumbnail not found"}, status=404)

Example 5: Category Management and Organization

from zibanu.django.repository.models import Category

# Create hierarchical categories
root = Category.objects.create(
    name="Documents",
    published=True
)

invoices = Category.objects.create(
    name="Invoices",
    parent=root,
    file_types="pdf,jpg,png",
    extract_metadata=True,
    published=True
)

contracts = Category.objects.create(
    name="Contracts",
    parent=root,
    file_types="pdf,docx",
    extract_metadata=True,
    published=True
)

# Query categories
print(f"Root categories: {Category.objects.filter(parent__isnull=True).count()}")
print(f"Invoices path: {invoices}")  # Documents // Invoices
print(f"Has children: {root.has_children}")
print(f"Total files: {root.has_files}")

Example 6: Advanced File Operations

from zibanu.django.repository.lib.controllers import FileController
from zibanu.django.repository.models import File

# Update existing file
controller = FileController(uuid="file-uuid", request=request)

attrs = {
    "description": "Updated description",
    "file_extended": {
        "title": "New Title",
        "published": True
    }
}

# Save new version
controller.save_from_file(file=new_file_handler, attrs=attrs)

# Retrieve all file information
print(f"Size: {controller.file_size} bytes")
print(f"Hash: {controller.file_hash}")
print(f"MIME: {controller.mime_type}")
print(f"Metadata: {controller.metadata}")

Testing

Run Unit Tests

# Run all repository tests
python manage.py test zibanu.django.repository

# Run specific test class
python manage.py test zibanu.django.repository.tests.TestFileController

# Run with verbose output
python manage.py test zibanu.django.repository -v 2

# Run tests with logging
python manage.py test zibanu.django.repository --debug-mode

Run with Coverage

# Generate coverage report
coverage run --source='zibanu.django.repository' manage.py test zibanu.django.repository
coverage report

# Generate HTML coverage report
coverage html
# Open htmlcov/index.html in browser

Testing Best Practices

• Use Django's test client for view testing
• Mock external services (file storage, image processing)
• Test both success and error scenarios
• Verify database state after operations
• Clean up test files from media directory

Performance Considerations

File Content Loading

The file_content property loads the entire file into memory. For large files:

• Avoid using file_content for files > 100MB
• Use file_handler to stream content instead
• Prefer HTTP streaming endpoints for downloads

Example of Streaming Large Files

def download_large_file(request, file_uuid):
    controller = FileController(uuid=file_uuid, request=request)
    handler = controller.file_handler
    
    # Stream file instead of loading into memory
    response = FileResponse(handler)
    response['Content-Disposition'] = f'attachment; filename="{controller.file_name}"'
    return response

Metadata Extraction

Metadata extraction can be CPU-intensive for:

• Large PDFs (100+ pages)
• High-resolution images (4K+)
• Long videos (1+ hour)

Recommendation: Schedule metadata extraction as background tasks for large files.

Thumbnail Generation

Thumbnail generation for large images can use significant CPU and memory.

Recommendation: Pre-generate thumbnails during upload or use lazy generation with caching.

Database Query Optimization

# Optimized with prefetch_related and select_related
files = File.objects.prefetch_related(
    'file_versions',
    'file_extended__category',
    'owner'
).select_related('owner').all()

# More efficient than N+1 queries
for file in files:
    print(file.file_extended.title)
    print(file.owner.username)

Storage Optimization

• Use solid-state drives (SSDs) for better I/O performance
• Implement file cleanup policies for old versions
• Monitor disk space usage regularly
• Consider archiving old files to external storage

Requirements

Python and Django Versions

• Python 3.11+
• Django 4.0+

Core Dependencies

Dependency	Purpose	Min Version
Django	Web framework	4.0+
zibanu.django	Base utilities	Latest
Pillow	Image processing	8.0+

Optional Dependencies

Feature	Library	Purpose
PDF Metadata	pymupdf or pypdf	Extract PDF metadata
Image Processing	Pillow	Metadata extraction, thumbnails
Video Processing	PyAV or ffmpeg-python	Video metadata, MP4 conversion
Advanced Imaging	opencv-python	Advanced image analysis

Media Storage

• Minimum free space: 100MB (recommended based on expected file volume)
• Permissions: Write access required on MEDIA_ROOT
• Performance: SSD recommended for concurrent operations
• Backup: Regular backups recommended for production environments

System Configuration

# settings.py - Django Media Configuration
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

# Ensure proper permissions
# Linux/Mac: chmod 755 media/

Troubleshooting

"File not found" Error

Symptom: ValidationError: File not found.

Cause: File was deleted from filesystem but database record remains.

Solution:

# Identify orphaned records
from zibanu.django.repository.models import File
import os
from django.conf import settings

media_root = settings.MEDIA_ROOT
for file in File.objects.all():
    full_path = os.path.join(media_root, file.full_file_name)
    if not os.path.exists(full_path):
        print(f"Orphaned: {file.uuid}")
        # Consider deleting: file.delete()

Thumbnail Generation Fails

Symptom: FileNotFoundError or empty thumbnail.

Causes:

• Pillow not installed
• Category has gen_thumb=False
• Unsupported image format

Solution:

# Install Pillow
pip install Pillow

# Verify category settings
python manage.py shell
>>> from zibanu.django.repository.models import Category
>>> cat = Category.objects.get(id=1)
>>> print(cat.gen_thumb)  # Should be True

"Invalid hash method" Error

Symptom: ValidationError: Invalid hash method.

Cause: Hash method not in allowed list.

Solution:

# settings.py
from django.conf import settings

# Verify current method
print(settings.ZB_REPOSITORY_HASH_METHOD)  # Should be 'sha384' or 'sha256'

# Update if needed
ZB_REPOSITORY_HASH_METHOD = "sha256"
ZB_REPOSITORY_ALLOWED_HASH_METHODS = ["sha256", "sha384", "sha512"]

Metadata Extraction Returns Empty Dict

Symptom: metadata = {} despite extract_metadata=True.

Note: Images now degrade gracefully. An unreadable EXIF field — for example a GPS coordinate stored as an invalid 0/0 rational (a camera that wrote a GPS block without a fix) — is simply omitted, while the format, dimensions and the rest of the EXIF are still returned. A single bad tag no longer wipes the whole dictionary, so a genuinely empty {} for an image points to a missing Pillow, an unreadable/corrupted file, or an image type outside SUPPORTED_MIME_TYPES.

Causes of a truly empty dict:

• The MIME type is unsupported by design — e.g. legacy application/msword (.doc) files, which are not listed in SUPPORTED_MIME_TYPES.
• The required extractor library is not installed (pymupdf for PDF, Pillow for images, PyAV for video).
• A PDF, Office/OpenDocument or video file is corrupted or unreadable.

Solution:

# Install optional dependencies
pip install pymupdf Pillow PyAV

# Verify supported types
python manage.py shell
>>> from zibanu.django.repository.lib.classes import FileMetadata
>>> from zibanu.django.lib import FileUtils
>>> file = FileUtils(file_name="test.pdf", file_dir="/path")
>>> meta = FileMetadata(file)
>>> print(meta.metadata)

URL Reverse Errors or 404 on Endpoints

Symptom: NoReverseMatch or file download returns 404.

Cause: Repository URL patterns not included in main urls.py.

Solution:

# main urls.py
from django.urls import path, include

urlpatterns = [
    # ...
    path("zbrepo/", include("zibanu.django.repository.urls")),
]

# Test the URL pattern (app_name is "zb_repository")
python manage.py shell
>>> from django.urls import reverse
>>> reverse('zb_repository:file-get', args=['file-uuid'])
>>> reverse('zb_repository:file-download', args=['file-uuid'])

Performance Issues with Large Files

Symptom: Slow uploads or downloads, high memory usage.

Solution:

• Use file streaming instead of loading entire content
• Implement chunked uploads for large files
• Configure separate storage backend for files
• Use CDN for file distribution

# Use Django storages for external storage
STORAGES = {
    'default': {
        'BACKEND': 'storages.backends.s3boto3.S3Boto3Storage',
    },
}

Missing Metadata or MIME Types

Symptom: mime_type is None or metadata extraction skipped.

Causes:

• extract_metadata=False in category
• File not properly uploaded
• Corrupted file header

Solution:

# Enable metadata extraction
category.extract_metadata = True
category.save()

# Re-process file
controller = FileController(uuid="file-uuid", request=request)
print(controller.mime_type)
print(controller.metadata)

License

Copyright © 2019-2025 CQ Inversiones SAS. All rights reserved.

Licensed under the Apache License 2.0. See LICENSE file for details.

Support

For issues, feature requests, or contributions:

Reporting Issues

1. Include Information:

   • Error traceback and full message
   • Django and Python versions
   • Relevant settings configuration
   • File UUID and file type
   • Steps to reproduce the issue

2. Debug Information:

python manage.py shell
>>> from zibanu.django.repository.models import File
>>> file = File.objects.get(uuid="uuid")
>>> print(file.__dict__)
>>> print(file.file_extended.__dict__)

3. Check Logs:
   • Enable Django logging for zibanu.django.repository
   • Review media directory permissions
   • Verify disk space availability

Contact

• Issues: File an issue with reproduction steps
• Documentation: Refer to this README
• Community: Consult zibanu.django documentation

Changelog

All notable changes to zibanu.django.repository are documented here, one section per principal version (x.y.z). Pre-releases of the same core (-dev, -alpha, -beta) are folded into their section; several intermediate plain-core bumps in this package's history (e.g. 2.0.6–2.0.9, 2.1.0–2.1.1, 2.2.0-alpha.0, 3.0.4, 3.0.6–3.0.7) were never tagged or shipped as an independent installer build and are folded the same way, into whichever tagged release they led up to.

[4.0.0] — 2026-10-01

Added

• Category listing (get_categories_files_allowed) now respects each category's published flag when computing the allowed set.
• The file admin now persists the selected category across "Save and add another", and gives precedence to a category passed via the query string over a previously persisted one.
• Corrupted/damaged packets in a source video no longer abort the whole conversion — they are discarded with a warning and the rest of the frames are converted.
• Bundled OpenAPI/Swagger specs (with a DEBUG-only Swagger UI), and a refreshed README.md / README.es.md and public docstrings.

Changed

• Breaking: the file/category manager class was renamed File → FileManager, and the category-tree helper set_children_publish() was renamed to unpublish_children() — its publish direction had been a silent no-op, so the corrected, renamed method is the first version that actually publishes children.
• Breaking: a category's allowed file-type restriction can no longer be bypassed with a substring match against the extension list; every call site now routes through Category.allowed_file_types.
• Breaking: access_type and category.is_public now also constrain authenticated callers, not only anonymous ones — closing a gap where a logged-in user could read files from a private category.
• A File row is now written to the database only after the physical upload succeeds; a failed disk write no longer leaves an orphaned database record.
• Video frame timestamps (constant-frame-rate re-timing) and audio timestamps are now computed explicitly instead of trusting the source container or letting the encoder invent them, fixing out-of-order frames and audio/video drift on re-encoded videos.
• Non-finite float values (NaN/Infinity) are sanitized out of extracted file metadata before it is stored or serialized.
• Image metadata extraction no longer discards the whole metadata payload when a single GPS coordinate is invalid — the rest of the metadata is preserved.
• A further internal hardening pass (~70 items) removed dead code and unused imports/parameters, corrected docstrings/messages that no longer matched behavior, and closed minor consistency gaps across the API services, apps.py, PDF/metadata generation, thumbnailing, video conversion, filters and enums — none of it user-observable beyond the items already listed here and under Fixed.

Fixed

• Saving a new version of an existing file wrote the binary under the file's original year/month directory instead of the current one, because a positional-argument shift silently bound today's date to the wrong parameter.
• get_path() no longer resolves to a path outside its configured base directory.
• publish_categories() is now atomic per row and reports a per-row result instead of an all-or-nothing update.
• Specific ValidationErrors raised during file/category validation are no longer swallowed and rewritten into a generic error by an overly broad except Exception.
• PDF generation now closes the source file handle before cleaning up its temporary files.
• A category's cached level is refreshed after save(), so a subsequent level_changed check no longer reads stale state.
• change_category() now validates the target category and reports an error instead of failing silently when it doesn't exist.
• The background file-deletion thread now handles its own exceptions, runs as a daemon thread, and closes its own database connection instead of leaking one.
• Thumbnail generation from an RGBA source image no longer fails silently when saved as JPEG.
• A ZIP handle opened while extracting Office-document metadata is now closed.
• Admin category/file forms replaced an unguarded Category.objects.get(...) with a field-scoped ValidationError, so a missing category shows a normal form error instead of a server error.
• File deletion no longer orphans version files on disk: a pre_delete signal handler now captures each version's physical path before the database cascade removes the FileVersion rows, so the background cleanup can still find them.
• Admin category filters, get_root_category() and set_children_level() no longer run one query per row/node (select_related, a cycle guard, and one query pair per tree depth respectively); Category.save() skips cascade-recalculation queries on insert; the file admin changelist uses list_select_related.

urls.py's file_id path converter was briefly tightened to <uuid:file_id> and then reverted back to <str:file_id> for compatibility before this version shipped — no net change for callers of the current main-tracked API.

[3.0.5] — 2026-07-21

Fixed

• Video conversion (the video_format query parameter) is now only applied when the stored file's MIME type is an actual video document; images, PDFs and other documents are always streamed as their original file even if a client requests a video format.
• Fixed conversion of .m4v sources — they are now force-converted to .mp4 instead of failing.

[3.0.3] — 2026-07-20

• No user-facing changes. Published minutes after 3.0.2 to finalize the version number for distribution; the code is unchanged from 3.0.2.

[3.0.2] — 2026-07-20

Added

• FileController can now change a file's category after upload, not only at creation time.

Changed

• Breaking: video metadata extraction and thumbnail generation now use PyAV instead of OpenCV + ffmpeg; the opencv-python and ffmpeg-python dependencies were dropped from the package. This also fixes swapped red/blue color channels and an always-zero duration that OpenCV reported for video thumbnails/metadata. Deployments that installed those packages specifically for this library can remove them.
• Video downloads are remuxed (no re-encode) when the source codec already matches the requested container, falling back to a full transcode only when needed; FFmpeg threading is enabled and the x264 preset changed to veryfast (roughly 4x faster transcodes).
• Converted videos are now cached on disk, keyed by file hash and target format, under the new ZB_REPOSITORY_CACHE_DIR setting (default "ZbCache") — only the first request for a given file/format pays the conversion cost.
• A filesystem path can now be passed directly as the conversion source, avoiding an in-memory/temp-file copy (a chunked-copy fallback still applies for non-local storage handlers).

Fixed

• Muxing AVI/MP3 audio into MP4 no longer raises EINVAL: decoded audio frame timestamps are cleared and re-timed by the AAC encoder instead of reused as-is.
• Fixed silent loss of the last frames of a converted video: end-of-file flush packets are now decoded instead of skipped.
• get_converted_video() returns None on failure instead of an unusable File(None).
• FileController.full_path_filename is now a real property (it was missing its @property decorator).

[3.0.1] — 2026-05-15

Added

• PDF files can now generate a thumbnail (previously only image and video sources were supported).
• Permission validation on the (now on-demand) thumbnail-generation flow.

Changed

• Breaking: thumbnails are now generated on demand instead of eagerly at upload time; the automatic thumbnail-creation step at save time was removed.
• create_file was reworked to fix the file-saving error described below and related file-creation issues.

Removed

• Breaking: the now-unused ZB_REPOSITORY_THUMBNAILS_DIR setting was removed, along with the internal helper that persisted thumbnails at save time.

Fixed

• Fixed an error saving a new file ("error save from file").
• Cleaned up assorted unused variables left over from earlier development.

[3.0.0] — 2026-03-07

Added

• Video support: thumbnail generation from a representative video frame, and extraction of video metadata (duration, codecs, frame size, GPS/creation info).
• A hash field on the file model, exposed through the extended GET/download services for integrity and de-duplication checks.
• File version date tracking (generated_at) as its own persisted field, added via migration independently of created_at.
• A configurable list of forbidden file extensions, to block risky uploads.
• Root-category creation now reports a clear validation error instead of an obscure one.
• Support for streaming file content directly (file_content) in addition to file paths.
• Documentation and initial scaffolding for an optional MQTT client entry point in apps.py (groundwork for the sibling zibanu.django.mqtt package's later integration; not itself a working client in this package).

Changed

• Breaking: the full_file_name property was replaced by full_filename plus a new full_path_filename property, fixing an extension-handling bug in file_utils.py / file_controller.py in the same pass.
• Video-to-format conversion was narrowed to .avi sources specifically, after broader automatic conversion proved unreliable.

Fixed

• Assorted errors surfaced during this development cycle around file deletion, thumbnail storage/generation, and file-extension handling.

This range spans roughly seven months (Aug 2025 – Mar 2026) across ten alpha and eight beta pre-releases before the production tag. It is the package's biggest feature build-out (over 30 tracked user stories, ZDR-3…ZDR-36); the summary above groups them by user-visible capability rather than listing each one. The history also briefly touched an abandoned 2.2.0-alpha.0 / 2.1.6-beta.x / 2.1.7-dev.0 line that was discarded in favor of 3.0.0 and never shipped.

[2.1.5] — 2025-08-23

Added

• User-configurable icon customization for categories/files, together with a switch from BigAutoField to AutoField for the package's auto-incrementing primary keys.

Fixed

• Category level recalculation, including a persistent bug in nested-category level recalculation.
• Unpublishing a category no longer leaves the public documents it contains publicly reachable.

[2.1.2] — 2025-06-21

Added

• Signal-based auditing of user file/category access.

Changed

• The repository's "published" filter now also applies to a category's parent categories, not only to leaf categories (paired with a sort-order parameter added to the sibling zb_query_filter template tag, shipped from zibanu.django.template).

Fixed

• Root-category creation no longer fails.
• A UniqueConstraint on FileVersion that raised on Django versions older than 4.1 was corrected for compatibility.
• A version-numbering bug when adding new files.

[2.0.5] — 2025-04-21

Added

• Thumbnail generation from the original image, via a new FileThumbnail class and supporting fields on the File model.

Fixed

• A small file_controller.py bug carried over from the 2.0.4 release.

[2.0.4] — 2025-04-05

Fixed

• A small file_controller.py bug ahead of the PyPI 2.0.4 release.

[2.0.2] — 2025-04-01

Added

• Base Geometry model and its supporting enum/migration (foundation later reused by the split-out zibanu.django.gis package).

Fixed

• FileAdmin: an error in FileUtils's file-type validation, and a missing file-existence check in file_admin_view.py's clean() that triggered file-type validation even when it wasn't required.

[2.0.0] — 2025-03-14

Added

• Initial release of zibanu.django.repository as its own installable package, split out of the zibanu-django monorepo.