django-camel-spitter

Hi. I am a very rude camel 🐫 and I like to spit logs 💦 directly into your database 🗄️.

Introduction

Project django-camel-spitter adds a new handler to standard django logging system.

Purpose of this handler is to store logs straight to the database, primary with simplicity and opportunity to easy extend this solution.

Installation

# pip
pip install django-camel-spitter

# pipenv
pipenv install django-camel-spitter

# poetry
poetry add django-camel-spitter

Setup

1. Adding camel_spitter to settings.INSTALLED_APPS:

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'camel_spitter'
)

2. Adding additional logging db connection to settings.DATABASES:

This additional connection is needed for handling DB transaction atomicity. Exception without own connection cannot create DB log, when rollback was made. Transactions in django are default handled on default DB connection. So rollback will stop default connection execution, but logging connection not.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'HOST': os.getenv('DATABASE_HOST'),
        'PORT': os.getenv('DATABASE_PORT', 5432),
        'NAME': os.getenv('DATABASE_NAME'),
        'USER': os.getenv('DATABASE_USER'),
        'PASSWORD': os.getenv('DATABASE_PASSWORD', None)
    },
    'logging': {
        'ENGINE': 'django.db.backends.postgresql',
        'HOST': os.getenv('DATABASE_HOST'),
        'PORT': os.getenv('DATABASE_PORT', 5432),
        'NAME': os.getenv('DATABASE_NAME'),
        'USER': os.getenv('DATABASE_USER'),
        'PASSWORD': os.getenv('DATABASE_PASSWORD', None)
    }
}

3. Adding model, filter and handler to settings.LOGGING:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'filters': {
        'db_filter': {
            '()': 'camel_spitter.db_filter.DBFilter',
        },
    },
    'handlers': {
        'db': {
            'level': 'INFO',
            'class': 'camel_spitter.db_handler.DBHandler',
            'model': 'tests.models.BasicLogEntry',  # path to your custom model
            'filters': ['db_filter']
        }
    },
    'loggers': {
        'logger': {
            'handlers': ['db'],
            'level': 'INFO'
        }
    }
}

4. Creating a log model:

Only importance is inheritance from camel_spitter.models.BaseLogModel.

from camel_spitter.models import BaseLogModel


class BasicLogEntry(BaseLogModel):
    class Meta:
        app_label = 'tests'
        db_table = 'log_entries'
        default_permissions = ()

Example

1. Quick use

If you did all setup steps, you are ready to log to the database.

import logging
from app.models import BasicLogEntry

logging.getLogger('logger').error('Foo Bar Error')
logged_information = BasicLogEntry.objects.get(message='Foo Bar Error')

# logged_information = {BasicLogEntry}BasicLogEntry object (1)

---

2. Example of extended model

If you like to log some additional data, for example: [request.body, user_name], you need to first add these fields to model:

from camel_spitter.models import BaseLogModel
from django.db import models

class ExtendedLogEntry(BaseLogModel):
    class Meta:
        app_label = 'tests'
        db_table = 'extended_log_entries'
        default_permissions = ()

    request_body = models.JSONField(null=True)
    user_name = models.CharField(max_length=100, null=True)

As a second, you need to add these additional data to logging:

import logging

logging.getLogger('logger').error('Foo Bar Error', extra={
    'request_body': json.loads(request.body), 'user_name': 'Foo Bar'
})

Important notes

Testing with pytest and file DB

When tests are made with pytest library and file DB like SQLite, tests will make two separate database files from specified connections even though path and engine are the same. So retrieving log is needed to be executed with using like BasicLogEntry.objects.using('logging').get(message='Foo Bar'). To avoid this issue you can specify mirror database in test settings.

    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': DB_PATH,
    },
    'logging': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': DB_PATH,
        'TEST': {
            'MIRROR': 'default',
        }
    },

---

Made with ❤ by Adam Žúrek & BACKBONE s.r.o.

0.1.0 : 2020-06-27

• Initial release

0.2.0 : 2020-06-29

• Changed python version requirement from "^0.8" to "^0.6"

0.3.0 : 2020-08-04

• DB logging is successful even if it is wrapped in a transaction, which made rollback.

0.3.1 : 2020-11-02

• Changed django version requirement from "==3.0.*" to ">=2.0"

0.3.2 : 2021-05-10

• 🐪 Added id AutoField to BaseLogModel (According to Django 3.2+ recommendations)