How to set up?

1. Add fieldlogger to your INSTALLED_APPS

2. Run python manage.py migrate to initialize the model

3. Add FIELD_LOGGER_SETTINGS to your settings.py file.

FIELD_LOGGER_SETTINGS = {
    'ENCODER': 'path.to.your.json.Encoder', # (default: None)
    'DECODER': 'path.to.your.json.Decoder', # (default: None)
    'LOGGING_ENABLED': True, # (default: True)
    'FAIL_SILENTLY': True, # (default: True)
    'LOG_ACTOR': True, # (default: True)
    'LOG_REMOTE_ADDR': True, # (default: True)
    'LOGGING_APPS': {
        'your_app': {
            'logging_enabled': True, # (default: True)
            'fail_silently': True, # (default: True)
            'log_actor': True, # (default: True)
            'log_remote_addr': True, # (default: True)
            'models': {
                'YourModel': {
                    'logging_enabled': True, # (default: True)
                    'fail_silently': True, # (default: True)
                    'log_actor': True, # (default: True)
                    'log_remote_addr': True, # (default: True)
                    'fields': ['field1', 'field2'], # or '__all__' (default: [], no field is logged)
                    'exclude_fields': ['field3', 'field4'], # (default: [])
                    'callbacks': [
                        lambda instance, fields, logs: print(instance, fields, logs),
                        'yourapp.app.callbacks.your_function_name'
                    ], # (default: [])
                },
            },
            'callbacks': [
                lambda instance, fields, logs: print(instance, fields, logs),
                'yourapp.app.callbacks.your_function_name'
            ], # (default: [])
        },
    },
    'CALLBACKS': [
        lambda instance, fields, logs: print(instance, fields, logs),
        'yourapp.app.callbacks.your_function_name'
    ], # (default: [])
}

• ENCODER and DECODER are optional. If you want to encode/decode your model instance fields, you can specify your encoder/decoder classes here. Your encoder/decoder classes must be subclasses of json.JSONEncoder and json.JSONDecoder respectively. To keep the default handling of Django field values, subclass fieldlogger.encoding.Encoder. Changing them does not require a migration.

• LOGGING_ENABLED is optional. If you want to disable logging globally, you can set this to False. To disable it only for a block of code, see Disabling logging temporarily.

• FAIL_SILENTLY is optional. If it is set to False, exceptions will be raised if the callback function fails.

• LOG_ACTOR and LOG_REMOTE_ADDR are optional. Set them to False not to store the user or the client’s IP address (see Recording who made the change).

• LOGGING_ENABLED, FAIL_SILENTLY, LOG_ACTOR and LOG_REMOTE_ADDR can also be set per app and per model, in lowercase (logging_enabled, fail_silently, log_actor, log_remote_addr). Each one is on unless it is False in the global settings, the app or the model.

• LOGGING_APPS apps to be logged.

  • models models to be logged.

    • fields the fields to log: a list of field names, or '__all__' to log every field. If it is not specified, no field is logged.

    • exclude_fields is optional. Fields listed here are never logged, even if they are in fields. Combine it with 'fields': '__all__' to log every field except a few.

      Only fields with their own database column are loggable: reverse relations and database-generated fields (GeneratedField) are always skipped. Many-to-many fields are supported through the m2m_changed signal (see Many-to-many fields).

    • callbacks is optional. If you want to add a callback function to be called after logging all models in all apps, you can add it here. Callback functions must be callable objects. You can optionally specify a callback function path in your configuration. The best practice is to place your callback function in yourapp/callbacks.py. Callback functions must have three parameters as follows:

      # callback as a named function
      def your_callback(instance, fields, logs):
          # your code here
      
      # callback as a lambda function
      lambda instance, fields, logs: # your code here

      • instance the model instance that is being logged.

      • fields list of fields that are being logged.

      • logs dict of logs that are being created. The key is the field name and the value is the FieldLog instance.

How it works?

• Obtains the FIELD_LOGGER_SETTINGS from your respective settings file based on your environment.

• Initializes LOGGING_APPS with the relative project paths of your models based on your configuration variable.

• Binds to the pre_save and post_save signals of each loggable model, and to the m2m_changed signal of each loggable many-to-many field.

• For each field specified in the configuration variable, creates a record in the FieldLog model for each instance update.

• The logged values are read back from the database after saving, so they are the stored ones: expressions such as F('count') + 1 are logged as their result, and values the database rounds or converts (e.g. a DecimalField) as stored. This costs one extra query per save(), and per batch in bulk operations.

• Fixture loading (loaddata) is a restore, not a change, so it is never logged.

Example

This section serves as a small example to demonstrate how to use this package.

Supposing you have this configuration in your settings.py file:

FIELD_LOGGER_SETTINGS = {
    'LOGGING_APPS': {
        'drivers': {
            'models': {
                'Driver': {
                    'fields': ['driver_name']
                },
            },
        },
    },
}

Supposing you have a model called Driver with fields called latest_speed, driver_name, driver_id:

from fieldlogger.models import FieldLog

driver = Driver.objects.last()
driver.latest_speed = 5
driver.save()  # fieldlogger won't create a record since 'latest_speed' was not among the loggable fields

driver.driver_name = 'John Doe'
driver.save()  # a record with this driver is created

driver.driver_name = 'Jane Doe'
driver.save()  # a record with this driver is created

instance_id = driver.id
app_label = driver._meta.app_label
model_name = driver._meta.model_name

log = FieldLog.objects.filter(instance_id=instance_id, app_label=app_label, model_name=model_name).last()
print(log.field, log.old_value, log.new_value)  # prints: driver_name John Doe Jane Doe

Callback example

Supposing you have this function in yourapp/callbacks.py which sets the extra_data field of the FieldLog model:

def set_extra_data_for_driver_name(instance, fields, logs):
    log = logs.get('driver_name')
    if log:
        log.extra_data = {
            'name_length': len(log.new_value)
        }
        log.save()

Then you can add this callback function to your configuration like this:

FIELD_LOGGER_SETTINGS = {
    'LOGGING_APPS': {
        'drivers': {
            'models': {
                'Driver': {
                    'fields': ['driver_name'],
                    'callbacks': [
                        'yourapp.callbacks.set_extra_data_for_driver_name'
                    ]
                },
            },
        },
    },
}

The model structure

This package provides you a django model which is called FieldLog; which tracks each change to a model instance specified in your configuration mapping. An example record is as follows:

{
    'id': 2,
    'app_label': 'drivers',
    'model_name': 'driver',
    'instance_id': 1,
    'field': 'driver_name',
    'timestamp': datetime.datetime(2024, 1, 16, 9, 1, 14, 619568, tzinfo=<UTC>), # set when the log is created
    'old_value': 'John Doe',
    'new_value': 'Jane Doe',
    'extra_data': {}, # this is a JSONField, you can store any extra data here using callbacks or by overriding it directly
    'created': False, # this is a boolean field, if it is True, it means that instance is a newly created instance
    'actor_id': '7', # who made the change, see "Recording who made the change" (default: None)
    'actor_username': 'admin', # (default: None)
    'remote_addr': '127.0.0.1', # (default: None)
}

Additionally, FieldLog model provides the following properties:

• model: returns the model class of the instance that is being logged.

• instance: returns the instance that is being logged.

• previous_log: returns the previous log of the same field of the same instance, if any.

• actor: returns the user who made the change, if known and not deleted since.

Values are stored as JSON and converted back to Python objects when a log is loaded:

• Binary values are stored base64-encoded, so any binary content is supported.

• Foreign key values are resolved back to model instances lazily (no query until the value is accessed). If the related instance was deleted, an unsaved instance carrying only the primary key is returned, so reading old logs never fails.

• Models with a composite primary key (Django >= 5.2) are not supported.

The FieldLoggerMixin

This package provides you a mixin class which is called FieldLoggerMixin. This mixin class provides you the following property:

• fieldlog_set since the FieldLog model has not a direct relation to the model that is being logged, you can use this property to get the logs of the instance that is being logged.

  driver = Driver.objects.last()
  logs = driver.fieldlog_set.all()

The FieldLoggerManager

Django signals are not fired on bulk operations, so changes made through bulk_create and bulk_update are not logged by default. This package provides a manager called FieldLoggerManager that overrides both methods to log field changes as well:

from django.db import models

from fieldlogger.managers import FieldLoggerManager

class Driver(models.Model):
    # ...

    objects = FieldLoggerManager()

The methods are defined on FieldLoggerQuerySet, the QuerySet the manager returns, so they are also logged after chained calls such as Driver.objects.using('other').bulk_create(...) or Driver.objects.filter(...).bulk_update(...), and through their async variants abulk_create and abulk_update (available on the manager from Django 4.1, on the QuerySet on every version).

Your own QuerySet keeps logging: FieldLoggerManager.from_queryset(YourQuerySet), and a FieldLoggerManager subclass whose get_queryset() returns another QuerySet, both combine it with FieldLoggerQuerySet. You can also subclass FieldLoggerQuerySet and use YourQuerySet.as_manager().

All of them accept two extra keyword arguments:

• log_fields set it to False to skip logging for that call (default: True). To skip it for every operation in a block, see Disabling logging temporarily.

• run_callbacks set it to False to skip the configured callbacks for that call (default: True).

Driver.objects.bulk_create([Driver(driver_name='John Doe')])
Driver.objects.bulk_update(drivers, ['driver_name'], run_callbacks=False)

With bulk_create(update_conflicts=True), objects that update an existing row are logged as changes of the update_fields instead of as creations.

QuerySet.update() does not fire signals and is not logged.

Many-to-many fields

Many-to-many changes do not go through save(), so they are logged from the m2m_changed signal instead. Any loggable many-to-many field gets one log per change, holding the sorted lists of related primary keys before and after. Given a cars field on Driver with 'fields': ['cars'] in its configuration:

class Driver(FieldLoggerMixin, models.Model):
    cars = models.ManyToManyField(Car)

driver.cars.add(car1, car2)
log = driver.fieldlog_set.get(field='cars')
print(log.old_value, log.new_value)  # prints: [] [1, 2]

Changes made from the reverse side of the relation (the car.driver_set accessor) are logged too, but always on the model that declares the field (Driver here); Car gets no logs. Every driver affected by the change gets its own log:

car3.driver_set.add(driver, other_driver)
log = driver.fieldlog_set.filter(field='cars').last()
print(log.old_value, log.new_value)  # prints: [1, 2] [1, 2, 3]

car3.driver_set.clear()  # logs [1, 2, 3] -> [1, 2] on driver and [3] -> [] on other_driver

• add, remove, set and clear are logged, from both sides of the relation.

• Changes made directly on an explicit through model (e.g. Membership.objects.create(...)) do not fire m2m_changed, so they are not logged; this mirrors Django’s own behavior.

Recording who made the change

Add FieldLoggerMiddleware after Django’s AuthenticationMiddleware to record, in every log created while handling a request, the user and the client’s IP address:

MIDDLEWARE = [
    # ...
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'fieldlogger.middleware.FieldLoggerMiddleware',
]

The logs then get:

• actor_id: the primary key of the user, as text, like instance_id in the database. Filter with FieldLog.objects.filter(actor_id=user.pk), and compare it with str(user.pk) in Python.

• actor_username: the user’s get_username() at the time of the change.

• remote_addr: the client’s IP address.

Values that are not known are None, except actor_id and actor_username on Oracle, which reads empty text columns back as '': test them with if log.actor_id:.

There is no foreign key to the user model, so both actor_id and actor_username stay when the user is renamed or deleted; log.actor loads the user, or returns None if it was deleted. Anonymous requests record only the IP address, and request.user is only loaded when a log is created. The middleware works with both WSGI and ASGI.

Code that does not run in a request, such as Celery tasks, management commands or scripts, can set the user with set_actor(), which can also decorate a function:

from fieldlogger import set_actor

with set_actor(user):
    driver.save()  # logged with this user

To keep the user or the IP address out of the logs of some apps or models, set log_actor or log_remote_addr to False in their configuration, or LOG_ACTOR or LOG_REMOTE_ADDR to turn them off everywhere. Without log_actor, request.user is not even loaded for those logs:

FIELD_LOGGER_SETTINGS = {
    'LOGGING_APPS': {
        'hr': {
            'log_actor': False,  # no app model records the user
            'models': {...},
        },
        'drivers': {
            'models': {
                'Driver': {
                    'fields': '__all__',
                    'log_remote_addr': False,  # only this model skips the IP
                },
            },
        },
    },
}

The IP address is personal data. It comes from REMOTE_ADDR, which behind a reverse proxy is the proxy’s address. Override get_remote_addr() to read the header your proxy sets instead; do not read X-Forwarded-For unless your proxy sets it, since clients can send it with any value. Values that are not valid IP addresses are not stored.

from fieldlogger.middleware import FieldLoggerMiddleware

class ProxyFieldLoggerMiddleware(FieldLoggerMiddleware):
    def get_remote_addr(self, request):
        return request.headers.get('X-Real-IP')

django.contrib.auth is not required: without it, the middleware records only the IP address.

Disabling logging temporarily

disable_logging() turns logging off inside a with block, for example while importing data:

from fieldlogger import disable_logging

with disable_logging():
    for driver in drivers:
        driver.save()  # not logged
    Driver.objects.bulk_create(new_drivers)  # not logged

Inside the block, save(), many-to-many changes and the bulk operations of FieldLoggerManager and its querysets (including abulk_create and abulk_update) create no logs and run no callbacks, and the queries that read the previous values are skipped. Changes made inside the block are not logged later either: the next log of a field starts from the value stored at that point.

The state is kept in a contextvars.ContextVar, so it only affects the current thread or async task: other requests keep logging. Blocks can be nested, and it can also decorate a function:

@disable_logging()
def import_drivers(rows):
    ...

In async code, use it as a with block inside the coroutine.

License

Copyright (c) 2024 Sergio Rodríguez

Released under the MIT License. See the LICENSE file for details.