Motivation

The problem which led to this solution is the need to consume data from end users. Usually, and especially when the users are clients, this data can not be relied upon to meet the input specifications of your system.

This usually leads one of these sub-optimal solutions:

• Have custom scripts to pre-process data per client

• Adding a bunch of if statements, or other similar logic to the core product

• Attempting to make transformation scripts generic enough to re-use, thus making them less useful for their primary purpose and harder to debug

• Hard-coding data into transformation scripts

There is often a custom transformation per client or project, so these solutions do not scale well.

The goals of AnyFilter are:

• Minimize the amount of custom code in the primary codebase

• Store configurations outside the application’s database in a portable format

• Allow updates of configuration data without deploying code (privileged users may even edit a configuration via some sort of GUI)

Benefits

• create pluggable data filters

• store configurations outside your application’s database

• easily back up and restore configurations

• easily duplicate configurations across servers

Planned features

• Export and import configs

• Convert configs to and from HTML forms for easy front-end functionality

• Easily revert to a prior config

• Comprehensive unit tests

Sample usage

#!/usr/bin/env python

"""
This is a simple example of the use of the Filter class. In this case, a
dictionary has some keys renamed. This is a trivial example; filters
can be as complex as required.
"""

from anyfilter import Filter

class NameFilter(Filter):

    """
    A filter that changes the names of dictionary keys.

    'data' should be an iterable of dictionaries
    """

    def __call__(self, data):

        """
        The contents of this function are the least-important part of
        this demo. This is where you custom code will go, doing whatever
        it is you need with whatever config format and content you need.
        """

        for rec in data:

            # The config for this filter is a dictionary where the
            # key is the key name to replace, and the value is the new name.

            # update values in "data" dict
            for key, value in self.config.items():
                if key in rec:
                    rec[value] = rec[key]
                    del rec[key]

        return data

if __name__ == '__main__':

    import os # for dealing with the environment variable manually

    # set environment variable for demo purposes
    original_envvar = os.environ.get('FILTER_CONFIG_DIR', '')
    os.environ['FILTER_CONFIG_DIR'] = '/tmp'

    # Instantiate subclass. The only argument is the uid of the subject
    # of the filter. For example, if you need to store different rules
    # per user of your site, you might use the user's primary key here.
    # This allows storage of configs per filter *and* per user.
    name_filter = NameFilter('foo')

    # Set some filter items. This normally won't be a part of the flow.
    # It's here for demo purposes. In normal usage, the config would
    # already be set and probably rarely updated.
    name_filter.config = {
        'dog': 'canine',
        'cat': 'feline',
        'horse': 'equine',
    }

    name_filter.save_config(user='example')

    data = [{
        'cat': 'meow',
        'dog': 'woof',
        'horse': 'neigh',
        'foo': 'bar',
    }]

    print data # original
    print name_filter(data) # altered

    # Put it back like we found it, just to be good citizens.
    os.environ['FILTER_CONFIG_DIR'] = original_envvar

Sample output

[{'horse': 'neigh', 'foo': 'bar', 'dog': 'woof', 'cat': 'meow'}]
[{'equine': 'neigh', 'feline': 'meow', 'canine': 'woof', 'foo': 'bar'}]