django-zodb 0.2

Using Django and ZODB together

Django-ZODB

Django-ZODB is a simple ZODB database backend for Django Framework. It’s strongly inpired in repoze.zodbconn.

Installation

Django-ZODB requires the following packages:

• Django 1.2.3 or newer

• ZODB 3.10.0a2 or newer

If you need to store your data in a RDBMS system you will need to install the following packages too:

• RelStorage 1.5.0a1 or newer - ZODB storage system that store pickles in a relational database (in a non-relational format).

• MySQLdb 1.2.3 or newer - required to connect MySQL database.

• psycopg2 2.3.0-beta1 or newer - required to connect PostgreSQL database.

• cx_Oracle 5.0.3 or newer - required to connect Oracle database.

Note

Not tested with psycopg2 and cx_Oracle but we believe that everything will work as expected because we use RelStorage to connect to the database.

Install from sources:

$ python setup.py install

Or from PyPI (using easy_install):

$ easy_install -U django-zodb

Running tests

Install coverage if you need test coverage informations:

$ easy_install -U coverage

To run tests:

$ python manage.py test

Configuration

You need to configure your settings.py like this:

ZODB = {
    'default': [
        'mysql://user@passwd:localhost/relstorage_db?database_name=app',
        'postgresql://user@passwd:pg_test:5678/app1_db',
    ],
    'test':      [ 'mem://', 'mem://?database_name=catalog' ],
    'legacy_db': [ 'zconfig:///srv/www/zodb_media.conf' ],
    'user_dir':  [
        'zeo://main_db.intranet:7899?database_name=main',
        'zeo://catalog.intranet:7898?database_name=catalog'
    ],
    'old_app':   [
        'file:///var/lib/sitedata.db?blob_dir=/var/lib/blob_dir'
    ],
}

You can find a list of schemes and connection adapters in Connection Schemes.

Creating sample application

I strongly believe in “learn by doing” strategy, so, let’s create a sample Wiki application that stores their pages in ZODB.

I suggest the reading of the following tutorials and articles if you don’t know ZODB or the Traversal Algorithm (that we will use in our tutorial):

• ZODB Tutorial

• ZODB Programming Guide

• Traversal chapter at Repoze.BFG documentation.

Note

Repoze.BFG is now known as Pyramid.

Starting Django Project and Application

We will start a project called intranet with a Django application called wiki:

$ django-admin.py startproject intranet
$ cd intranet
intranet $ python manage.py startapp wiki

Now we need to modify our settings.py to include this new application and configure our database connections:

#!/usr/bin/env python
# settings.py

import os
ROOTDIR = os.path.dirname(os.path.realpath(__file__))

# No relational database...
DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = ':memory:'

# append the following lines:
ZODB = {
    'default': ['file://' + os.path.join(ROOTDIR, 'wiki_db.fs')],
}

# ... other Django configurations ...

MIDDLEWARE_CLASSES = (
    # ... other middlewares ...

    # If everything is ok (aka no exception raised) this middleware will
    # run a transaction.commit() on response.
    'django_zodb.middleware.TransactionMiddleware',
)

INSTALLED_APPS = (
    'django_zodb',  # enable manage.py zshell command
    'wiki',
)

Let’s create our model classes. We will need a “root” object that will store our objects (let’s name it Wiki) and a model to store the wiki pages itself (Page):

#!/usr/bin/env python
# wiki/models.py

import markdown  # http://pypi.python.org/pypi/Markdown
from django_zodb import models

# models.RootContainer - Define a 'root' object for database. This class
#                        defines __parent__ = __name__ = None
class Wiki(models.RootContainer):
    def pages(self):
        for pagename in sorted(self):
            yield self[pagename]

    def get_absolute_url(self):
        return "/wiki"

    # It's possible to change models.RootContainer settings using Meta
    # configurations. Here we will explicitly define the default values
    class Meta:
        database = 'default'  # Optional. Default: 'default'
        rootname = 'wiki'     # Optional. Default: RootClass.__name__.lower()

# models.Container - We will use Container to add support to subpages.
class Page(models.Model):
    def __init__(self, content="Empty Page."):
        super(Page, self).__init__()
        self.content = content

    def html(self):
        md = markdown.Markdown(safe_mode="escape",
                extensions=('codehilite', 'def_list', 'fenced_code'))
        return md.convert(self.content)

    @property
    def name(self):
        return self.__name__

    def get_absolute_url(self):
        return u"/".join((self.__parent__.get_absolute_url(), self.name))

We’ve a configured application and models. It’s time to map an URL to our view function:

#!/usr/bin/env python
# urls.py

# ... Django default URL configurations ...

urlpatterns = patterns('',
    # ... other URL mappings ...
    (r'^(?P<path>.*)/?$', 'wiki.views.page'),
)

And wiki/views.py:

#!/usr/bin/env python
# views.py

import re

from django.shortcuts import render_to_response
from django.http import HttpResponseRedirect
from django import forms

import transaction
from django_zodb import views
from django_zodb import models

from samples.wiki.models import Wiki, Page

wikiwords = re.compile(ur"\b([A-Z]\w+([A-Z]+\w+)+)")


class PageEditForm(forms.Form):
    content = forms.CharField(widget=forms.Textarea)


class WikiView(views.View):
    def __index__(self, request, context, root, subpath, traversed):
        return HttpResponseRedirect("FrontPage")

    def add(self, request, context, root, subpath, traversed):
        try:
            name = subpath[0]
        except IndexError:
            return HttpResponseRedirect("/")

        if request.method == "POST":
            form = PageEditForm(request.POST)
            if form.is_valid():
                page = Page(form.cleaned_data['content'])
                root[name] = page
                return HttpResponseRedirect(page.get_absolute_url())
        else:
            form = PageEditForm()

        page_data = {
            'name': name,
            'cancel_link': "javascript:history.go(-1)",
            'form': form,
        }
        return render_to_response("edit.html", page_data)
views.registry.register(model=Wiki, view=WikiView())


class PageView(views.View):
    def __index__(self, request, context, root, subpath, traversed):
        content = context.html()

        def check(match):
            word = match.group(1)
            if word in root:
                page = root[word]
                view_url = page.get_absolute_url()
                return '<a href="%s">%s</a>' % (view_url, word)
            else:
                add_url = models.model_path(root, "", "add", word)
                return '<a href="%s">%s</a>' % (add_url, word)

        content = wikiwords.sub(check, content)

        page_data = {
            'context': context,
            'content': content,
            'edit_link': context.get_absolute_url() + "/edit",
            'root': root,
        }
        return render_to_response("page.html", page_data)

    def edit(self, request, context, root, subpath, traversed):
        context_path = models.model_path(context)

        if request.method == "POST":
            form = PageEditForm(request.POST)
            if form.is_valid():
                context.content = form.cleaned_data['content']
                return HttpResponseRedirect(context_path)
        else:
            form = PageEditForm(initial={'content': context.content})

        page_data = {
            'name': context.name,
            'context': context,
            'cancel_link': context_path,
            'form': form,
        }
        return render_to_response("edit.html", page_data)
views.registry.register(model=Page, view=PageView())


def create_frontpage(root):
    frontpage = Page()
    root["FrontPage"] = frontpage
    return root

def page(request, path):
    root = models.get_root(Wiki, setup=create_frontpage)
    return views.get_response_or_404(request, root=root, path=path)

Traversal

From Repoze.BFG documentation:

Traversal is a context finding mechanism. It is the act of finding a context and a view name by walking over an object graph, starting from a root object, using a request object as a source of path information.

Django-ZODB implements the traversal algorithm in function django_zodb.views.traverse() that receive two arguments:

• root - an instance of Root model.

• path - a string with the path to be traversed.

And return a views.TraverseResult object with the following attributes:

• context - model object found by traversal.

• method_name - a method name if exists.

• subpath - aditional path arguments.

• traversed - path elements ‘traversed’.

• root - root object.

We’ve created some shortcuts functions to interpret these results:

• get_response(request, root, path) -> HttpResponse

• get_response_or_404(request, root, path) -> HttpResponse or Http404

These functions will traverse the model tree and call a registered view function that handle the context model object found. For example:

def handle_page_objects(request, result):
    # result is a TraverseResult object.
    # result.context is a Page object found by traverse
    return render_to_response(...)

# Register handle_page_objects function to handle Page objects:
views.registry.register(model=Page, view=handle_page_objects)

You can register a views.View() instance to handle model objects:

class PageView(views.View):
    # This is the 'default' handle (no method_name)
    def __index__(self, request, context, root, subpath, traversed):
        # ... context is a Page object ...
        return render_to_response(...)

    # called when method_name == "edit"
    def edit(self, request, context, root, subpath, traversed):
        # ... context is a Page object ...
        return render_to_response(...)

# Register a PageView *instance* to handle Page objects
views.registry.register(model=Page, view=PageView())

Connection Schemes

You can specify a ZODB connection using a URI. This URI is composed of the following arguments:

scheme://username:password@host:port/path?arg1=foo&arg2=bar#fraction

Depending on the chosen scheme some of these arguments are required and others optional.

Database and Connection settings

Arguments related to database connection settings. These arguments are optional and must be passed as query argument in URI (eg. ?database_name=db&...).

• database_name - str - database name used by ZODB.

• connection_cache_size - int - size (in bytes) of database cache.

• connection_pool_size - int - size of connection pool.

These arguments are passed to ZODB.DB.DB() constructor.

Memory Storage mem: (ZODB.MappingStorage)

Returns an in-memory storage. It’s basically a Python dict() object.

Valid URIs:

mem
mem:
mem://
mem?database_name=memory

Optional Arguments

• See Demo storage argument.

• See Blob storage arguments.

File Storage file: (ZODB.FileStorage)

Returns a database stored in a file. You need to specify an absolute path to the database file.

Valid URIs:

file:///tmp/Data.fs
file:///tmp/main.db?database_name=file

Invalid URIs:

file://subdir/Data.fs

Required Arguments

• path - str - absolute path to file where database will be stored.

Optional Arguments

• create - bool - create database file if does not exist. Default: create=True.

• read_only - bool - open storage only for reading. Default: read_only=False.

• quota - int - storage quota. Default: disabled (quota=None).

• See Demo storage argument.

• See Blob storage arguments.

zconfig: (ZODB.DB.DB)

Returns database (or databases) specified in ZCML configuration file.

Note

This scheme has some small differences with other schemes because it returns a DB object instead of a Storage. It’s a problem only in cases where you are creating the connection ‘by hand’ instead of use a higher level API.

URIs Examples:

zconfig:///my/app/zodb_config.zcml
zconfig:///my/app/zodb_config.zcml#main

Required Arguments

• path (str) - absolute path to file where database will be stored.

Optional Arguments (and default values)

• #fragment='' (str) - Get only an specific database. By default ('') get only the first database specified in configuration file. We don’t use a query argument (&arg=...) to specify database name to keep compatibility with repoze.zodbconn.

zeo: (ZEO.ClientStorage.ClientStorage)

Returns a connection to a ZEO server.

TODO

mysql: (RelStorage)

Returns a database stored in a MySQL relational server. This scheme uses RelStorage to establish connection with database server.

URIs Examples:

mysql://user:password@host:3306?compress=true#mysql_db_name
mysql:///tmp/mysql.sock#local_database
mysql://localhost#database

Arguments

• See Relational storage arguments.

postgresql (RelStorage)

Returns a database stored in a PostgreSQL relational server. This scheme uses RelStorage to establish connection with database server.

URIs Examples:

postgresql://user:password@host:5432#mysql_db_name

Arguments

• See Relational storage arguments.

Demo storage argument

• demostorage (bool) - Enable the ZODB’s demo storage wrapper.

Blob storage arguments

• blob_dir (str) - Directory where blob objects will be stored.

Relational storage arguments

Django-ZODB uses Relstorage to connect to RDBMS and we preserve the same arguments used by RelStorage. The only difference between RelStorage`s arguments and Django-ZODB arguments is that we use “_” (underline) instead of “-” (dash). For example: the RelStorage’s argument “shared-blob-dir” becomes “shared_blob_dir”.

Contributing

Hi, I’m accepting all kind of collaborations to this project. You can open issues in our issue tracker, send me a patch, an e-mail message with your questions, etc.

All kind of collaboration will be welcome.

TODO

• Review my ‘engrish’ in documentation

• Create a new Website

• Release 0.2 version (and announce)

• Test Relstorage connections with Oracle and PostgreSQL

• Create more manage.py commands for ZODB management

• Create a Django-ORM layer (wow!)

• Evaluate some fulltext-search, catalog, etc integrations

• Fix performance issues (?)

• … and fix (tons of) bugs! :D