Skip to main content

DSE - Delayed SQL Executor

Project description

DSE - Delayed SQL Executor

Version : 0.9.0
Author : Thomas Weholt <>
License : GPL v3.0
Status : Beta
Url :
Docs at


DSE is concept of caching SQL-statements, both inserts and updates, and executing them when a specified
number of statements has been prepared. This is done using DB API cursor.executemany(list of cached statements)
and this is way faster than executing SQL-statements in sequence.

DSE also is a way to solve a recurring problem when using the Django ORM; how to insert or update a bunch of
records without the huge performance hit of using the ORM to do it, for instance when you want to
scan a filesystem and add or update a record for each file found.

It has been designed to be used outside Django as well, but the main focus is good Django integration.


pip install dse


hg clone

==Example usage==

You got a model like:

class Person(models.Model):
name = models.CharField(max_length = 30)
age = models.IntegerField()
sex = models.CharField(max_length = 1, choices = (('M', 'Male'), ('F', 'Female')))

Using dse, you`ll be doing something like this:

import dse
dse.patch_models() # This will monkey patch all your models and expose dse for all models:

with Person.dse as d:
for name, age, sex in (('Thomas', 36, 'M'), ,('Joe', 40, 'M'), ('Jane', 28, 'F')):
d.add_item(dict(name = name, age = age, sex = sex))

Nothing will be inserted into the database before the loop is done ( or you insert 10000 items ).
Then the items will be inserted using executemany, using plain SQL - no ORM in sight.

Version 0.9.1 also introduced support for singletons ( NB! very experimental, no locking or thread support yet! ):

import dse.singleton

p1 = dse.singleton.Person()
p2 = dse.singleton.Person()
print p1 is p2 # should print True
p1.add_item(dict(name = 'Joe'))
print Person.objects.all().count() # should print 1

Singletons makes it possible to cache entries across pieces of code and cache even more data, hitting the db less.

==Release notes==

0.9.1 : - Refactored code even more, added usage.rst, singleton support in the singleton-package and some performance tests. Models not monkey patched be default anymore, must call dse.patch_models().

0.9.0 : - Refactored code and cleaned up tests folder. Focus on getting singleton support in before 1.0.0. And more tests.

0.8.2 : - added 'pysqlite2' to _DBMAP. Thanks to David Marble for 0.8.1 and 0.8.2.

0.8.1 : - attempt to fix quoting problems with fields on postgresql.

0.8.0 : - fixed crash when more than one database connection has been configured. No ModelFactory will be triggered.

0.7.0 : - don`t remember.

0.6.0 : - added support for the with-statement.
- added an ModelDelayedExecutor-instance to each model, so you can do Model.dse.add_item
instead of dse.ModelFactory.Model.add_item.
- renamed dse.modelfactory to dse.ModelFactory to be more style-compliant.

0.5.1 : just some notes on transaction handling.

0.5.0 :
- added modelfactory. Upon first import a modelfactory will be created in the DSE module. It`s basically just a
helper-class containing ModelDelayedExecutor-instances for all models in all apps found in INSTALLED_APPS in
- to change the default item limit before automatic execution of cached SQL statements to 10000 instead of the default 1000::

import dse
dse.ITEM_LIMIT = 10000

0.4.0 :
- fixed serious bug when using mass updates. Using cursor.executemany is only possible when values
for all columns are specified. If only values for a subset of the columns is specified that will be
executed as a seperate SQL-call. NOTE! Using dex.get_items() or Djangos Model.objects.values() will give you
all the fields.
- code clean-up.
- added custom exceptions; UpdateManyException, UpdateOneException and InsertManyException.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dse, version 0.9.1
Filename, size File type Python version Upload date Hashes
Filename, size dse-0.9.1.linux-x86_64.tar.gz (13.6 kB) File type Dumb Binary Python version any Upload date Hashes View hashes
Filename, size dse-0.9.1.tar.gz (7.8 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page