Skip to main content

DSE - Delayed SQL Executor

Project description

DSE - Delayed SQL Executor
==========================


Version : 0.1.0
Author : Thomas Weholt <thomas@weholt.org>
License : GPL v3.0
Status : Beta

Description:
------------

dse is a simple and crude way of not executing sql queries in sequence, but caching values until a given max value has been
met and then execute them using the executemany-method. The result can be huge speed gains.

Only tested on SQLite3 but intended for use in django as well.

Example usage:
--------------

import sqlite3 # for testing purposes

from dse import DelayedSqlExecutor

conn = sqlite3.connect(':memory:')
cursor = conn.cursor()
cursor.execute('create table filedata (id INTEGER PRIMARY KEY, filepath TEXT, filename TEXT, filesize INTEGER)')
d = DelayedSqlExecutor(cursor, paramtoken= '?') # using the ? paramtoken here for sqlite3. Leave it blank and it`ll use %s as support by Django etc.
d.addObject('filedata', ('id', 'filepath', 'filename', 'filesize'))

for i in range(0, 999):
# adding some dummy data. Notice the absence of the id-field. This will trigger inserts.
#Adding the id-field would trigger an update for data not yet in the db
d.addItem('filedata', {'filepath': '/tmp/', 'filename': 'test%s.txt' % i, 'filesize': i})
# No SQL has been executed yet, the default limit is 1000 items

# Adding another item will trigger the execution of SQLs and reset the d-instance
d.addItem('filedata', {'filepath': '/tmp/', 'filename': 'test%s.txt' % i, 'filesize': i})

# Adding some records to update
d.addItem('filedata', {'id': 1, 'filepath': '/tmp/', 'filename': 'testmore%s.txt' % i, 'filesize': 100})

# calling close will execute any remaining SQLs
d.close()

# you might be required to call commit on the cursor to commit the data. Depends on how you set up the cursor/connection.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dse, version 0.1.0
Filename, size File type Python version Upload date Hashes
Filename, size dse-0.1.0.zip (5.7 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page