Skip to main content

Put your data in a bag and get it back out again

Project description

PUT YOUR DATA IN A BAG

Pretty simple library for splatting stuff to disk and getting it back out with minimal fuss.

It's sort of a long term file based dictionary with enhanced range type filtering.

updated for python3

wait...

Yep - it's a nosql type, document oriented database wrapper on top of sqlite3.

features

  • Easy to use and quite efficient at accessing relatively large number of items (not talking big data here, but a couple of thousand items works well)
  • Requires no other libs, everything is python batteries included.
  • Built on top of sqlite3 so it's fast and stable (which is included in Python stdlib)
  • Easy to use - just create one and use it like a dictionary. Most dict methods supported. Also can add to it like a set by not specifying a key. One will be created on the fly.
  • Pretty well tested
  • Ideal for running on small vm instances. Doesn't require any other daemon to provide data access
  • Core code is about 400 lines - very easy to understand.
  • Automatically compresses data with bz2 in cases that benefit from it
  • offers versioned records if you so choose
  • You can always query the data with native sqlite3 libs from other languages if you need to. It's just strings in the database.
  • Since the underlying datafile is sqlite3, multiple processes can work with the same file (multiple read, write locks, etc)
  • Every object gets a ts object attached to it for convenience when it's saved. This is accessed via bag.when('key')

versioning

Simplified versioning is provided. Just create your DataBag like:::

>>> dbag = DataBag(versioned=True, fpath='/tmp/some.db')

and then you can do things like...

>>> dbag['blah'] = 'blip'
>>> dbag['blah'] = 'new blip'
>>> dbag['blah'] = 'newer blip'
>>> dbag.get('blah', version=-2)
u'blip'
>>> dbag.get('blah', version=-1)
u'new blip'
>>> dbag.get('blah')
u'newer blip'
>>> dbag['blah']
u'newer blip'

The default is to keep 10 versions but that can be set with the history parameter when initializing your bag.

If you don't specify an fpath argument, the database is only created in memory.
By specifying fpath, you specify the location of the file on the filesystem.

A bag.get(...) method works much like a dictionary's .get(...) but with an additional keyword argument of version that indicates how far back to go.

examples

>>> from databag import DataBag
>>> bag = DataBag() # will store sqlite db in memory
>>> bag['xyz'] = 'some string' # will save in the db
>>> s = bag['xyz'] # retrieves from db
>>> s
'some string'
>>> 'xyz' in bag # True
True
>>> bag['abc'] = {'x':22, 'y':{'a':'blah'}} # works
>>> bag['abc']
{u'y': {u'a': u'blah'}, u'x': 22}
>>> [k for k in bag]
['abc', 'xyz']
>>> bag.when('xyz')
datetime.datetime(2011, 12, 31, 2, 45, 47, 187621)
>>> del bag['xyz']
>>> 'xyz' in bag
False
>>> meh = DataBag(bag='other') # set name of storage table

DictBag example

>>> from databag import DictBag, Q
>>> d = DictBag()
>>> d.ensure_index(('name', 'age'))
>>> person1 = {'name':'joe', 'age':23}
>>> person2 = {'name':'sue', 'age':44}
>>> d.add(person1)
'fachVqv6RxsmCXAZgJMJ5p'
>>> d.add(person2)
'fpC7cAtx2ZQLadprQR7aa6'
>>> d.find(Q('age')>40).next()
(u'fpC7cAtx2ZQLadprQR7aa6', {u'age': 44, u'name': u'sue'})
>>> age = Q('age')
>>> [p for p in d.find(20 < age < 50) ]
[(u'fachVqv6RxsmCXAZgJMJ5p', {u'age': 23, u'name': u'joe'}),
    (u'fpC7cAtx2ZQLadprQR7aa6', {u'age': 44, u'name': u'sue'})]

There's also some syntactic sugar that lets you also use a Q object directly if the key name is a proper symbol name in python.

>>> [p for p in d.find(20 < Q.age < 50) ]
[(u'fachVqv6RxsmCXAZgJMJ5p', {u'age': 23, u'name': u'joe'}),
    (u'fpC7cAtx2ZQLadprQR7aa6', {u'age': 44, u'name': u'sue'})]
>>>

Mongo Style Queries

>>> d.find({'age':23})
>>> d.find({'age':{"$gt":20}} )

limitations

  • although a lot of the basic data types in python are supported for the values (lists, dictionaries, tuples, ints, strings)... datetime objects can be saved fine but they come out of the bag as an iso format string of the original datetime.
  • when saving a dictionary, the keys must be a string in the dictionary. If they are not, they will be when coming back from the bag
  • if using versioning, be sure to instantiate your DataBag object with versioning enabled and the same history size each time. Failure to do so will cause interesting things to happen, in particular, your databag will act unversioned and overwrite recent updates w/o cascading the historical change to records.

DataBag ORM

There are times an ORM makes life a little easier.

from databag.orm.model import set_db_path, Model, Field, IntField, Q

set_db_path(':memory:')

# define one
class SomeThing(Model):
    thingname = Field(str)
    num = IntField()

# make and save one
mything = SomeThing(thingname='oobleck', num=23).save()

# use one
print(mything.name)

# get it from db again
k = mything.key
samething = SomeThing.grab(k)

# or search for it with the same syntax as DictBag, but get obj instead
otherthing = SomeThing.find_one(num=23) # just one

# returns a generator, so list gets all of them
things = list(SomeThing.find(Q.num > 19))

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

databag-1.5.0.tar.gz (13.5 kB view details)

Uploaded Source

Built Distribution

databag-1.5.0-py3-none-any.whl (12.7 kB view details)

Uploaded Python 3

File details

Details for the file databag-1.5.0.tar.gz.

File metadata

  • Download URL: databag-1.5.0.tar.gz
  • Upload date:
  • Size: 13.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.41.0 CPython/3.7.5

File hashes

Hashes for databag-1.5.0.tar.gz
Algorithm Hash digest
SHA256 b2065b4678f973146a2ea06913c967ea22a4f9ea2b1d5673916c6de685481976
MD5 176ee6a337af24f991d0e4fe60847a6e
BLAKE2b-256 e2a620c0a5bb9fef41e6b6db0ef12e7f95da507047c3ff023a70587e7e605f51

See more details on using hashes here.

File details

Details for the file databag-1.5.0-py3-none-any.whl.

File metadata

  • Download URL: databag-1.5.0-py3-none-any.whl
  • Upload date:
  • Size: 12.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.41.0 CPython/3.7.5

File hashes

Hashes for databag-1.5.0-py3-none-any.whl
Algorithm Hash digest
SHA256 e800e52c41fc6fc971aa9eb83c7c973d5cc9878a6327bd87847217f57fc36bc5
MD5 8ec5b2485aff5e38a38a4d0482e83bfa
BLAKE2b-256 1608f333b6cbe2737feae05abe650a6e59d0e19dfe2021eda5c903b8632a9740

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page