Skip to main content

JSON Object storage backed by Git & Elastic Search

Project description

Adventures in an declarative object-y thing backed by Git and using Elasticsearch as a query backend.

Note

Here be massive dragons.

Continuous Integration Code Coverage Elastic-Git Documentation

Usage

from elasticgit import EG
from elasticgit.models import Model, IntegerField, TextField


workspace = EG.workspace('/Users/sdehaan/Desktop/test-repo/')
workspace.setup('Simon de Haan', 'simon@praekeltfoundation.org')


"""
# The model looks like this

class Person(Model):
    age = IntegerField('The Age')
    name = TextField('The Name')
"""
from elasticgit.tests.base import TestPerson as Person

person1 = Person({'age': 10, 'name': 'Foo'})
workspace.save(person1, 'Saving Person 1')

person2 = Person({'age': 20, 'name': 'Bar'})
workspace.save(person2, 'Saving Person 2')

person3 = Person({'age': 30, 'name': 'Baz'})
workspace.save(person3, 'Saving Person 3')

Data is now persisted in a git repository and is queryable via elasticsearch:

>>> from elasticgit import EG
>>> from elasticgit.tests.base import TestPerson as Person
>>> workspace = EG.workspace('/Users/sdehaan/Desktop/test-repo/')
>>> for person in workspace.S(Person).filter(age__gte=20):
...     print person.name, person.age
...
Bar 20
Baz 30

Check the examples/ directory for some more code samples.

$ python -m examples.basic_usage
e6cb25f00870472fa5223d76dc361667 Baz 30
2bd470372243411c9abd8fdcb969dcf5 Bar 20

Schema Management

We’ve followed the example of Apache Avro when it comes to schema evolution. Avro compatible schemas can be generated from the command line.

Model definitions can be rebuilt from Avro JSON schema files.

A sample model file:

class TestFallbackPerson(Model):
    age = IntegerField('The Age')
    name = TextField('The name', fallbacks=[
        SingleFieldFallback('nick'),
        SingleFieldFallback('obsolete'),
    ])
    nick = TextField('The nickname', required=False)
    obsolete = TextField('Some obsolete field', required=False)

Generating the Avro spec file

$ python -m elasticgit.tools dump-schema \
>   elasticgit.tests.base.TestFallbackPerson > avro.json
$ python -m elasticgit.tools load-schema avro.json > models.py

The generated model file:

# NOTE:
#
#   This is an automatically generated Elasticgit Model definition
#   from an Avro schema. Do not manually edit this file unless you
#   absolutely know what you are doing.
#
# timestamp: 2014-10-14T18:51:23.916194
# namespace: elasticgit.tests.base
# type: record
# name: TestFallbackPerson
#

from elasticgit import models

class TestFallbackPerson(models.Model):

    name = models.TextField(u"""The name""", fallbacks=[models.SingleFieldFallback('nick'),models.SingleFieldFallback('obsolete'),])
    age = models.IntegerField(u"""The Age""")
    obsolete = models.TextField(u"""Some obsolete field""")
    _version = models.ModelVersionField(u"""Model Version Identifier""")
    nick = models.TextField(u"""The nickname""")
    uuid = models.TextField(u"""Unique Identifier""")

We’re using ConfModel’s fallbacks feature and encode this in Avro’s Schema as aliases. This allows you to fall back to older names for fields:

>>> TestFallbackPerson({'obsolete': 'oldest name', 'age': 10}).name
'oldest name'
>>> TestFallbackPerson({'nick': 'older name', 'age': 10}).name
'older name'
>>> TestFallbackPerson({'name': 'current name', 'age': 10}).name
'current name'

Project details


Release history Release notifications

This version
History Node

1.5.0

History Node

1.4.4

History Node

1.4.3

History Node

1.4.2

History Node

1.4.1

History Node

1.4.0

History Node

1.3.1

History Node

1.3.0

History Node

1.2.0

History Node

1.1.0

History Node

1.0.2

History Node

1.0.1

History Node

1.0.0

History Node

0.3.7

History Node

0.3.6

History Node

0.3.5

History Node

0.3.4

History Node

0.3.3

History Node

0.3.2

History Node

0.3.1

History Node

0.3.0

History Node

0.2.12

History Node

0.2.11

History Node

0.2.10

History Node

0.2.9

History Node

0.2.8

History Node

0.2.7

History Node

0.2.6

History Node

0.2.5

History Node

0.2.4

History Node

0.2.3

History Node

0.2.2

History Node

0.2.0

History Node

0.1.5

History Node

0.1.4

History Node

0.1.2

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
elastic-git-1.5.0.tar.gz (39.9 kB) Copy SHA256 hash SHA256 Source None Mar 7, 2016

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page