This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Help us improve Python packaging - Donate today!
Project Description

Generational Sets

Generational sets (GSets) are designed to facilitate data synchronization between a server and clients.

Goals:

  • Make synchronization simple by sending all updates for a tree of sets at once.
  • Allow clients to be updated very quickly.
  • Reduce data-transfer volume by sending only changes.
  • Avoid conflict resolution.

Assumptions:

  • Disconnected data updates aren’t needed.

  • Clients mirror server data.

    This implies that the server data, or more specifically, the user’s view of the server data, aren’t too large to store on the client.

GSets track state by generation. A client presents a generation and is sent updates made since the presented generation. GSets can be grouped into trees with a shared generation. A client can present a single generation and be send updates for all of the sets making up a database.

This implementation of generational sets uses ZODB to store data on the server.

High-level usage pattern

  • Define a tree of sets representing the data in an application. This may be user specific.
  • Clients make updates via REST calls to a server. They don’t make local changes except in response to server updates.
  • Client requests include their data generation.
  • Most (JSON) responses to server calls have optional updates property that contains generational updates since the generation provided by the client. When the client gets updates, which include the new generation, it applies the updates to it’s internal data store.
  • For native apps, the server sends push notifications when there are updates for a user and, in response, the client polls for the updates. This allows updates to be extremely timely without constant polling.

Note that this package only provides the data structure implementation. Wrapping the data structure in a REST interface or sending notifications is up to applications.

API

Every object in a GSet must have an id. By default, this is provided by an id attribute, but you can configure a GSet to use another attribute or some other mechanism to get an id for an object.

When an object is added to a GSet, call the add method on the Gset with the object:

>>> from zc.generationalset import GSet
>>> things = GSet()
>>> athing = Thing(42)
>>> things.add(athing)

When an object is changed, call the changed method on the Gset with the object. If object is not present in the Gset, update will raise a KeyError:

>>> things.changed(athing)
>>> things.changed(Thing(43))
Traceback (most recent call last):
...
KeyError: 43

>>> things.generational_updates(0)
{'generation': 3, 'adds': [Thing(42)]}

To remove an object, call the remove method with the object:

>>> things.remove(athing)

To get updates to a set since a given generation, call generational_updates, passing a generation:

>>> things.generational_updates(0)
{'generation': 4, 'removals': [42]}

>>> things.add(Thing(1))
>>> things.generational_updates(0)
{'generation': 5, 'removals': [42], 'adds': [Thing(1)]}
>>> things.generational_updates(3)
{'generation': 5, 'adds': [Thing(1)]}

Note that generations start at 1.

The result of calling generational_updates is a dictionary with keys:

generation
The current generation of the set
adds
Objects added since the given generation.
removals
Ids of objects removed since the given generation.
contents

All of the object in the set.

contents are returned when there have been many removals since the given generation. A generational set only keeps track of a limited number (99 by default, but configurable) of removals. If a client is too out of date for the set to have relevant removals, it returns the entire contents, instead of returning adds and removals.

GSets support iteration, and querying length and containment. They don’t currently support set operations, like intersection and union. You can also retrieve an item from a GSet using it’s id:

>>> len(things)
1
>>> list(things)
[Thing(1)]
>>> Thing(1) in things
True
>>> things[1]
Thing(1)

Nested sets

To define nested sets:

  • Define a parent set:

    >>> parent = GSet(superset=True)
    

    Note the use of the superset parameter.

  • Define child sets, and add them to the parent:

    >>> messages = GSet("messages", parent)
    >>> parent.add(messages)
    

    When defining child sets, specify an id and the parent.

We haven’t tested more than one level of nesting.

When asking for generational updates on parent sets, the adds and contents contain the generational updates for subsets, with ids, but without subset generations:

>>> messages.add(Thing(42))
>>> parent.generational_updates(0)
{'generation': 3, 'adds': [{'id': 'messages', 'adds': [Thing(42)]}]}

Changes

0.3.0 (2014-08-28)

  • Added a changed method to make intent clear when simply recording changes.
  • Fixed: exceptions were raised when objects added to generational sets quacked a little like generational sets.

0.2.0 (2014-08-10)

  • Improved subset APIs:
    • No longer need to specify superset flag.
    • Can have set and non-set children.
    • A subset can be created without a parent and the parent will be set when it’s added to a containing set.

0.1.2 (2014-06-09)

Fixed: Internal data structures were misshandled when there were more
than the maximum number of removals.

(Release 0.1.1 was made in error.)

0.1.0 (2014-06-08)

Initial release

Release History

Release History

0.3.0

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
zc.generationalset-0.3.0.tar.gz (10.7 kB) Copy SHA256 Checksum SHA256 Source Aug 28, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting