Skip to main content

a persistent sequence generator for elasticsearch

Project description

An integer id sequence generator using elasticsearch to create bulks of ids. Elasticsearch can not create integer ids for newly created documents. This package helps to be able to create unique integer ids in a distributed system.

Features

  • makes sure multiple processes are able to create unique ids

  • optimized by requesting multiple ids with a single Elasticsearch request

Elasticsearch Setup

The sequences must be assigned to an elasticsearch client instance:

>>> from elasticsearch import Elasticsearch
>>> es_client = Elasticsearch(['localhost:%s' % crate_port])

To globally assign the client to all sequences in the application the client can be assigned to the class property ES:

>>> from lovely.essequence import Sequence
>>> Sequence.ES = es_client

Usage

There is a Sequence class:

>>> from lovely.essequence import Sequence

It must be instantiated with the name of a sequence:

>>> s1 = Sequence('s1')

Now the iids can be requested:

>>> s1.next()
1
>>> s1.next()
2

Multiple sequence instances for the same sequence name use the same bulk:

>>> s2 = Sequence('s1', bulk_size=100)
>>> s2.next()
3

After the bulk size has been consumed the request of the next bulk is transparently handled inside the next method:: Consume all the cached ids from s1:

>>> for i in range(10): s1.next()
4
5
6
7
8
9
10
11
12
13

Multiple independed sequences can be used:

>>> other = Sequence('other')
>>> other.next()
1

Transforming the provided iid

A transformation function can be provided to transform the provided integer value into something else when calling next:

>>> asstring = Sequence('other', transform=str)
>>> isinstance(asstring.next(), str)
True

>>> def transformer(iid):
...     return {'iid': iid}
>>> Sequence('other', transform=transformer).next()
{'iid': 3}

Resetting a Sequence

For testing purposes it is possible to reset a sequence.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lovely.essequence-0.1.0.tar.gz (3.7 kB view details)

Uploaded Source

File details

Details for the file lovely.essequence-0.1.0.tar.gz.

File metadata

File hashes

Hashes for lovely.essequence-0.1.0.tar.gz
Algorithm Hash digest
SHA256 46ef85f3c8d03cc6374ff48d4307d85c3d72cc6be78fd7ef4efa1fb399ef247d
MD5 3c8c855154ed92ff1d97a9cd049a2c01
BLAKE2b-256 e80f3f78bf140b3f8d5b2f299b7454540bd3d76b2c9dcde4884ebaffc3343ae7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page