Contains a variety of ordered structures, in particular a SkipList.
Project description
SkipList
This project contains a SkipList implementation in C++ with Python bindings.
A SkipList behaves as an always sorted list with, typically, O(log(n)) cost for insertion, look-up and removal. This makes it ideal for such operations as computing the rolling median of a large dataset.
See the full documentation on this project at ReadTheDocs.
A SkipList is implemented as a singly linked list of ordered nodes where each node participates in a subset of, sparser, linked lists. These additional 'sparse' linked lists provide rapid indexing and mutation of the underlying linked list. It is a probabilistic data structure using a random function to determine how many 'sparse' linked lists any particular node participates in. As such SkipList is an alternative to binary tree, Wikipedia has a introductory page on SkipLists.
An advantage claimed for SkipLists are that the insert and remove logic is simpler (however I do not subscribe to this). The drawbacks of a SkipList include its larger space requirements and its O(log(N)) lookup behaviour compared to other, more restricted and specialised, data structures that may have either faster runtime behaviour or lower space requirements or both.
This project contains a SkipList implementation in C++ with Python bindings with:
- No capacity restrictions apart from available memory.
- Works with any C++ type that has meaningful comparison operators.
- The C++ SkipList can be compiled as thread safe.
- The Python SkipList is thread safe.
- The SkipList has exhaustive internal integrity checks.
- Python SkipLists can be long/float/bytes/object types, the latter can have user defined comparison functions.
- With Python 3.8+ SkipLists can be combined with the multiprocessing.shared_memory module for concurrent operation on large arrays. For example concurrent rolling medians which speed up near linearly with the number of cores.
- The implementation is extensively performance tested in C++ and Python.
There are a some novel features to this implementation:
- A SkipList is a probabilistic data structure but we have deterministic tests that work for any (sane) random number generator. See Testing a Probabilistic Structure
- This SkipList can dynamically generate visualisations of its current internal state. See Visualising a Skip List
Credits
Originally written by Paul Ross with credits to: Wilfred Hughes (AHL), Luke Sewell (AHL) and Terry Tsantagoeds (AHL).
Installation
C++
This SkipList requires:
- A C++11 compiler.
-I<skiplist>/src/cpp
as an include path.<skiplist>/src/cpp/SkipList.cpp
to be compiled/linked.- The macro
SKIPLIST_THREAD_SUPPORT
set if you want a thread safe SkipList using C++ mutexes.
Python
This SkipList version supports Python 3.7, 3.8, 3.9, 3.10, 3.11 (and, probably, some earlier Python 3 versions).
From PyPi
$ pip install orderedstructs
From source
$ git clone https://github.com/paulross/skiplist.git
$ cd <skiplist>
$ python setup.py install
Testing
This SkipList has extensive tests for correctness and performance.
C++
To run all the C++ functional and performance tests:
$ cd <skiplist>/src/cpp
$ make release
$ ./SkipList_R.exe
To run the C++ functional tests with agressive internal integrity checks:
$ cd <skiplist>/src/cpp
$ make debug
$ ./SkipList_D.exe
To run all the C++ functional and performance tests for a thread safe SkipList:
$ cd <skiplist>/src/cpp
$ make release CXXFLAGS=-DSKIPLIST_THREAD_SUPPORT
$ ./SkipList_R.exe
Python
Testing requires pytest
and hypothesis
:
To run all the C++ functional and performance tests:
$ cd <skiplist>
$ py.test -vs tests/
Examples
Here are some examples of using a SkipList in your code:
C++
#include "SkipList.h"
// Declare with any type that has sane comparison.
OrderedStructs::SkipList::HeadNode<double> sl;
sl.insert(42.0);
sl.insert(21.0);
sl.insert(84.0);
sl.has(42.0) // true
sl.size() // 3
sl.at(1) // 42.0, throws OrderedStructs::SkipList::IndexError if index out of range
sl.remove(21.0); // throws OrderedStructs::SkipList::ValueError if value not present
sl.size() // 2
sl.at(1) // 84.0
The C++ SkipList is thread safe when compiled with the macro SKIPLIST_THREAD_SUPPORT
, then a SkipList can then be shared across threads:
#include <thread>
#include <vector>
#include "SkipList.h"
void do_something(OrderedStructs::SkipList::HeadNode<double> *pSkipList) {
// Insert/remove items into *pSkipList
// Read items inserted by other threads.
}
OrderedStructs::SkipList::HeadNode<double> sl;
std::vector<std::thread> threads;
for (size_t i = 0; i < thread_count; ++i) {
threads.push_back(std::thread(do_something, &sl));
}
for (auto &t: threads) {
t.join();
}
// The SkipList now contains the totality of the thread actions.
Python
An example of using a SkipList of always ordered floats:
import orderedstructs
# Declare with a type. Supported types are long/float/bytes/object.
sl = orderedstructs.SkipList(float)
sl.insert(42.0)
sl.insert(21.0)
sl.insert(84.0)
sl.has(42.0) # True
sl.size() # 3
sl.at(1) # 42.0
sl.has(42.0) # True
sl.size() # 3
sl.at(1) # 42.0, raises IndexError if index out of range
sl.remove(21.0); # raises ValueError if value not present
sl.size() # 2
sl.at(1) # 84.0
The Python SkipList can be used with user defined objects with a user defined sort order. In this example the last name of the person takes precedence over the first name:
import functools
@functools.total_ordering
class Person:
"""Simple example of ordering based on last name/first name."""
def __init__(self, first_name, last_name):
self.first_name = first_name
self.last_name = last_name
def __eq__(self, other):
try:
return self.last_name == other.last_name and self.first_name == other.first_name
except AttributeError:
return NotImplemented
def __lt__(self, other):
try:
return self.last_name < other.last_name or self.first_name < other.first_name
except AttributeError:
return NotImplemented
def __str__(self):
return '{}, {}'.format(self.last_name, self.first_name)
import orderedstructs
sl = orderedstructs.SkipList(object)
sl.insert(Person('Peter', 'Pan'))
sl.insert(Person('Alan', 'Pan'))
assert sl.size() == 2
assert str(sl.at(0)) == 'Pan, Alan'
assert str(sl.at(1)) == 'Pan, Peter'
The Python SkipList is thread safe when using any acceptable Python type even if that type has user defined comparison methods. This uses Pythons mutex machinery which is independent of C++ mutexes.
History
0.3.14 (2024-08-02)
- Support for Python 3.8, 3.9, 3.10, 3.11, 3.12, 3.13.
- Drop support for Python 3.7.
0.3.13 (2023-09-05)
- Documentation improvements.
0.3.12 (2023-08-29)
- Minor fixes to the documentation and examples.
0.3.11 (2023-08-28)
- Minor fixes missed in 0.3.10 release.
0.3.10 (2023-08-28)
- Minor fixes missed in 0.3.9 release.
0.3.9 (2023-08-28)
- Minor fixes missed in 0.3.8 release.
0.3.8 (2023-08-28)
- Add cmake build.
- Support for Python 3.7, 3.8, 3.9, 3.10, 3.11.
- Remove mention of Python 2.
0.3.7 (2021-12-18)
- Fix build on GCC (Issue #8).
0.3.6 (2021-12-18)
- Add documentation on NaN in rolling median.
- Add plots using shared_memory.
- Add Python 3.10 support.
0.3.5 (2021-05-02)
- Fix uncaught exception when trying to remove a NaN.
0.3.4 (2021-04-28)
- Improve documentation mainly around multiprocessing.shared_memory and tests.
0.3.3 (2021-03-25)
- Add Python benchmarks, fix some build issues.
0.3.2 (2021-03-18)
- Fix lambda issues with Python 3.8, 3.9.
0.3.1 (2021-03-17)
- Support Python 3.7, 3.8, 3.9.
0.3.0 (2017-08-18)
- Public release.
- Allows storing of
PyObject*
and rich comparison.
0.2.0
Python module now named orderedstructs
.
0.1.0
Initial release.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for orderedstructs-0.3.14-cp313-cp313-macosx_10_13_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ae565efd04d8436c0e5c0e0ddb9c38b3ae7cdf79640546f18d77091844d63469 |
|
MD5 | 6860aad790091ed1aba675f8c78a11c2 |
|
BLAKE2b-256 | cd1a8ef002b36fc5e301c43a8ac10019054bc1f1539dd6df4b2864798f532a09 |
Hashes for orderedstructs-0.3.14-cp312-cp312-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a51173612b1019027e1c37911aef9dc7450ff7f0bd9e218bde4a100efa36984 |
|
MD5 | 13501e5e76f11cf43adef492c61d7e7b |
|
BLAKE2b-256 | 630094cb9ef0e214c82f78ac8bb168571851cf05650c5ae22ac602c1d795202f |
Hashes for orderedstructs-0.3.14-cp311-cp311-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e160d949f4141a5b44e1f9c8957a69eb38d7b283f0d3f3e90a84ac813f9f2f25 |
|
MD5 | 03a72d4b78f551af70898a653181c41f |
|
BLAKE2b-256 | 51ae60bcf61f735d0d97ace9a35955b224cd0130979c31cbe9652dd7852b75a2 |
Hashes for orderedstructs-0.3.14-cp310-cp310-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd8ac78446af79963552b305388faa6fe9be5ffc1334551085d00941a4b854bb |
|
MD5 | 1a7e5d21e74cadf3ea7cae393945297f |
|
BLAKE2b-256 | 7f302c9616c5fd282509927397188d55cd0dc565005e4f4d158ae4fb56dc50e2 |
Hashes for orderedstructs-0.3.14-cp39-cp39-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 411e2a9746977b4ff2ac608d2f67b86d918454f345567f3fda9b100ea3889571 |
|
MD5 | 46c753b16b7f230adcc35fb9cadff8c3 |
|
BLAKE2b-256 | 1a99fcf320d80b56790eea9c3947081df97ee336635c629b64c93721758a4f35 |
Hashes for orderedstructs-0.3.14-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c214f25bc57afc5c1d40f4363884d72a4c5202851ded7b7dd6b40a7658a256ea |
|
MD5 | 3ed8eb94d64341c8fbd47bd733df0f70 |
|
BLAKE2b-256 | 965dd30ea1d12c4459e7eeaf77b5d19f2887551bd8c5f12d4794b92a91b4ef71 |