humpack 0.3.4

Human Readable Object Serialization and more

This package contains several high level utilities for python development, including:

• Packing: Human Readable Serialization of Python Objects

• Transactions: Grouping multiple python statements into an atomic operation that can be committed or aborted

• Containers: dropin replacements for several common python types (including dict, list, set, etc.)

• Cryptography: simple utilities for common cryptography functionality (very high-level on top of standard python libraries)

Install

---

Everything is tested with Python 3.7 on Ubuntu 18.04, but there is no reason it shouldn’t also work for Windows.

You can install this package through pip:

pip install humpack

Alternatively, you can clone this repo and install the local version for development:

git clone https://github.com/felixludos/HumPack

pip install -e ./HumPack

Quick Start

---

Containers

---

The provided containers: tdict, tlist, and tset serve as drop-in replacements for pythons dict, list, and set types that are Transactionable and Packable (more info below). Furthermore, all keys in adict that are valid attribute names, can be treated as attributes.

A few examples:

from humpack import adict, tdict, tlist, tset

from humpack import json_pack, json_unpack

from humpack import AbortTransaction



d = adict({'apple':1, 'orange':10, 'pear': 3})

d.apple += 10

d.update({'non-det banana':tset({2,3,7}), 'orange': None})

del d.pear

assert d.apple == 11 and 2 in d['non-det banana'] and 'pear' not in d

options = tlist(d.keys())

options.sort()

first = options[0]

assert first == 'apple'

d.order = options



json_d = json_pack(d)

assert isinstance(json_d, str)



d.begin() # starts a transaction (tracking all changes)

assert options.in_transaction()



d['non-det banana'].discard(7)

d.cherry = 4.2

assert 'cherry' in d and len(d['non-det banana']) == 2

d['order'].extend(['grape', 'lemon', 'apricot'])

assert 'grape' in options

del d.order[0]

del d['orange']

d.order.sort()

assert options[0] == 'apricot'



d.abort()

assert 'cherry' not in d and 7 in d['non-det banana']

assert 'grape' not in options



with d:

    assert d['non-det banana'].in_transaction()

    d.clear()

    assert len(d) == 0

    d.melon = 100j

    assert 'melon' in d and d['melon'].real == 0

    raise AbortTransaction



assert 'melon' not in d



assert json_pack(d) == json_d

assert sum(d['non-det banana']) == sum(json_unpack(json_d)['non-det banana'])



with d:

    assert 'cherry' not in d

    d.cherry = 5

    # automatically commits transaction on exiting the context if no exception is thrown



assert 'cherry' in d

When starting with data in standard python, it can be converted to using the “t” series counter parts using containerify.

from humpack import containerify

from humpack import AbortTransaction



x = {'one': 1, 1:2, None: ['hello', 123j, {1,3,4,5}]}



d = containerify(x)



assert len(x) == len(d)

assert len(x[None]) == len(d[None])

assert x['one'] == d.one

with d:

    assert d[None][-1].in_transaction()

    del d.one

    d.two = 2

    d[None][-1].add(1000)

    assert d['two'] == 2 and 'one' not in d and sum(d[None][-1]) > 1000

    raise AbortTransaction

assert 1000 not in d[None][-1] and 'one' in d and 'two' not in d

Finally, there are a few useful containers which don’t have explicit types in standard python are also provided including heaps and stacks: theap and tstack.

Packing (serialization)

---

To serializing an object into a human-readable, json compliant format, this library implements packing and unpacking. When an object is packed, it can still be read (and manipulated, although that not recommended), converted to a valid json string, or encrypted/decrypted (see the Security section below). However for an obejct to be packable it and all of it’s submembers (recursively) must either be primitives (int, float, str, bool, None) or registered as a Packable, which can be done

Packing and unpacking is primarily done using the pack and unpack functions, however, several higher level functions are provided to combine packing and unpacking with other common features in object serialization. For custom classes to be Packable, they must implement three methods: __pack__, __create__, __unpack__ (for more info see the documentation for Packable). When implementing these methods, all members of the objects that should be packed/unpacked, must use pack_member and unpack_member to avoid reference loops.

from humpack import pack, unpack



x = {'one': 1, 1:2, None: ['hello', 123j, {1,3,4,5}]}



p = pack(x) # several standard python types are already packable

assert isinstance(p, dict)

deepcopy_x = unpack(p)

assert repr(x) == repr(deepcopy_x)



from humpack import json_pack, json_unpack # Convert to/from json string



j = json_pack(x)

assert isinstance(j, str)

deepcopy_x = json_unpack(j)

assert repr(x) == repr(deepcopy_x)





from humpack import save_pack, load_pack # Save/load packed object to disk as json file

import os, tempfile



fd, path = tempfile.mkstemp()

try:

    with open(path, 'w') as tmp:

        save_pack(x, tmp)

    with open(path, 'r') as tmp:

        deepcopy_x = load_pack(tmp)

finally:

    os.remove(path)

assert repr(x) == repr(deepcopy_x)

For examples of how to any types can registered to be Packable or objects can be wrapped in Packable wrappers, see the humpack/common.py and humpack/wrappers.py scripts.

Transactions

---

For examples of how Transactionable objects behave see the “Containers” section above.

To enable transactions for a class, it must be a subclass of Transactionable and implement the four required functions: begin, in_transaction, commit, and abort. Assuming these functions are implemented as specified (see documentation), you can manipulate instances of these classes in a transaction and then roll back all the changes by aborting the transaction.

One important thing to note with subclassing Transactionable: any members of instances of Transactionable subclasses should be checked for if they are also Transactionable, and if so, they the call should be delegated. In the example below, Account has to take into account that its attribute user could be Transactionable.

from humpack import Transactionable



class Account(Transactionable):

    def __init__(self, user, balance=0.):

        super().__init__()

        self._in_transaction = False

        self._shadow_user = None



        self.user = user

        self.balance = balance



    def change(self, delta):



        if self.balance + delta < 0.:

            raise ValueError

        self.balance += delta



    def begin(self):

        # FIRST: begin the transaction in self

        self._shadow_user = self.user.copy(), self.balance # Assuming `user` can be shallow copied with `copy()`

        self._in_transaction = True



        # THEN: begin transactions in any members that are Transactionable

        if isinstance(self.user, Transactionable):

            self.user.begin()



        # To be extra safe, you could also check `self.balance`, but we'll assume it's always a primitive (eg. float)



    def in_transaction(self):

        return self._in_transaction



    def commit(self):

        # FIRST: commit the transaction in self

        self._in_transaction = False

        self._shadow_user = None



        # THEN: commit transactions in any members that are Transactionable

        if isinstance(self.user, Transactionable):

            self.user.commit()



    def abort(self):

        # FIRST: abort the transaction in self

        if self.in_transaction(): # Note that this call only has an effect if self was in a transaction.

            self.user, self.balance = self._shadow_user



        self._in_transaction = False

        self._shadow_user = None



        # THEN: abort transactions in any members that are Transactionable

        if isinstance(self.user, Transactionable):

            self.user.abort()

Optionally, for a more pythonic implementation, you can use try/except statements instead of type checking with isinstance.

Security

---

There are a few high-level cryptography routines. Nothing special, just meant to make integration in larger projects simple and smooth.

TODO

---

Features that could be added/improved:

• Enable simple conversion from containers to standard python (eg. decontainerify)

• Add security functions to encrypt/decrypt files and directories (collecting/zipping contents in a tar)

• Add Transactionable/Packable replacements for more standard python types (especially tuples)

• Possibly add 1-2 tutorials

• Write more comprehensive unit tests and report test coverage

• Allow packing bound methods of Packable types

• Add option to save class attributes

Contributions and suggestions are always welcome.