Skip to main content

A Python package for reference counting and interop with native pointers

Project description

refcount

license status Documentation Status master: Build status - master testing: Build status - devel

Reference counted native handles

This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boils down to "simply" maintaining a set of counters, it is deceptively complicated to do so properly and not end up with memory leaks or crashes. This package offers structured options for reliably managing external native resources. Surprisingly I could not locate an existing package doing just what I needed. Other use cases requiring reference counting, aside from native library resources, may benefit from reusing and extending classes in refcount.

refcount from version 0.7 includes classes using cffi. Other low-level interoperability mechanisms may well be added in the future.

License

MIT (see License.txt)

Documentation

Hosted at pyrefcount.readthedocs.io

Source code

The code repository is on GitHub.

Installation

pip install refcount

From source:

pip install -r requirements.txt
python setup.py install

Sample use

The following example is based on one of the unit tests.

Say we have a C++ library with objects and a C API:

#define TEST_DOG_PTR  testnative::dog*
#define TEST_OWNER_PTR  testnative::owner*
#define TEST_COUNTED_PTR  testnative::reference_counter*

testnative::dog* create_dog();
testnative::owner* create_owner(testnative::dog* d);
void say_walk(testnative::owner* owner);
void release(testnative::reference_counter* obj);
// etc.

From the outside of the library the API is exported with opaque pointers void* (C structs pointers and native C99 types could be handled too).

void* create_dog();
void* create_owner(void* d);
void say_walk(void* owner);
void release(void* obj);
// etc.

Starting with the end in mind, from Python we want an API hiding the low level details close to the C API, in particular avoiding managing native memory via release C API calls, piggybacking the python GC instead.

dog = Dog()
owner = DogOwner(dog)
owner.say_walk()
print(dog.position)
dog = None # the "native dog" is still alive though, as the owner incremented the ref count
owner = None

This is doable with refcount and the cffi package. One possible design is:

ut_ffi = cffi.FFI()

ut_ffi.cdef('extern void* create_dog();')
ut_ffi.cdef('extern void* create_owner( void* d);')
ut_ffi.cdef('extern void say_walk( void* owner);')
ut_ffi.cdef('extern void release( void* obj);')
# etc.

ut_dll = ut_ffi.dlopen('c:/path/to/test_native_library.dll', 1) # Lazy loading

class CustomCffiNativeHandle(CffiNativeHandle):
    def __init__(self, pointer, prior_ref_count = 0):
        super(CustomCffiNativeHandle, self).__init__(pointer, type_id='', prior_ref_count = prior_ref_count)

    def _release_handle(self) -> bool:
        ut_dll.release(self.get_handle())
        return True

class Dog(CustomCffiNativeHandle):
    def __init__(self, pointer = None):
        if pointer is None:
            pointer = ut_dll.create_dog()
        super(Dog, self).__init__(pointer)
    # etc.

class DogOwner(CustomCffiNativeHandle):

    def __init__(self, dog):
        super(DogOwner, self).__init__(None)
        self._set_handle(ut_dll.create_owner(dog.get_handle()))
        self.dog = dog
        self.dog.add_ref() # Do note this important reference increment

    def say_walk(self):
        ut_dll.say_walk(self.get_handle())

    def _release_handle(self) -> bool:
        super(DogOwner, self)._release_handle()
        # super(DogOwner, self)._release_handle()
        self.dog.release()
        return True

Related work

Ancestry, acknowledgements

This python package refcount actually spawned from prior work for interoperability between C++, R and .NET (R.NET)

refcount features using cffi were also significantly informed by Kevin Plastow's work while he was at the Australian Bureau of Meteorology; this contribution is gratefully acknowledged.

In you have native interop needs you may also want to look at:

Other python packages

refcount was created in part because no existing prior (Python) work could quite fit the need. There are however packages that may better address your particular need:

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for refcount, version 0.9.1
Filename, size File type Python version Upload date Hashes
Filename, size refcount-0.9.1-py2.py3-none-any.whl (14.0 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size refcount-0.9.1.zip (24.8 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page