Skip to main content

Pond is a high performance object-pooling library for Python.

Project description

Pond

Pond is a high performance object-pooling library for Python, it has a smaller memory usage and a higher borrow hit rate. For more details, see our user's guide or my blog(https://qin.news).

English | 中文版

Pond is probably the first publicly available object pooling library in Python with complete test cases, over 90% coverage, complete code comments, and good documentation.

Inspired by Apache Commons Pool, Netty Recycler, HikariCP, Caffeine. Pond counts the frequency of usage of each object pool using approximate counting with extremely little memory usage, and recycles it automatically.

The default policy and weights reduce the memory usage by 48.85% and the borrow hit rate at 100% when the traffic is more randomly averaged.

The default policy and weights reduce the memory usage by 45.7% and the borrow hit rate at 100% when the traffic is more in line with the 2/8 law.

Overview

Using Pond requires the implementation of an object factory, PooledObjectFactory, which provides object creation, initialization, destruction, validation, and other operations called by Pond. So in order for the object pool to support holding completely different objects, Pond uses a dictionary to save the name of each factory class and the instantiated objects of the factory class it implements.

Ponds are thread-safe and coroutine-safe for borrowing and recycling, and Python's queue module provides a first-in, first-out (FIFO) data structure for multi-threaded programming. It can be used to safely pass messages or other data between producer and consumer threads. Locks are handled by the caller, and all multiple threads can safely and easily work with the same Queue instance. The borrowing and recycling of a Pond is all about manipulating the queue, so it is basically considered thread-safe.

The auto-recycle is executed at regular intervals, 300s by default. Automatically cleans up the objects in the infrequently used object pool.

It is recommended that you use Python 3.8 or greater.

User guide

To install Pond via pip, use the following pip command:

pip install pondpond
from pond import Pond, PooledObjectFactory, PooledObject

First you need to declare a factory class for the type of object you want to put in. For example, in the example below we want the pooled object to be Dog, so we first declare a PooledDogFactory class and implement PooledObjectFactory.

class Dog:
    name: str
    validate_result:bool = True


class PooledDogFactory(PooledObjectFactory):
    def createInstance(self) -> PooledObject:
        dog = Dog()
        dog.name = "puppy"
        return PooledObject(dog)

    def destroy(self, pooled_object: PooledObject):
        del pooled_object

    def reset(self, pooled_object: PooledObject) -> PooledObject:
        pooled_object.keeped_object.name = "puppy"
        return pooled_object

    def validate(self, pooled_object: PooledObject) -> bool:
        return pooled_object.keeped_object.validate_result

You need creat a new instance of Pond:

pond = Pond(borrowed_timeout=2,
            time_between_eviction_runs=-1,
            thread_daemon=True,
            eviction_weight=0.8)

Pond can be passed a number of parameters in:

borrowed_timeout: The maximum duration of the borrowed object. Defaults to 60.

time_between_eviction_runs: The interval for automatic recycling. Defaults to 300. If its value is -1, the recycling is turned off.

thread_daemon: A boolean value indicating whether the pond's thread is a daemon thread. Defaults to True.

eviction_weight: Automatic recycling weight. Defaults to 0.8.

loop: Automatic recycling weight. Defaults to 0.8.

Creat a new instance of your custom class of fatory:

factory = PooledDogFactory(pooled_maxsize=10, least_one=False)

pooled_maxsize: The maximum number of objects that can be placed in the object pool of the objects generated by this factory class.

least_one: If True, the object pool of objects generated by this factory class will retain at least one object when it enters auto-recycle.

Register the factory object with Pond; by default, the class name of the factory class is used as the PooledObjectTree's key.

pond.register(factory)

Of course you can also customize its name:

pond.register(factory, name="PuppyFactory")

If the register succeeds, the Pond will begin creating objects based on the pooled_maxsize set in the factory until the pool is full.

Borrow and recycle object(You can also use coroutine function with coroutine locks):

pooled_object: PooledObject = pond.borrow(factory)
# or
pooled_object: PooledObject = pond.async_borrow(factory)
dog: Dog = pooled_object.use()
pond.recycle(pooled_object, factory)
# or
pond.async_recycle(pooled_object, factory)

Borrow and recycle object by name:

pooled_object: PooledObject = pond.borrow(name="PuppyFactory")
dog: Dog = pooled_object.use()
pond.recycle(pooled_object, name="PuppyFactory")

Clear a object pool:

pond.clear(factory)
# or
pond.async_clear(factory)

Clear a object pool by name:

pond.clear(name="PuppyFactory")

More Details

For more details, see our user's guide or my blog(https://qin.news).

References

@software{Pond,
  author = {Andy Qin},
  title = {{Pond: A high performance object-pooling library for Python}},
  year = {2022},
  url = {https://github.com/T-baby/pondpond},
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pondpond-1.4.1.tar.gz (15.2 kB view details)

Uploaded Source

Built Distribution

pondpond-1.4.1-py3-none-any.whl (14.5 kB view details)

Uploaded Python 3

File details

Details for the file pondpond-1.4.1.tar.gz.

File metadata

  • Download URL: pondpond-1.4.1.tar.gz
  • Upload date:
  • Size: 15.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: pdm/2.12.3 CPython/3.11.6

File hashes

Hashes for pondpond-1.4.1.tar.gz
Algorithm Hash digest
SHA256 8afa34b869d1434d21dd2ec12644abc3b1733fcda8fcf355300338a13a79bb7b
MD5 49e347e258540ec50c76c6e1b6b214bc
BLAKE2b-256 a69b8411458ca8ce8b5b9b135e4a19823f1caf958ca9985883db104323492982

See more details on using hashes here.

File details

Details for the file pondpond-1.4.1-py3-none-any.whl.

File metadata

  • Download URL: pondpond-1.4.1-py3-none-any.whl
  • Upload date:
  • Size: 14.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: pdm/2.12.3 CPython/3.11.6

File hashes

Hashes for pondpond-1.4.1-py3-none-any.whl
Algorithm Hash digest
SHA256 641028ead4e8018ca6de1220c660ddd6d6fbf62a60e72f410655dd0451d82880
MD5 cf98e3862bfc1dbf0ad798888708b5ba
BLAKE2b-256 c4d4f18d6985157cc68f76469480182cbee2a03a45858456955acf57f9dcbb4c

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page