Skip to main content

A very simple shared memory dict implementation

Project description

Shared Memory Dict

A very simple shared memory dict implementation.

Requires: Python >= 3.8

>>> # In the first Python interactive shell
>> from shared_memory_dict import SharedMemoryDict
>> smd = SharedMemoryDict(name='tokens', size=1024)
>> smd['some-key'] = 'some-value-with-any-type'
>> smd['some-key']
'some-value-with-any-type'

>>> # In either the same shell or a new Python shell on the same machine
>> existing_smd = SharedMemoryDict(name='tokens', size=1024)
>>> existing_smd['some-key']
'some-value-with-any-type'
>>> existing_smd['new-key'] = 'some-value-with-any-type'


>>> # Back in the first Python interactive shell, smd reflects this change
>> smd['new-key']
'some-value-with-any-type'

>>> # Clean up from within the second Python shell
>>> existing_smd.shm.close()  # or "del existing_smd"

>>> # Clean up from within the first Python shell
>>> smd.shm.close()
>>> smd.shm.unlink()  # Free and release the shared memory block at the very end
>>> del smd  # use of smd after call unlink() is unsupported

The arg name defines the location of the memory block, so if you want to share the memory between process use the same name. The size (in bytes) occupied by the contents of the dictionary depends on the serialization used in storage. By default pickle is used.

Installation

Using pip:

pip install shared-memory-dict

Locks

To use multiprocessing.Lock on write operations of shared memory dict set environment variable SHARED_MEMORY_USE_LOCK=1.

Serialization

We use pickle as default to read and write the data into the shared memory block.

You can create a custom serializer by implementing the dumps and loads methods.

Custom serializers should raise SerializationError if the serialization fails and DeserializationError if the deserialization fails. Both are defined in the shared_memory_dict.serializers module.

An example of a JSON serializer extracted from serializers module:

NULL_BYTE: Final = b"\x00"


class JSONSerializer:
    def dumps(self, obj: dict) -> bytes:
        try:
            return json.dumps(obj).encode() + NULL_BYTE
        except (ValueError, TypeError):
            raise SerializationError(obj)

    def loads(self, data: bytes) -> dict:
        data = data.split(NULL_BYTE, 1)[0]
        try:
            return json.loads(data)
        except json.JSONDecodeError:
            raise DeserializationError(data)

Note: A null byte is used to separate the dictionary contents from the bytes that are in memory.

To use the custom serializer you must set it when creating a new shared memory dict instance:

>>> smd = SharedMemoryDict(name='tokens', size=1024, serializer=JSONSerializer())

Caveat

The pickle module is not secure. Only unpickle data you trust.

See more here.

Django Cache Implementation

There's a Django Cache Implementation with Shared Memory Dict:

# settings/base.py
CACHES = {
    'default': {
        'BACKEND': 'shared_memory_dict.caches.django.SharedMemoryCache',
        'LOCATION': 'memory',
        'OPTIONS': {'MEMORY_BLOCK_SIZE': 1024}
    }
}

Install with: pip install "shared-memory-dict[django]"

Caveat

With Django cache implementation the keys only expire when they're read. Be careful with memory usage

AioCache Backend

There's also a AioCache Backend Implementation with Shared Memory Dict:

From aiocache import caches

caches.set_config({
    'default': {
        'cache': 'shared_memory_dict.caches.aiocache.SharedMemoryCache',
        'size': 1024,
    },
})

This implementation is very based on aiocache SimpleMemoryCache

Install with: pip install "shared-memory-dict[aiocache]"

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

shared-memory-dict-0.7.2.tar.gz (9.3 kB view details)

Uploaded Source

Built Distribution

shared_memory_dict-0.7.2-py3-none-any.whl (10.4 kB view details)

Uploaded Python 3

File details

Details for the file shared-memory-dict-0.7.2.tar.gz.

File metadata

  • Download URL: shared-memory-dict-0.7.2.tar.gz
  • Upload date:
  • Size: 9.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.4 CPython/3.8.5 Linux/5.11.0-1022-azure

File hashes

Hashes for shared-memory-dict-0.7.2.tar.gz
Algorithm Hash digest
SHA256 8294257e1a2466afb5e19a878a8c446325b8913d57d6d9578ccba905564a974d
MD5 fee3a656e7fda58d4a96797a0a32d43a
BLAKE2b-256 340a16c63e478658ce9ccb9e5d04b7650347a1e00f4db6da11487fdeb45b6b94

See more details on using hashes here.

File details

Details for the file shared_memory_dict-0.7.2-py3-none-any.whl.

File metadata

  • Download URL: shared_memory_dict-0.7.2-py3-none-any.whl
  • Upload date:
  • Size: 10.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.4 CPython/3.8.5 Linux/5.11.0-1022-azure

File hashes

Hashes for shared_memory_dict-0.7.2-py3-none-any.whl
Algorithm Hash digest
SHA256 b1d5a40a7a40e764187ae3a95c6096a4c4a8b27625408c63950d9db2b768506b
MD5 53004321f8604b90e30f36711816968d
BLAKE2b-256 0ffc8147ce28a7e16029bc6432f81a461575ea7e7d1b852a75e2acdb0011ce41

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page