Skip to main content

The Wrapped Exchange Array is a convenient way in order to exchange array data easily via processes or remote nodes

Project description

wea.py

What is wea?

Giving the package a meaning - wea stands for Wrapped Exchange Array. If you want to share array-packed data with different processes, remote nodes or different language executables ( yes, that's the vision ), wea is aiming to be a lean, lightweight and convenient alternative to Protocol Buffers and Co.

It's inspired and adopted partly from Julia’s InterProcessCommunication WrappedArray.

The wrapped exchange array can be accessed like a numpy array because under the hood, numpy is applied.

If this sounds good to you, just give it a try.

Getting started

Install the package from Pypi

pip install wea

Quick API guide

Shared memory

import wea
import numpy as np

...
wa = wea.shared_memory.create_shared_array('/awesome-1', np.dtype('float64'), (10, 2))
wa[:] = my_new_data[:]
...

Creating a shared memory segment

In order to create a new shared memory segment, use the following snippet

import wea
import numpy as np

type = np.dtype('float64')
dims = (10, 2)
wa = wea.shared_memory.create_shared_array('/awesome-1', type, dims)
wa[:] = np.random.randn(dims[0], dims[1])

If creating was not possible because the segment already exists , a FileExistsError exception will be thrown.

Attaching to an existing shared memory array

If a wrapped exchange array was already created, you can attach to it simply by

import wea
import numpy as np

wa = wea.shared_memory.attach_shared_array('/awesome-1')
wa[:] = np.random.randn(dims[0], dims[1])

The metadata of the array are stored in the shared memory header segment and will be retrieved for the numpy array creation.

If attaching was not possible because the segment does not exist so far, a FileNotFoundError exception will be thrown.

Bytearray buffer memory

import wea
import numpy as np

...
wa = wea.buffered_memory.create_buffered_array(np.dtype('float64'), (10, 2))
wa[:] = my_new_data[:]
buf = wa.exchange_buffer
share(buf) # where share calls your prefered communication protocol
...

Creating a buffered memory segment

In order to create a new buffered memory segment, use the following snippet

import wea
import numpy as np

type = np.dtype('float64')
dims = (10, 2)
wa = wea.buffered_memory.create_buffered_array(type, dims)
wa[:] = np.random.randn(dims[0], dims[1])
buf: bytearray = wa.exchange_buffer

Actually it copies the content from the numpy array into the buffer. Thus, the current behavior is like a deep copy.

Loading from an existing buffered memory segment

If a wrapped exchange array was already created, you can load from it simply by

import wea
import numpy as np

buf: bytearray = receive() # where receive via your prefered communication protocol
wa = wea.buffered_memory.load_buffered_array(buf)

The metadata of the array are stored in the buffered memory header segment and will be retrieved for the numpy array creation.

Contributing

I welcome any contributions, enhancements, and bug-fixes. Open an issue on GitHub and submit a pull request.

License

wea.py is 100% free and open-source, under the MIT license. Use it however you want.

This package is Treeware. If you use it in production, then we ask that you buy the world a tree to thank us for our work. By contributing to the Treeware forest you’ll be creating employment for local families and restoring wildlife habitats.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

wea-0.1.3.tar.gz (15.8 kB view details)

Uploaded Source

Built Distribution

wea-0.1.3-py3-none-any.whl (11.2 kB view details)

Uploaded Python 3

File details

Details for the file wea-0.1.3.tar.gz.

File metadata

  • Download URL: wea-0.1.3.tar.gz
  • Upload date:
  • Size: 15.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7

File hashes

Hashes for wea-0.1.3.tar.gz
Algorithm Hash digest
SHA256 46e69895db3bcf93a6772daf91383aedad442da05739e1b921de5152ccdbab25
MD5 7f1d1202d5575a7df1778fbf5384b408
BLAKE2b-256 9b3a881b01dac83a23d5b8b08a32c6546aeb64ebf77cf0b69e998c8f36960611

See more details on using hashes here.

File details

Details for the file wea-0.1.3-py3-none-any.whl.

File metadata

  • Download URL: wea-0.1.3-py3-none-any.whl
  • Upload date:
  • Size: 11.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7

File hashes

Hashes for wea-0.1.3-py3-none-any.whl
Algorithm Hash digest
SHA256 0ab14dee99fa243e15fd1142e93e7b08e513ba1ad96ff6683ac58d2198f9bc28
MD5 e2820185048378f3bbbc77af4d11b740
BLAKE2b-256 edae8dcab0e547afd9588f1b34599351263b9f611cd5bb73829bd39eba6047a8

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page