No project description provided
Project description
ENN Ragged Buffer
This Python package implements an efficient RaggedBuffer
datatype that is similar to
a 3D numpy array, but which allows for variable sequence length in the second
dimension. It was created primarily for use in ENN-PPO
and currently only supports a small selection of the numpy array methods.
User Guide
Install the package with pip install ragged-buffer
.
The package currently supports three RaggedBuffer
variants, RaggedBufferF32
, RaggedBufferI64
, and RaggedBufferBool
.
Creating a RaggedBuffer
There are three ways to create a RaggedBuffer
:
RaggedBufferF32(features: int)
creates an emptyRaggedBuffer
with the specified number of features.RaggedBufferF32.from_flattened(flattened: np.ndarray, lenghts: np.ndarray)
creates aRaggedBuffer
from a flattened 2D numpy array and a 1D numpy array of lengths.RaggedBufferF32.from_array
creates aRaggedBuffer
(with equal sequence lenghts) from a 3D numpy array.
Creating an empty buffer and pushing each row:
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create an empty RaggedBuffer with a feature size of 3
buffer = RaggedBufferF32(3)
# Push sequences with 3, 5, 0, and 1 elements
buffer.push(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.float32))
buffer.push(np.array([[10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24]], dtype=np.float32))
buffer.push(np.array([], dtype=np.float32)) # Alternative: `buffer.push_empty()`
buffer.push(np.array([[25, 25, 27]], dtype=np.float32))
Creating a RaggedBuffer from a flat 2D numpy array which combines the first and second dimension, and an array of sequence lengths:
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24], [25, 25, 27]], dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
Creating a RaggedBuffer from a 3D numpy array (all sequences have the same length):
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
Get size
The size0
, size1
, and size2
methods return the number of sequences, the number of elements in a sequence, and the number of features respectively.
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.zeros((9, 64), dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
# Get size of the first/batch dimension.
assert buffer.size0() == 10
# Get size of individual sequences.
assert buffer.size1(1) == 5
assert buffer.size1(2) == 0
# Get size of the last/feature dimension.
assert buffer.size2() == 64
Convert to numpy array
as_aray
converts a RaggedBuffer
to a flat 2D numpy array that combines the first and second dimension.
import numpy as np
from ragged_buffer import RaggedBufferI64
buffer = RaggedBufferI64(1)
buffer.push(np.array([[1], [1], [1]], dtype=np.int64))
buffer.push(np.array([[2], [2]], dtype=np.int64))
assert np.all(buffer.as_array(), np.array([[1], [1], [1], [2], [2]], dtype=np.int64))
Indexing
You can index a RaggedBuffer
with a single integer (returning a RaggedBuffer
with a single sequence), or with a numpy array of integers selecting/permuting multiple sequences.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create a new `RaggedBufferF32`
buffer = RaggedBufferF32.from_flattened(
np.arange(0, 40, dtype=np.float32).reshape(10, 4),
np.array([3, 5, 0, 1], dtype=np.int64)
)
# Retrieve the first sequence.
assert np.all(
buffer[0].as_array() ==
np.array([[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]], dtype=np.float32)
)
# Get a RaggedBatch with 2 randomly selected sequences.
buffer[np.random.permutation(4)[:2]]
Addition
You can add two RaggedBuffer
s with the +
operator if they have the same number of sequences, sequence lengths, and features. You can also add a RaggedBuffer
where all sequences have a length of 1 to a RaggedBuffer
with variable length sequences, broadcasting along each sequence.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create ragged buffer with dimensions (3, [1, 3, 2], 1)
rb3 = RaggedBufferI64(1)
rb3.push(np.array([[0]], dtype=np.int64))
rb3.push(np.array([[0], [1], [2]], dtype=np.int64))
rb3.push(np.array([[0], [5]], dtype=np.int64))
# Create ragged buffer with dimensions (3, [1, 1, 1], 1)
rb4 = RaggedBufferI64.from_array(np.array([0, 3, 10], dtype=np.int64).reshape(3, 1, 1))
# Add rb3 and rb4, broadcasting along the sequence dimension.
rb5 = rb3 + rb4
assert np.all(
rb5.as_array() == np.array([[0], [3], [4], [5], [10], [15]], dtype=np.int64)
)
Concatenation
The extend
method can be used to mutate a RaggedBuffer
by appending another RaggedBuffer
to it.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb1 = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb2 = RaggedBufferF32.from_array(np.zeros((2, 5, 3), dtype=np.float32))
rb1.extend(r2)
assert rb1.size0() == 6
Clear
The clear
method removes all elements from a RaggedBuffer
without deallocating the underlying memory.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb.clear()
assert rb.size0() == 0
License
ENN Ragged Buffer dual-licensed under Apache-2.0 and MIT.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ragged_buffer-0.2.13-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 81ca40ec0d225dbe6bef04a09cf459c378faf9f020005177ca45ef73ec3ee8c4 |
|
MD5 | 6b4048ccbbc2d6b320534cee78478815 |
|
BLAKE2b-256 | e1725de67666134314080abae6326367c5b81a0edd95d0d96df3c0876c8ba880 |
Hashes for ragged_buffer-0.2.13-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7118f86e065fcd1ba597676eb65020c58a5101e6e9dde0f667bba5b3b0059c82 |
|
MD5 | fc46d7c2d7cd3e6534e667f370627ee9 |
|
BLAKE2b-256 | b70f44b01db3dc3236ac485670c015d17b30714ed15dd69a76611ef6e942c26a |
Hashes for ragged_buffer-0.2.13-cp39-cp39-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 990e4637693dba04d79e9e41100dbd4bf41de0897defd2699789429140a341e5 |
|
MD5 | 3222ce99a1cbd13ff8b16155d57d10bd |
|
BLAKE2b-256 | fdb059b7d9ee3d2a9b7bfe0cd26275a34566e926d8bd9b725f4bb9ddb2e4fbfd |
Hashes for ragged_buffer-0.2.13-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 05ce1eefbe9e44e05131cc60a5e12d79cd6fb6ad02807893623a6f32f5f193e6 |
|
MD5 | 0ac7f6526bd3304e6d40122ca24b5198 |
|
BLAKE2b-256 | b8f92dfbda0a2d2ff13e3b03146e925ef1af8ba5589af25d474519b83249609c |
Hashes for ragged_buffer-0.2.13-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 71025353de8d1c3c0977a18fd97cba37f6ae4821b005ee060a8a0fe116a03308 |
|
MD5 | 5c47402c7623374d75dcd0c02864efd5 |
|
BLAKE2b-256 | c95e99dd0d3f8b2bf6f8a0af2334cad065b399fc90fba7f1cfda2d934585c139 |
Hashes for ragged_buffer-0.2.13-cp38-cp38-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5eca63aea4609bea75792b874bbd7c5a5ba87beeb3981b9a584b2a9e94e95888 |
|
MD5 | d8477d490d5b21fe9a9740cbbbe36992 |
|
BLAKE2b-256 | de727224a33dbbc7cdd66a3eda27810677764e09017327494335e8cd10f32998 |
Hashes for ragged_buffer-0.2.13-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbafd7101f47c3576a8fcf04ababf58440508c9e9ca99b34c67a3f73ff870d4f |
|
MD5 | 52de747689c184b162ccfe8f04a1229d |
|
BLAKE2b-256 | 85d96169dd07ff72230d33de62860c52b17c38989c60d786d88d488b4a8017eb |
Hashes for ragged_buffer-0.2.13-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4d10f7815b339686e913e96320a8c97df17e7a1bc46332a8ec2175f0711db88 |
|
MD5 | 44fb9610a36895069385e4c67f4a7f74 |
|
BLAKE2b-256 | 011e944f9263c4cc14b04e366cb24890350c08c8dcda5d75c50015ffde0c6808 |
Hashes for ragged_buffer-0.2.13-cp37-cp37m-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e5c7b15145ad6443cd0584e0c17a15cad511caa044cd94eb891c0c64b58b30e |
|
MD5 | 1e7db2b37f052a760865a50b8409ed81 |
|
BLAKE2b-256 | 4e4e0f1d4c4f25ab5b1105325d0bdac07bdd425c93fb19a9fefc0ad6e278cba5 |
Hashes for ragged_buffer-0.2.13-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6f34064de4e62d8b49b877e8b675860c1f0601f734ef709327418de8c2e1de45 |
|
MD5 | 815bc0aecf5b151e462bbeb58dedf3d0 |
|
BLAKE2b-256 | e036a4abf299004241f7c9fa40e8cf052b7b6f24bd55368f4acbfc3a294fec84 |
Hashes for ragged_buffer-0.2.13-cp36-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 48cfc6b69844616fe06c9a774daaeece15867ed797a0e2f688bf3a109f967bad |
|
MD5 | 075af33e36ce7f5815760f6f2e3228b6 |
|
BLAKE2b-256 | 9e3a42ca3a387ca67fbdcb8f6fbd547af940940a79f70ecbdfa207549737dc4e |