No project description provided
Project description
ENN Ragged Buffer
This Python package implements an efficient RaggedBuffer
datatype that is similar to
a 3D numpy array, but which allows for variable sequence length in the second
dimension. It was created primarily for use in ENN-PPO
and currently only supports a small selection of the numpy array methods.
User Guide
Install the package with pip install ragged-buffer
.
The package currently supports two RaggedBuffer
variants, RaggedBufferF32
(storing float32 values) and RaggedBufferI64
(storing int64 values).
Creating a RaggedBuffer
There are three ways to create a RaggedBuffer
:
RaggedBufferF32(features: int)
creates an emptyRaggedBuffer
with the specified number of features.RaggedBufferF32.from_flattened(flattened: np.ndarray, lenghts: np.ndarray)
creates aRaggedBuffer
from a flattened 2D numpy array and a 1D numpy array of lengths.RaggedBufferF32.from_array
creates aRaggedBuffer
(with equal sequence lenghts) from a 3D numpy array.
Creating an empty buffer and pushing each row:
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create an empty RaggedBuffer with a feature size of 3
buffer = RaggedBufferF32(3)
# Push sequences with 3, 5, 0, and 1 elements
buffer.push(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.float32))
buffer.push(np.array([[10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24]], dtype=np.float32))
buffer.push(np.array([], dtype=np.float32))
buffer.push(np.array([[25, 25, 27]], dtype=np.float32))
Creating a RaggedBuffer from a flat 2D numpy array which combines the first and second dimension, and an array of sequence lengths:
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24], [25, 25, 27]], dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
Creating a RaggedBuffer from a 3D numpy array (all sequences have the same length):
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
Get size
The size0
, size1
, and size2
methods return the number of sequences, the number of elements in a sequence, and the number of features respectively.
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.zeros((9, 64), dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
# Get size of the first/batch dimension.
assert buffer.size0() == 10
# Get size of individual sequences.
assert buffer.size1(1) == 5
assert buffer.size1(2) == 0
# Get size of the last/feature dimension.
assert buffer.size2() == 64
Convert to numpy array
as_aray
converts a RaggedBuffer
to a flat 2D numpy array that combines the first and second dimension.
import numpy as np
from ragged_buffer import RaggedBufferI64
buffer = RaggedBufferI64(1)
buffer.push(np.array([[1], [1], [1]], dtype=np.int64))
buffer.push(np.array([[2], [2]], dtype=np.int64))
assert np.all(buffer.as_array(), np.array([[1], [1], [1], [2], [2]], dtype=np.int64))
Indexing
You can index a RaggedBuffer
with a single integer (returning a RaggedBuffer
with a single sequence), or with a numpy array of integers selecting/permuting multiple sequences.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create a new `RaggedBufferF32`
buffer = RaggedBufferF32.from_flattened(
np.arange(0, 40, dtype=np.float32).reshape(10, 4),
np.array([3, 5, 0, 1], dtype=np.int64)
)
# Retrieve the first sequence.
assert np.all(
buffer[0].as_array() ==
np.array([[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]], dtype=np.float32)
)
# Get a RaggedBatch with 2 randomly selected sequences.
buffer[np.random.permutation(4)[:2]]
Addition
You can add two RaggedBuffer
s with the +
operator if they have the same number of sequences, sequence lengths, and features. You can also add a RaggedBuffer
where all sequences have a length of 1 to a RaggedBuffer
with variable length sequences, broadcasting along each sequence.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create ragged buffer with dimensions (3, [1, 3, 2], 1)
rb3 = RaggedBufferI64(1)
rb3.push(np.array([[0]], dtype=np.int64))
rb3.push(np.array([[0], [1], [2]], dtype=np.int64))
rb3.push(np.array([[0], [5]], dtype=np.int64))
# Create ragged buffer with dimensions (3, [1, 1, 1], 1)
rb4 = RaggedBufferI64.from_array(np.array([0, 3, 10], dtype=np.int64).reshape(3, 1, 1))
# Add rb3 and rb4, broadcasting along the sequence dimension.
rb5 = rb3 + rb4
assert np.all(
rb5.as_array() == np.array([[0], [3], [4], [5], [10], [15]], dtype=np.int64)
)
Concatenation
The extend
method can be used to mutate a RaggedBuffer
by appending another RaggedBuffer
to it.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb1 = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb2 = RaggedBufferF32.from_array(np.zeros((2, 5, 3), dtype=np.float32))
rb1.extend(r2)
assert rb1.size0() == 6
Clear
The clear
method removes all elements from a RaggedBuffer
without deallocating the underlying memory.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb.clear()
assert rb.size0() == 0
License
ENN Ragged Buffer dual-licensed under Apache-2.0 and MIT.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ragged_buffer-0.2.5-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1281b4e6c27b2a29757a2f72e6f79bd05dff3af49e1d91b9e2cf477613a37b0 |
|
MD5 | 8b6667c31f84e625618caa03f7f044da |
|
BLAKE2b-256 | c17b1a78083041f571e1c6889b2cc8d3aacfe7751e4cce9be3fcb5ba6b74ce2a |
Hashes for ragged_buffer-0.2.5-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab94c5f7edf9936de9371ef1c39b4696015e13eae72f28462cf83e790526e835 |
|
MD5 | 14694e73ff440b16d6fa6e445f4ddeec |
|
BLAKE2b-256 | 892cbd05c18abe3f2eb4605ff300675f14ba6c276743a21c2f14f6209c67d8e5 |
Hashes for ragged_buffer-0.2.5-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7fa050fd391c2304aab53d4512501fe09c61b9c2be3081b401cf374617c8c1fb |
|
MD5 | 76119bedf6ec6113e3d256f85e96419e |
|
BLAKE2b-256 | 1be477d873ce82a67bdddd161b823c86aafad9542f08af9c22a64dd5bd4834e8 |
Hashes for ragged_buffer-0.2.5-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38762558d6685cf7840f853ddf5dbecd3e128348dd0c9eac7aaa2d44ed44ffe5 |
|
MD5 | cc036b1e8504f26dea224fd5d02ccc0e |
|
BLAKE2b-256 | e172853b67c764f4694cd724a7d69be4bb0186ae8c8c63ba4254b1428213985a |
Hashes for ragged_buffer-0.2.5-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ba4ba7115fb4d4014b7f432f341bd4f5007cb561748041b5abd9921daf54bdc |
|
MD5 | c788ef23fc4fa0cbf69bdd09bee1656e |
|
BLAKE2b-256 | 07a7a193053e1130897f0d3761ff4abc123320d1729de8ba6027f73452ee9cc9 |
Hashes for ragged_buffer-0.2.5-cp38-cp38-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2fdefc2c03a2aa36829c2fb81cba5a1852412902659ea92e39785229ff4f2451 |
|
MD5 | 1cdb86248a01dc8d33c4c62dbdf48d33 |
|
BLAKE2b-256 | 84561d2913efc1ca3e0fe03298faf0bbdd68f23279dcfef5b415a03b8a96b94e |
Hashes for ragged_buffer-0.2.5-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85a1c78d7088b43fbb3622197428410e8aaa16c7c648ee4ac8b64788cb872a80 |
|
MD5 | 51ff3c3029e35621555d120fff1ba6b7 |
|
BLAKE2b-256 | 209ec2046ceef33b55f006a7a000f563e73feb58726a69f1af0f9a0237637fee |
Hashes for ragged_buffer-0.2.5-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5d7475b4f8342341f1f569f6ce01694fabfcc407fb7ca71ca0ad43600c71b2ad |
|
MD5 | 62b4ae4f8f1fc75c6f72b44fc7eea084 |
|
BLAKE2b-256 | 9ffaa10ed7bc6bfcc9a8aa22ec7b8e68156131a1a24f248492a0ca53b23431f4 |
Hashes for ragged_buffer-0.2.5-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4b107c6b65dc43391b87726257b0b4d513065851ae30c5bd49bfd8202c7e9cb5 |
|
MD5 | 8a19bdb9f906edb8de6d38644db2b694 |
|
BLAKE2b-256 | b5b9f37ac0488db703b542b320d98c6c190630cafdae9b2690491478cc003f8f |
Hashes for ragged_buffer-0.2.5-cp36-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9317dd6fb244c46efdd2353dd15aa502fab6047476330842ff1f9ebf0ee5b4e9 |
|
MD5 | 8848160d4566944dfade6fc89a81eb16 |
|
BLAKE2b-256 | f2ea34f3ea0d3753feca4b6de89c5bd05e0088cb0ba647f4a7b764cddb5074e9 |