Efficient RaggedBuffer datatype that implements 3D arrays with variable-length 2nd dimension.
Project description
ENN Ragged Buffer
This Python package implements an efficient RaggedBuffer
datatype that is similar to
a 3D numpy array, but which allows for variable sequence length in the second
dimension. It was created primarily for use in ENN-PPO
and currently only supports a small selection of the numpy array methods.
User Guide
Install the package with pip install ragged-buffer
.
The package currently supports three RaggedBuffer
variants, RaggedBufferF32
, RaggedBufferI64
, and RaggedBufferBool
.
Creating a RaggedBuffer
There are three ways to create a RaggedBuffer
:
RaggedBufferF32(features: int)
creates an emptyRaggedBuffer
with the specified number of features.RaggedBufferF32.from_flattened(flattened: np.ndarray, lenghts: np.ndarray)
creates aRaggedBuffer
from a flattened 2D numpy array and a 1D numpy array of lengths.RaggedBufferF32.from_array
creates aRaggedBuffer
(with equal sequence lenghts) from a 3D numpy array.
Creating an empty buffer and pushing each row:
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create an empty RaggedBuffer with a feature size of 3
buffer = RaggedBufferF32(3)
# Push sequences with 3, 5, 0, and 1 elements
buffer.push(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.float32))
buffer.push(np.array([[10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24]], dtype=np.float32))
buffer.push(np.array([], dtype=np.float32)) # Alternative: `buffer.push_empty()`
buffer.push(np.array([[25, 25, 27]], dtype=np.float32))
Creating a RaggedBuffer from a flat 2D numpy array which combines the first and second dimension, and an array of sequence lengths:
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24], [25, 25, 27]], dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
Creating a RaggedBuffer from a 3D numpy array (all sequences have the same length):
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
Get size
The size0
, size1
, and size2
methods return the number of sequences, the number of elements in a sequence, and the number of features respectively.
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.zeros((9, 64), dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
# Get size of the first/batch dimension.
assert buffer.size0() == 10
# Get size of individual sequences.
assert buffer.size1(1) == 5
assert buffer.size1(2) == 0
# Get size of the last/feature dimension.
assert buffer.size2() == 64
Convert to numpy array
as_aray
converts a RaggedBuffer
to a flat 2D numpy array that combines the first and second dimension.
import numpy as np
from ragged_buffer import RaggedBufferI64
buffer = RaggedBufferI64(1)
buffer.push(np.array([[1], [1], [1]], dtype=np.int64))
buffer.push(np.array([[2], [2]], dtype=np.int64))
assert np.all(buffer.as_array(), np.array([[1], [1], [1], [2], [2]], dtype=np.int64))
Indexing
You can index a RaggedBuffer
with a single integer (returning a RaggedBuffer
with a single sequence), or with a numpy array of integers selecting/permuting multiple sequences.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create a new `RaggedBufferF32`
buffer = RaggedBufferF32.from_flattened(
np.arange(0, 40, dtype=np.float32).reshape(10, 4),
np.array([3, 5, 0, 1], dtype=np.int64)
)
# Retrieve the first sequence.
assert np.all(
buffer[0].as_array() ==
np.array([[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]], dtype=np.float32)
)
# Get a RaggedBatch with 2 randomly selected sequences.
buffer[np.random.permutation(4)[:2]]
Addition
You can add two RaggedBuffer
s with the +
operator if they have the same number of sequences, sequence lengths, and features. You can also add a RaggedBuffer
where all sequences have a length of 1 to a RaggedBuffer
with variable length sequences, broadcasting along each sequence.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create ragged buffer with dimensions (3, [1, 3, 2], 1)
rb3 = RaggedBufferI64(1)
rb3.push(np.array([[0]], dtype=np.int64))
rb3.push(np.array([[0], [1], [2]], dtype=np.int64))
rb3.push(np.array([[0], [5]], dtype=np.int64))
# Create ragged buffer with dimensions (3, [1, 1, 1], 1)
rb4 = RaggedBufferI64.from_array(np.array([0, 3, 10], dtype=np.int64).reshape(3, 1, 1))
# Add rb3 and rb4, broadcasting along the sequence dimension.
rb5 = rb3 + rb4
assert np.all(
rb5.as_array() == np.array([[0], [3], [4], [5], [10], [15]], dtype=np.int64)
)
Concatenation
The extend
method can be used to mutate a RaggedBuffer
by appending another RaggedBuffer
to it.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb1 = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb2 = RaggedBufferF32.from_array(np.zeros((2, 5, 3), dtype=np.float32))
rb1.extend(r2)
assert rb1.size0() == 6
Clear
The clear
method removes all elements from a RaggedBuffer
without deallocating the underlying memory.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb.clear()
assert rb.size0() == 0
License
ENN Ragged Buffer dual-licensed under Apache-2.0 and MIT.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ragged_buffer-0.3.8-pp37-pypy37_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd1f5fbe11fd28b05343e77b526173d5b3c62d7b0f54ea6ab7aca9fded357964 |
|
MD5 | 95d527a2fa5c5a9ed5cb4a92539f2d10 |
|
BLAKE2b-256 | cbe0ef9648440b9cb68b30fbac8dd5cb00a83de69a47955ace78599fb6f7377e |
Hashes for ragged_buffer-0.3.8-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13aa6d1d2c1e1e1db3fd4cbb4676be7953c84cb00e2f50886e9f2b8ef945ac0d |
|
MD5 | dfce28054c08d2dbc811f20726da313e |
|
BLAKE2b-256 | 7700a32a106e82d99ea235de9c2b81b5fa40ab27ebad718f1efa1cfa78296ff7 |
Hashes for ragged_buffer-0.3.8-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13954ac2cbbfe9bd791fa4afc08eb2afc3348e2ab26c2ebfe8a81dcd468f2312 |
|
MD5 | e10d7ae4c29e6c9ee4db36cfee95b985 |
|
BLAKE2b-256 | 80d5c8029efed5187ed524a0fbf65e878b34fd7efc72460c4383ad4090cadad5 |
Hashes for ragged_buffer-0.3.8-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a093fffa9238f81ef4cf726cf248d4bf01572c039d702d01a273500917aaec9c |
|
MD5 | cd5130ae8e8997bdaa556f23cb6e208a |
|
BLAKE2b-256 | 57d14b84dbb8bdd7bc9a2d42c85e8c11ee67de23bf7bc05363383bc56ff09901 |
Hashes for ragged_buffer-0.3.8-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0798e43168897d583ff5cf61722a3d97d45be72f9e96a6d4c166e663a33085cc |
|
MD5 | f7720ff9560449f11678697fb5f9527e |
|
BLAKE2b-256 | 3117655cad9497bdd08c5feef4503b5df6bb97d416d921cf262aeb9b983fb517 |
Hashes for ragged_buffer-0.3.8-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59c7a73001870e7c8043dbf431c1937f9aad50aa7a053c0fbef1a9d17aeed5bb |
|
MD5 | f794bc8089c0f48be98bb66e8800b2e4 |
|
BLAKE2b-256 | 2fb21e8b571dd9378edf2d72eddc3576e3e27054e837180291fea80a629a3230 |
Hashes for ragged_buffer-0.3.8-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 69673dee4c0163cd00605e40b527bcf64ddcc492a46c0b67162f12f11322ad53 |
|
MD5 | 01e8e4f52f4169fca3f88df8b4cacb54 |
|
BLAKE2b-256 | ba59319d7d6e0f9242d6505fde2c67ea90126f44c79ffdced90e999b9b5de834 |
Hashes for ragged_buffer-0.3.8-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b960ba1f0150e0fe4b1c22a34bc6655ce8ccbcbc84e3a7cbc94c9476a1326836 |
|
MD5 | fafe2050f2e2db2ecc7084096f76615b |
|
BLAKE2b-256 | a1dca93fc9b468b3a9137618318583bb61209444861b3a1b88f646cd6f267548 |
Hashes for ragged_buffer-0.3.8-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa6a324eac66db7fb7adc4b17a408709f5bb22bc1a13768b755fe78f2d1e75d2 |
|
MD5 | f34a24c452b82a28d292f0c96eb93a74 |
|
BLAKE2b-256 | 1756de17575031fda8bd406023b7f98028133fcbc8827769467093c6bd821b6f |
Hashes for ragged_buffer-0.3.8-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e58ac8e6793b99e8f5300cb749e695bc012948c5a380fdaef466d6f04f16a6a3 |
|
MD5 | ebf917928335a6644d6c453d5d4a1043 |
|
BLAKE2b-256 | b2e7df64d7cadc9440099c2ceb719182cd84f58078e8bbba899110696b1c98e3 |
Hashes for ragged_buffer-0.3.8-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 494514e69661e4484aa011458c1cea1f5f433aa78469ad718aecdc9020d09652 |
|
MD5 | 2f9b3da0a0c6229745e7f12a19204f1d |
|
BLAKE2b-256 | 9ebc29815c961322b1c20dc8017188965ec76a37e91211e5c6aae7e6ca6de864 |
Hashes for ragged_buffer-0.3.8-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e13429dceeb007a646c2b165130a70a64239aa6e41793879e84dbb47fab4f95 |
|
MD5 | e904f3983431f5d75c7cf8a3b1c46cd7 |
|
BLAKE2b-256 | 57bc74f6704c5f24122aae87b9cb542bccd9771f7a5c80fa3294fab3c0869d24 |
Hashes for ragged_buffer-0.3.8-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a73b0b38332b7fcfde48d49ec34058df809fa52c64c746aaa884d18a7575d15 |
|
MD5 | 20483b771ea2e44e7b4ddeddd50bbcc8 |
|
BLAKE2b-256 | 96bf1e49c236969ca03028c6ca0f2975bf56ba235b72f7f2a4c109ef603ea45b |