No project description provided
Project description
ENN Ragged Buffer
This Python package implements an efficient RaggedBuffer
datatype that is similar to
a 3D numpy array, but which allows for variable sequence length in the second
dimension. It was created primarily for use in ENN-PPO
and currently only supports a small selection of the numpy array methods.
User Guide
Install the package with pip install ragged-buffer
.
The package currently supports two RaggedBuffer
variants, RaggedBufferF32
(storing float32 values) and RaggedBufferI64
(storing int64 values).
Creating a RaggedBuffer
There are three ways to create a RaggedBuffer
:
RaggedBufferF32(features: int)
creates an emptyRaggedBuffer
with the specified number of features.RaggedBufferF32.from_flattened(flattened: np.ndarray, lenghts: np.ndarray)
creates aRaggedBuffer
from a flattened 2D numpy array and a 1D numpy array of lengths.RaggedBufferF32.from_array
creates aRaggedBuffer
(with equal sequence lenghts) from a 3D numpy array.
Creating an empty buffer and pushing each row:
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create an empty RaggedBuffer with a feature size of 3
buffer = RaggedBufferF32(3)
# Push sequences with 3, 5, 0, and 1 elements
buffer.push(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.float32))
buffer.push(np.array([[10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24]], dtype=np.float32))
buffer.push(np.array([], dtype=np.float32))
buffer.push(np.array([[25, 25, 27]], dtype=np.float32))
Creating a RaggedBuffer from a flat 2D numpy array which combines the first and second dimension, and an array of sequence lengths:
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24], [25, 25, 27]], dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
Creating a RaggedBuffer from a 3D numpy array (all sequences have the same length):
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
Get size
The size0
, size1
, and size2
methods return the number of sequences, the number of elements in a sequence, and the number of features respectively.
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.zeros((9, 64), dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
# Get size of the first/batch dimension.
assert buffer.size0() == 10
# Get size of individual sequences.
assert buffer.size1(1) == 5
assert buffer.size1(2) == 0
# Get size of the last/feature dimension.
assert buffer.size2() == 64
Convert to numpy array
as_aray
converts a RaggedBuffer
to a flat 2D numpy array that combines the first and second dimension.
import numpy as np
from ragged_buffer import RaggedBufferI64
buffer = RaggedBufferI64(1)
buffer.push(np.array([[1], [1], [1]], dtype=np.int64))
buffer.push(np.array([[2], [2]], dtype=np.int64))
assert np.all(buffer.as_array(), np.array([[1], [1], [1], [2], [2]], dtype=np.int64))
Indexing
You can index a RaggedBuffer
with a single integer (returning a RaggedBuffer
with a single sequence), or with a numpy array of integers selecting/permuting multiple sequences.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create a new `RaggedBufferF32`
buffer = RaggedBufferF32.from_flattened(
np.arange(0, 40, dtype=np.float32).reshape(10, 4),
np.array([3, 5, 0, 1], dtype=np.int64)
)
# Retrieve the first sequence.
assert np.all(
buffer[0].as_array() ==
np.array([[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]], dtype=np.float32)
)
# Get a RaggedBatch with 2 randomly selected sequences.
buffer[np.random.permutation(4)[:2]]
Addition
You can add two RaggedBuffer
s with the +
operator if they have the same number of sequences, sequence lengths, and features. You can also add a RaggedBuffer
where all sequences have a length of 1 to a RaggedBuffer
with variable length sequences, broadcasting along each sequence.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create ragged buffer with dimensions (3, [1, 3, 2], 1)
rb3 = RaggedBufferI64(1)
rb3.push(np.array([[0]], dtype=np.int64))
rb3.push(np.array([[0], [1], [2]], dtype=np.int64))
rb3.push(np.array([[0], [5]], dtype=np.int64))
# Create ragged buffer with dimensions (3, [1, 1, 1], 1)
rb4 = RaggedBufferI64.from_array(np.array([0, 3, 10], dtype=np.int64).reshape(3, 1, 1))
# Add rb3 and rb4, broadcasting along the sequence dimension.
rb5 = rb3 + rb4
assert np.all(
rb5.as_array() == np.array([[0], [3], [4], [5], [10], [15]], dtype=np.int64)
)
Concatenation
The extend
method can be used to mutate a RaggedBuffer
by appending another RaggedBuffer
to it.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb1 = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb2 = RaggedBufferF32.from_array(np.zeros((2, 5, 3), dtype=np.float32))
rb1.extend(r2)
assert rb1.size0() == 6
Clear
The clear
method removes all elements from a RaggedBuffer
without deallocating the underlying memory.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb.clear()
assert rb.size0() == 0
License
ENN Ragged Buffer dual-licensed under Apache-2.0 and MIT.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ragged_buffer-0.2.10-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 43524b1a3d90da57439fd449a85fe510f5ae300c4a320e679f2dc44b01b7d2bf |
|
MD5 | 096c06867ac7d955bb7e2923a868f58f |
|
BLAKE2b-256 | 6dbc9764efcbcf2d16f23aca6cf572d7d374953ecc184c3a0b33d5e59b7bdb9b |
Hashes for ragged_buffer-0.2.10-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cab38efd4c5edba23626a3a38595607bfd82e85ce7556df73826d970558f46c0 |
|
MD5 | dd4982def946eda65bc4d2cb87a33df8 |
|
BLAKE2b-256 | d4bae7a9a21abb17db0feb4826e3fa84ac7b4ebc7b12580764f65b048727f99d |
Hashes for ragged_buffer-0.2.10-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2579d231de52f3e335ec10bffaa6540889a5c4397cc77d33f28fa332dbcd69f7 |
|
MD5 | 1e193e793990c44938d09ccae7edddf9 |
|
BLAKE2b-256 | 755052b554a39e8a45ed93cbaff78f6534d8a25d1bc6723de3a76e914c6aa025 |
Hashes for ragged_buffer-0.2.10-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b3e3b5b1cf8979693ce72741fcd7756df1a276adab0c6b71583e88512af71abe |
|
MD5 | 4573bc23431be0cfb6cee86a9ab7da13 |
|
BLAKE2b-256 | 9afc502efe9a3ca4dea890d060e5b224dacfa1bc824ff9dfaeb1f037174b31e4 |
Hashes for ragged_buffer-0.2.10-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c84ddb2e396d86c009127673926cfa764df8f58120f0b58504d27da3cae4fe2 |
|
MD5 | b0eb88e35ed85492c3298a7caecd98f7 |
|
BLAKE2b-256 | 7e77ba48f785036c33ee2bf801d2e42554b9101ee151cad0104c8eb9d21cb97e |
Hashes for ragged_buffer-0.2.10-cp38-cp38-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bbc074ec1fee2a1cc0b6c8c8b351152e11e17cc1b0e6f6218d4af10bcebf04d2 |
|
MD5 | 770092d46bae75907f34a5cf355a2bf7 |
|
BLAKE2b-256 | 3d9a1485a1b3148aa73641865cf8aa19afdc65a1ba7c1722ecd24340b18f13b7 |
Hashes for ragged_buffer-0.2.10-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93484877250abdd8d8c496e2b4b840a8d531390ed451cd512407ebb4aa8df8ce |
|
MD5 | cf72e463de3334b493f43caba0c7320a |
|
BLAKE2b-256 | 966de140400d3b37ea01b35c17d6835b5e4d12ba3123b42f3d21c22a521972da |
Hashes for ragged_buffer-0.2.10-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5feb9d282ad4bd9599e8d71654ce1419f213006fd86291b81963b88075f6a833 |
|
MD5 | 71955314d8c16c128c4f762011ad0359 |
|
BLAKE2b-256 | 9ab5614b5f19bd70c242738166609f9f59a66cacb4e60d66e61405c7a2e6c428 |
Hashes for ragged_buffer-0.2.10-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce5e07335bb133ddd8b6c7f60bd8eff5737e91ce58da31e9bd4b2ddca6bfa027 |
|
MD5 | eeda407b62e3537fe81a1bc3305e7587 |
|
BLAKE2b-256 | b7f9598e7b156b8fde48babda53276f005668ae10683a8fc2e7323b611238460 |
Hashes for ragged_buffer-0.2.10-cp36-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ea13b75e12fd1079e84c06691a7966be57d0e07810c01474dff6961f35a3c664 |
|
MD5 | f6521eea681c19f798606508763ff321 |
|
BLAKE2b-256 | f9b63775a9efe4e6e4a848aa4ea203fd46ec39a0ea0476ad981d2f90f967b5ad |