No project description provided
Project description
ENN Ragged Buffer
This Python package implements an efficient RaggedBuffer
datatype that is similar to
a 3D numpy array, but which allows for variable sequence length in the second
dimension. It was created primarily for use in ENN-PPO
and currently only supports a small selection of the numpy array methods.
User Guide
Install the package with pip install enn-ragged-buffer
.
The package currently supports two RaggedBuffer
variants, RaggedBufferF32
(storing float32 values) and RaggedBufferI64
(storing int64 values).
Creating a RaggedBuffer
There are three ways to create a RaggedBuffer
:
RaggedBufferF32(features: int)
creates an emptyRaggedBuffer
with the specified number of features.RaggedBufferF32.from_flattened(flattened: np.ndarray, lenghts: np.ndarray)
creates aRaggedBuffer
from a flattened 2D numpy array and a 1D numpy array of lengths.RaggedBufferF32.from_array
creates aRaggedBuffer
(with equal sequence lenghts) from a 3D numpy array.
Creating an empty buffer and pushing each row:
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create an empty RaggedBuffer with a feature size of 3
buffer = RaggedBufferF32(3)
# Push sequences with 3, 5, 0, and 1 elements
buffer.push(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.float32))
buffer.push(np.array([[10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24]], dtype=np.float32))
buffer.push(np.array([], dtype=np.float32))
buffer.push(np.array([[25, 25, 27]], dtype=np.float32))
Creating a RaggedBuffer from a flat 2D numpy array which combines the first and second dimension, and an array of sequence lengths:
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24], [25, 25, 27]], dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
Creating a RaggedBuffer from a 3D numpy array (all sequences have the same length):
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
Get size
The size0
, size1
, and size2
methods return the number of sequences, the number of elements in a sequence, and the number of features respectively.
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.zeros((9, 64), dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
# Get size of the first/batch dimension.
assert buffer.size0() == 10
# Get size of individual sequences.
assert buffer.size1(1) == 5
assert buffer.size1(2) == 0
# Get size of the last/feature dimension.
assert buffer.size2() == 64
Convert to numpy array
as_aray
converts a RaggedBuffer
to a flat 2D numpy array that combines the first and second dimension.
import numpy as np
from ragged_buffer import RaggedBufferI64
buffer = RaggedBufferI64(1)
buffer.push(np.array([[1], [1], [1]], dtype=np.int64))
buffer.push(np.array([[2], [2]], dtype=np.int64))
assert np.all(buffer.as_array(), np.array([[1], [1], [1], [2], [2]], dtype=np.int64))
Indexing
You can index a RaggedBuffer
with a single integer (returning a RaggedBuffer
with a single sequence), or with a numpy array of integers selecting/permuting multiple sequences.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create a new `RaggedBufferF32`
buffer = RaggedBufferF32.from_flattened(
np.arange(0, 40, dtype=np.float32).reshape(10, 4),
np.array([3, 5, 0, 1], dtype=np.int64)
)
# Retrieve the first sequence.
assert np.all(
buffer[0].as_array() ==
np.array([[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]], dtype=np.float32)
)
# Get a RaggedBatch with 2 randomly selected sequences.
buffer[np.random.permutation(4)[:2]]
Addition
You can add two RaggedBuffer
s with the +
operator if they have the same number of sequences, sequence lengths, and features. You can also add a RaggedBuffer
where all sequences have a length of 1 to a RaggedBuffer
with variable length sequences, broadcasting along each sequence.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create ragged buffer with dimensions (3, [1, 3, 2], 1)
rb3 = RaggedBufferI64(1)
rb3.push(np.array([[0]], dtype=np.int64))
rb3.push(np.array([[0], [1], [2]], dtype=np.int64))
rb3.push(np.array([[0], [5]], dtype=np.int64))
# Create ragged buffer with dimensions (3, [1, 1, 1], 1)
rb4 = RaggedBufferI64.from_array(np.array([0, 3, 10], dtype=np.int64).reshape(3, 1, 1))
# Add rb3 and rb4, broadcasting along the sequence dimension.
rb5 = rb3 + rb4
assert np.all(
rb5.as_array() == np.array([[0], [3], [4], [5], [10], [15]], dtype=np.int64)
)
Concatenation
The extend
method can be used to mutate a RaggedBuffer
by appending another RaggedBuffer
to it.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb1 = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb2 = RaggedBufferF32.from_array(np.zeros((2, 5, 3), dtype=np.float32))
rb1.extend(r2)
assert rb1.size0() == 6
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ragged_buffer-0.1.2-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 202779e0d75685fce3141af1cc6c0780b115b7dfa63c1e479e7f5f2008d26310 |
|
MD5 | 9f0fb3ffd86e2ba30b0f08b026d0fd4f |
|
BLAKE2b-256 | b34740ba689984125eb54cc4da6136087af1cb9aaff32619bfab4c3476975e8c |
Hashes for ragged_buffer-0.1.2-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ea355203cb3f13f8eedfda0c069c222773f5e0302ca3dadb1c29b53bb5515734 |
|
MD5 | 4a33a072dfaeed2f99b63617a623a371 |
|
BLAKE2b-256 | c0771e5c1c38c7f4b7db49f64e921315fd01afd75959240e59642e4bd84c5913 |
Hashes for ragged_buffer-0.1.2-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0ae8b4d11a475a95d90f719145be11f8af56b247945df49d67de71a461fad160 |
|
MD5 | 7a409b79922423c6cca14e88b5af8ecb |
|
BLAKE2b-256 | 0e1a782d033421ff68f2641240973a81de158b120270eabe9df4ff4048922ca1 |
Hashes for ragged_buffer-0.1.2-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | baa1692795be034475adba0f4c3d97bcfcae88d065f080b8b29ea36e4606a882 |
|
MD5 | cdb735e8e99b0c10fbb36853189edefb |
|
BLAKE2b-256 | bc565064bca2e7c98a0fddd130024388d9263eea81fc9d93972509cb5bf59f7b |
Hashes for ragged_buffer-0.1.2-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0ea82565365324a9f58b831b13ece2fbbda48d78473ab999860581db2e55dc5a |
|
MD5 | 1609a01b9a9e2307817c774fa8eb9dd7 |
|
BLAKE2b-256 | 7632e8c2f36575979fa27bc1ab9943b13aec4322b1ff118902f4ba04ee5dc0a0 |
Hashes for ragged_buffer-0.1.2-cp38-cp38-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2183d1230de5b4b72a3bb6e53f8199b0879bd05c794c5c7d2913e35f6d14d21 |
|
MD5 | 32afed9669fe674eb7a7370a7f1946db |
|
BLAKE2b-256 | 30b971d829e8376c0aedf63c331417b73a9272a936e61509c478b9adc9122ab8 |
Hashes for ragged_buffer-0.1.2-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 75772f34e60a118cc3ecf2d233d5135d72d8c38214737f29a4b3bb7c529c91b0 |
|
MD5 | ea131ac49c7b43f64768745a6d1225c4 |
|
BLAKE2b-256 | e1824ac904c66c7d85c647d87b8997ffccd83fde30e3c98879c0fb0728882a2f |
Hashes for ragged_buffer-0.1.2-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dea86d67765c66a677e83591ecef3883fc364151c6612c0e1f4b621a67809f9b |
|
MD5 | f2155e7f7d1294d73f2860d627d7d4d6 |
|
BLAKE2b-256 | 7f38c4fb29f193ddc453825bc92a32b747a82e1d6fb65bba237eb00582aac467 |
Hashes for ragged_buffer-0.1.2-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d77cc5dd625e5ea5b1bb036e303da2004a0a56f3c5a0255be5fac5d3891af4ce |
|
MD5 | 2cceead10732b7429961225bdc5ac113 |
|
BLAKE2b-256 | b214764f57032b7fcebdd0187b56da3f216f54c9a6aa24c02c376c85a9133694 |
Hashes for ragged_buffer-0.1.2-cp36-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e5e856b29fe3d09cc36b18d6a241f13030521bd922c07605a8a4022abd9a894d |
|
MD5 | 19b29218cecb37e2b6858faffb92d517 |
|
BLAKE2b-256 | 0c98f265a7f2c811fdc00b95fe39bdbc8445c81ef4e20867b6afe4715466a941 |
Hashes for ragged_buffer-0.1.2-cp36-cp36m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 99ab6770bec0e5b43ba1d5a724d20a160e428aa2c53ec6376584fd0342944f1e |
|
MD5 | 3d4753a80becc311e8356a66a02df634 |
|
BLAKE2b-256 | 3de4b03ce80051c05a660dfbb792dbadf091b076cf3e9c8b6fd01d950ea2ce6f |