No project description provided
Project description
ENN Ragged Buffer
This Python package implements an efficient RaggedBuffer
datatype that is similar to
a 3D numpy array, but which allows for variable sequence length in the second
dimension. It was created primarily for use in ENN-PPO
and currently only supports a small selection of the numpy array methods.
User Guide
Install the package with pip install ragged-buffer
.
The package currently supports two RaggedBuffer
variants, RaggedBufferF32
(storing float32 values) and RaggedBufferI64
(storing int64 values).
Creating a RaggedBuffer
There are three ways to create a RaggedBuffer
:
RaggedBufferF32(features: int)
creates an emptyRaggedBuffer
with the specified number of features.RaggedBufferF32.from_flattened(flattened: np.ndarray, lenghts: np.ndarray)
creates aRaggedBuffer
from a flattened 2D numpy array and a 1D numpy array of lengths.RaggedBufferF32.from_array
creates aRaggedBuffer
(with equal sequence lenghts) from a 3D numpy array.
Creating an empty buffer and pushing each row:
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create an empty RaggedBuffer with a feature size of 3
buffer = RaggedBufferF32(3)
# Push sequences with 3, 5, 0, and 1 elements
buffer.push(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.float32))
buffer.push(np.array([[10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24]], dtype=np.float32))
buffer.push(np.array([], dtype=np.float32))
buffer.push(np.array([[25, 25, 27]], dtype=np.float32))
Creating a RaggedBuffer from a flat 2D numpy array which combines the first and second dimension, and an array of sequence lengths:
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24], [25, 25, 27]], dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
Creating a RaggedBuffer from a 3D numpy array (all sequences have the same length):
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
Get size
The size0
, size1
, and size2
methods return the number of sequences, the number of elements in a sequence, and the number of features respectively.
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.zeros((9, 64), dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
# Get size of the first/batch dimension.
assert buffer.size0() == 10
# Get size of individual sequences.
assert buffer.size1(1) == 5
assert buffer.size1(2) == 0
# Get size of the last/feature dimension.
assert buffer.size2() == 64
Convert to numpy array
as_aray
converts a RaggedBuffer
to a flat 2D numpy array that combines the first and second dimension.
import numpy as np
from ragged_buffer import RaggedBufferI64
buffer = RaggedBufferI64(1)
buffer.push(np.array([[1], [1], [1]], dtype=np.int64))
buffer.push(np.array([[2], [2]], dtype=np.int64))
assert np.all(buffer.as_array(), np.array([[1], [1], [1], [2], [2]], dtype=np.int64))
Indexing
You can index a RaggedBuffer
with a single integer (returning a RaggedBuffer
with a single sequence), or with a numpy array of integers selecting/permuting multiple sequences.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create a new `RaggedBufferF32`
buffer = RaggedBufferF32.from_flattened(
np.arange(0, 40, dtype=np.float32).reshape(10, 4),
np.array([3, 5, 0, 1], dtype=np.int64)
)
# Retrieve the first sequence.
assert np.all(
buffer[0].as_array() ==
np.array([[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]], dtype=np.float32)
)
# Get a RaggedBatch with 2 randomly selected sequences.
buffer[np.random.permutation(4)[:2]]
Addition
You can add two RaggedBuffer
s with the +
operator if they have the same number of sequences, sequence lengths, and features. You can also add a RaggedBuffer
where all sequences have a length of 1 to a RaggedBuffer
with variable length sequences, broadcasting along each sequence.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create ragged buffer with dimensions (3, [1, 3, 2], 1)
rb3 = RaggedBufferI64(1)
rb3.push(np.array([[0]], dtype=np.int64))
rb3.push(np.array([[0], [1], [2]], dtype=np.int64))
rb3.push(np.array([[0], [5]], dtype=np.int64))
# Create ragged buffer with dimensions (3, [1, 1, 1], 1)
rb4 = RaggedBufferI64.from_array(np.array([0, 3, 10], dtype=np.int64).reshape(3, 1, 1))
# Add rb3 and rb4, broadcasting along the sequence dimension.
rb5 = rb3 + rb4
assert np.all(
rb5.as_array() == np.array([[0], [3], [4], [5], [10], [15]], dtype=np.int64)
)
Concatenation
The extend
method can be used to mutate a RaggedBuffer
by appending another RaggedBuffer
to it.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb1 = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb2 = RaggedBufferF32.from_array(np.zeros((2, 5, 3), dtype=np.float32))
rb1.extend(r2)
assert rb1.size0() == 6
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ragged_buffer-0.1.3-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cffa19e8f29a7e5700d63a1ef6c31e97f41c54a7747b978a65b8371dbdde1f54 |
|
MD5 | 2401e9b25d7d996ae5394b41c9122027 |
|
BLAKE2b-256 | a020d024ab93218502c817d671aacdb5c973b964f192ef68c71a2598a107d405 |
Hashes for ragged_buffer-0.1.3-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5742cf8148ee4849528aae390673e35f84d391ef7c47568f63557800a8a239fc |
|
MD5 | 093b74cf484c5439996b4d527cf63df7 |
|
BLAKE2b-256 | b490a85c24a171f5f17c402577466a51ec63d7eeceb512a917a7207653a05b29 |
Hashes for ragged_buffer-0.1.3-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 637ff53836b0adcf9c6d1923e94790b3d89ffd0be6f56727e9c7d94fae732d34 |
|
MD5 | 98b5d378ce615b5ab0d024d615806d5c |
|
BLAKE2b-256 | c13378b01ce2e84736b5477862a2d39c1defbdf180643ea81d2f9fa96ba2a694 |
Hashes for ragged_buffer-0.1.3-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 072ec7feff0a9390873a03c7e57235595af244a09a6f9c67e3336813a482e1ac |
|
MD5 | 8ccf1e93e33476fbf3d7e73f44258659 |
|
BLAKE2b-256 | af8887e5d2617ea6235eed088673ddc52fde523ecefd6f19f9bd7a7b841b6a4a |
Hashes for ragged_buffer-0.1.3-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 23931e093f210c657f61b4f06ca68a980b676554157dad1d30c78775ff1b6621 |
|
MD5 | 0f30358b81f2da5df3a9a4aa62101882 |
|
BLAKE2b-256 | 6d6aa95a48a64a7854c3ab02300f3c510a017eabc333240038ff6fe8caff0dbc |
Hashes for ragged_buffer-0.1.3-cp38-cp38-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0bed7ed40fc5c588c6f4b475944551d8650d95cd95cf5ce6bbe2955c64d94de6 |
|
MD5 | 8c9b96a0b308c5bfd4471fc04fc82a21 |
|
BLAKE2b-256 | b90bb88bdb15572e8d196a09ccb426f1c2e9c397d4b4193ca81cf7afd5a23107 |
Hashes for ragged_buffer-0.1.3-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f51b1d1876590c0435ee6d7341494bb5872a76f953d207804bfee666b4eefc75 |
|
MD5 | ab4d80a3d66d343a63d798d10d641291 |
|
BLAKE2b-256 | f5aa4a33c33a662f1d632d4a3c58e59da2cad6053edb9d5d19926ec23e68fab6 |
Hashes for ragged_buffer-0.1.3-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 712baeed28e01942491c6545bff20b17ffbb829cb7eaa30504348a1fc612f3de |
|
MD5 | 0c4e902ac5d87ab4300e40ec8e8cecd6 |
|
BLAKE2b-256 | b3a800899ff34b89f983ab4360fa3bf87d62658b81f57c01d96b5cda4f5b824c |
Hashes for ragged_buffer-0.1.3-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 097c647382573bca02b82ab45075eafa28002f2e1d596702389e4b172ad3d83b |
|
MD5 | 35ac90f90ec6cdf057d15c4d782bcb43 |
|
BLAKE2b-256 | 746a196ad3aa1e484463adbd0a588b00febdbc76f2ef284ea7da01a476e12659 |
Hashes for ragged_buffer-0.1.3-cp36-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2382b2a80d390fa54c350fda66537cfb226d21f9988cd82b4ed1b804f6b2343d |
|
MD5 | 509552739403c0880c8f71b970793a2a |
|
BLAKE2b-256 | 8e5fb007337580c3f8728a9ad3b859301b345e89d943c3afbd5673c9a54e6567 |
Hashes for ragged_buffer-0.1.3-cp36-cp36m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 048f619d03ce0ae40926eba4d60d13e05fb6e203be85adddca60215e3be16aa5 |
|
MD5 | da12b4985461c8fa9532a6319b7a8f5e |
|
BLAKE2b-256 | 71012286b74974da59fd947c48c7ff173be97f7c7cf3283c57d45d4d74f52fbf |