No project description provided
Project description
ENN Ragged Buffer
This Python package implements an efficient RaggedBuffer
datatype that is similar to
a 3D numpy array, but which allows for variable sequence length in the second
dimension. It was created primarily for use in ENN-PPO
and currently only supports a small selection of the numpy array methods.
User Guide
Install the package with pip install ragged-buffer
.
The package currently supports three RaggedBuffer
variants, RaggedBufferF32
, RaggedBufferI64
, and RaggedBufferBool
.
Creating a RaggedBuffer
There are three ways to create a RaggedBuffer
:
RaggedBufferF32(features: int)
creates an emptyRaggedBuffer
with the specified number of features.RaggedBufferF32.from_flattened(flattened: np.ndarray, lenghts: np.ndarray)
creates aRaggedBuffer
from a flattened 2D numpy array and a 1D numpy array of lengths.RaggedBufferF32.from_array
creates aRaggedBuffer
(with equal sequence lenghts) from a 3D numpy array.
Creating an empty buffer and pushing each row:
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create an empty RaggedBuffer with a feature size of 3
buffer = RaggedBufferF32(3)
# Push sequences with 3, 5, 0, and 1 elements
buffer.push(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.float32))
buffer.push(np.array([[10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24]], dtype=np.float32))
buffer.push(np.array([], dtype=np.float32)) # Alternative: `buffer.push_empty()`
buffer.push(np.array([[25, 25, 27]], dtype=np.float32))
Creating a RaggedBuffer from a flat 2D numpy array which combines the first and second dimension, and an array of sequence lengths:
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24], [25, 25, 27]], dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
Creating a RaggedBuffer from a 3D numpy array (all sequences have the same length):
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
Get size
The size0
, size1
, and size2
methods return the number of sequences, the number of elements in a sequence, and the number of features respectively.
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.zeros((9, 64), dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
# Get size of the first/batch dimension.
assert buffer.size0() == 10
# Get size of individual sequences.
assert buffer.size1(1) == 5
assert buffer.size1(2) == 0
# Get size of the last/feature dimension.
assert buffer.size2() == 64
Convert to numpy array
as_aray
converts a RaggedBuffer
to a flat 2D numpy array that combines the first and second dimension.
import numpy as np
from ragged_buffer import RaggedBufferI64
buffer = RaggedBufferI64(1)
buffer.push(np.array([[1], [1], [1]], dtype=np.int64))
buffer.push(np.array([[2], [2]], dtype=np.int64))
assert np.all(buffer.as_array(), np.array([[1], [1], [1], [2], [2]], dtype=np.int64))
Indexing
You can index a RaggedBuffer
with a single integer (returning a RaggedBuffer
with a single sequence), or with a numpy array of integers selecting/permuting multiple sequences.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create a new `RaggedBufferF32`
buffer = RaggedBufferF32.from_flattened(
np.arange(0, 40, dtype=np.float32).reshape(10, 4),
np.array([3, 5, 0, 1], dtype=np.int64)
)
# Retrieve the first sequence.
assert np.all(
buffer[0].as_array() ==
np.array([[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]], dtype=np.float32)
)
# Get a RaggedBatch with 2 randomly selected sequences.
buffer[np.random.permutation(4)[:2]]
Addition
You can add two RaggedBuffer
s with the +
operator if they have the same number of sequences, sequence lengths, and features. You can also add a RaggedBuffer
where all sequences have a length of 1 to a RaggedBuffer
with variable length sequences, broadcasting along each sequence.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create ragged buffer with dimensions (3, [1, 3, 2], 1)
rb3 = RaggedBufferI64(1)
rb3.push(np.array([[0]], dtype=np.int64))
rb3.push(np.array([[0], [1], [2]], dtype=np.int64))
rb3.push(np.array([[0], [5]], dtype=np.int64))
# Create ragged buffer with dimensions (3, [1, 1, 1], 1)
rb4 = RaggedBufferI64.from_array(np.array([0, 3, 10], dtype=np.int64).reshape(3, 1, 1))
# Add rb3 and rb4, broadcasting along the sequence dimension.
rb5 = rb3 + rb4
assert np.all(
rb5.as_array() == np.array([[0], [3], [4], [5], [10], [15]], dtype=np.int64)
)
Concatenation
The extend
method can be used to mutate a RaggedBuffer
by appending another RaggedBuffer
to it.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb1 = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb2 = RaggedBufferF32.from_array(np.zeros((2, 5, 3), dtype=np.float32))
rb1.extend(r2)
assert rb1.size0() == 6
Clear
The clear
method removes all elements from a RaggedBuffer
without deallocating the underlying memory.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb.clear()
assert rb.size0() == 0
License
ENN Ragged Buffer dual-licensed under Apache-2.0 and MIT.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ragged_buffer-0.3.5-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7089e6d5fc0a26623922c84f302936307a284053b6d354d85f3bc9d7976d547c |
|
MD5 | 46bd95ef1ef93af9099f3cae8fcb97c4 |
|
BLAKE2b-256 | 157fae992a0d68827bb7123db6205495d715a73cafd8154ca705490269bf23db |
Hashes for ragged_buffer-0.3.5-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b68dfe2f48dbe64e21a3ddaf5b006aff6cec3acb15e14e769bb11905125b9d07 |
|
MD5 | 463fa00e716b4046f485002a56bea52e |
|
BLAKE2b-256 | 97b5c68bc9fbe427de12c391fcb8cdc64df748ed387fd0cbccab4d6e57d0b866 |
Hashes for ragged_buffer-0.3.5-cp39-cp39-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 523c2e4a6515845a001228d1c9e2bd4f24b8b193ddcdf0f38a2fd2d5da60e7b9 |
|
MD5 | 057e39d48214a856b6cf206c30b6540b |
|
BLAKE2b-256 | d5ea0355aa50ad53719a2309449fc70683dd60aa7e565afb4de209df33469e36 |
Hashes for ragged_buffer-0.3.5-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 193712847c8fc542d4d0035d70c4651469daf2fafad7a205921b5787b01469ae |
|
MD5 | 42430a2c6829f185685830bad0cab115 |
|
BLAKE2b-256 | c2c189fc307a9adb83ef5ccdf65bc14f73035e02477c8b8cb6c7456e3546999d |
Hashes for ragged_buffer-0.3.5-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ed3f512ca7707647fcb33dcf62bfd0563a866a0d02d0b9966c5b4a1410d69510 |
|
MD5 | 221d923bfd89423d725f315e63997ce2 |
|
BLAKE2b-256 | a022b81740d628048dfdd2e6f08edf8ab1a4c0c7cc0ff74d6379672adc54abbf |
Hashes for ragged_buffer-0.3.5-cp38-cp38-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ea4cca71faff4bfa5ece258a86181a5d3ea288ce33795be3ba369e5662557fb4 |
|
MD5 | 667ed6abc3f7adffc3b02cf9cb1df9d1 |
|
BLAKE2b-256 | ee8fb4b7d28fbf7e14062fec5f5d577460b809819ed042d01e8410fcabd7e2da |
Hashes for ragged_buffer-0.3.5-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c8d4ef497fedac45dd0dbcf840adfdf718130be2fa68b53225f2c1134938f90 |
|
MD5 | 2ab88d50326441890d8959ad85516d4e |
|
BLAKE2b-256 | cf9c6220d407264c841494219bff50c9d6ace55f7f7e5913d522f1c7b7e49f67 |
Hashes for ragged_buffer-0.3.5-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3406059d5e83def5d1935762fcc3b1e573ef6b4db611792212e461298939d209 |
|
MD5 | 931b31306e4e46a0d2bba9ea0581acd7 |
|
BLAKE2b-256 | 0c6fe89e27047705b04e5315be69b2c48779479ef7bcccfdcba355f7254d5382 |
Hashes for ragged_buffer-0.3.5-cp37-cp37m-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fbf00b54ed8474582e012bc2ffab1b087085822100f4128cc692a42a9106dfc4 |
|
MD5 | ff4268b74df15ac24b13c52385c3d8a6 |
|
BLAKE2b-256 | d59f045eb6d193422f3964cfa7b681c9c9f5aac491727a784dac2a5108254318 |
Hashes for ragged_buffer-0.3.5-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 18a9d57c6dd485c9a9c5c934dd8f0d2e873e72c9ac8cb7971bc5d5463b6e4404 |
|
MD5 | 4cb90b7fbac7ce61485007db3a7da6f0 |
|
BLAKE2b-256 | 4f28d40f9ddd32f8bc51b5b6d2c27dc68c14f0f76bbd468aafab3ffbaddfbf48 |
Hashes for ragged_buffer-0.3.5-cp36-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4eadbb0a528e5af20fe02e1ca5d2527d628a1e9c2dbfec70a9b87ee36bf3d96 |
|
MD5 | 4f947c6a86ac8b4021ecd4ffb4f9a256 |
|
BLAKE2b-256 | 050eb65fa560d80b5cdf9d3e83abfb81cf301b93743c3557a62fced2f525bd8b |