No project description provided
Project description
ENN Ragged Buffer
This Python package implements an efficient RaggedBuffer
datatype that is similar to
a 3D numpy array, but which allows for variable sequence length in the second
dimension. It was created primarily for use in ENN-PPO
and currently only supports a small selection of the numpy array methods.
User Guide
Install the package with pip install ragged-buffer
.
The package currently supports three RaggedBuffer
variants, RaggedBufferF32
, RaggedBufferI64
, and RaggedBufferBool
.
Creating a RaggedBuffer
There are three ways to create a RaggedBuffer
:
RaggedBufferF32(features: int)
creates an emptyRaggedBuffer
with the specified number of features.RaggedBufferF32.from_flattened(flattened: np.ndarray, lenghts: np.ndarray)
creates aRaggedBuffer
from a flattened 2D numpy array and a 1D numpy array of lengths.RaggedBufferF32.from_array
creates aRaggedBuffer
(with equal sequence lenghts) from a 3D numpy array.
Creating an empty buffer and pushing each row:
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create an empty RaggedBuffer with a feature size of 3
buffer = RaggedBufferF32(3)
# Push sequences with 3, 5, 0, and 1 elements
buffer.push(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.float32))
buffer.push(np.array([[10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24]], dtype=np.float32))
buffer.push(np.array([], dtype=np.float32)) # Alternative: `buffer.push_empty()`
buffer.push(np.array([[25, 25, 27]], dtype=np.float32))
Creating a RaggedBuffer from a flat 2D numpy array which combines the first and second dimension, and an array of sequence lengths:
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24], [25, 25, 27]], dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
Creating a RaggedBuffer from a 3D numpy array (all sequences have the same length):
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
Get size
The size0
, size1
, and size2
methods return the number of sequences, the number of elements in a sequence, and the number of features respectively.
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.zeros((9, 64), dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
# Get size of the first/batch dimension.
assert buffer.size0() == 10
# Get size of individual sequences.
assert buffer.size1(1) == 5
assert buffer.size1(2) == 0
# Get size of the last/feature dimension.
assert buffer.size2() == 64
Convert to numpy array
as_aray
converts a RaggedBuffer
to a flat 2D numpy array that combines the first and second dimension.
import numpy as np
from ragged_buffer import RaggedBufferI64
buffer = RaggedBufferI64(1)
buffer.push(np.array([[1], [1], [1]], dtype=np.int64))
buffer.push(np.array([[2], [2]], dtype=np.int64))
assert np.all(buffer.as_array(), np.array([[1], [1], [1], [2], [2]], dtype=np.int64))
Indexing
You can index a RaggedBuffer
with a single integer (returning a RaggedBuffer
with a single sequence), or with a numpy array of integers selecting/permuting multiple sequences.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create a new `RaggedBufferF32`
buffer = RaggedBufferF32.from_flattened(
np.arange(0, 40, dtype=np.float32).reshape(10, 4),
np.array([3, 5, 0, 1], dtype=np.int64)
)
# Retrieve the first sequence.
assert np.all(
buffer[0].as_array() ==
np.array([[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]], dtype=np.float32)
)
# Get a RaggedBatch with 2 randomly selected sequences.
buffer[np.random.permutation(4)[:2]]
Addition
You can add two RaggedBuffer
s with the +
operator if they have the same number of sequences, sequence lengths, and features. You can also add a RaggedBuffer
where all sequences have a length of 1 to a RaggedBuffer
with variable length sequences, broadcasting along each sequence.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create ragged buffer with dimensions (3, [1, 3, 2], 1)
rb3 = RaggedBufferI64(1)
rb3.push(np.array([[0]], dtype=np.int64))
rb3.push(np.array([[0], [1], [2]], dtype=np.int64))
rb3.push(np.array([[0], [5]], dtype=np.int64))
# Create ragged buffer with dimensions (3, [1, 1, 1], 1)
rb4 = RaggedBufferI64.from_array(np.array([0, 3, 10], dtype=np.int64).reshape(3, 1, 1))
# Add rb3 and rb4, broadcasting along the sequence dimension.
rb5 = rb3 + rb4
assert np.all(
rb5.as_array() == np.array([[0], [3], [4], [5], [10], [15]], dtype=np.int64)
)
Concatenation
The extend
method can be used to mutate a RaggedBuffer
by appending another RaggedBuffer
to it.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb1 = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb2 = RaggedBufferF32.from_array(np.zeros((2, 5, 3), dtype=np.float32))
rb1.extend(r2)
assert rb1.size0() == 6
Clear
The clear
method removes all elements from a RaggedBuffer
without deallocating the underlying memory.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb.clear()
assert rb.size0() == 0
License
ENN Ragged Buffer dual-licensed under Apache-2.0 and MIT.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ragged_buffer-0.2.14-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2497139d61008dbf633a8bde5c1274915a548fa91c7fe6c741d70d706f8de845 |
|
MD5 | da6b4ecfd9ce4c6e672482e0b835c7f1 |
|
BLAKE2b-256 | 8d728e9d10277c8ba2af3f8d80e53b6d08e961f2790ff86fdc58992f15c80f5b |
Hashes for ragged_buffer-0.2.14-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 630d9a571c48720f9d6ec788f4b89c0ace3bc9f28e965b256b6aa12bf58fda0b |
|
MD5 | 7b8d1bb619c815fdbd01a73cebdb4124 |
|
BLAKE2b-256 | 275d7546ca8fed9ba548ae93a38a53ecdfb65c0bb6ee114e9e12ae8633a8bed1 |
Hashes for ragged_buffer-0.2.14-cp39-cp39-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 35a4ae45c7d47563a58f9524350d441549a83452377837637be9e320408f65aa |
|
MD5 | c78ea55f3b5696b5d5f017d9a594b4d1 |
|
BLAKE2b-256 | 26922b87c20489b4dae4abdea528856a0e0d7b99ab7ae46f2b11767ec6ab1bc7 |
Hashes for ragged_buffer-0.2.14-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9e894919653175d8d1ab52b75424341d1894d7fd67a90eb9e080f8d4a1dd2cf4 |
|
MD5 | 6e95ceb2eb61e1761f4d3d64fb2a231f |
|
BLAKE2b-256 | eb99872c545ac279dce7117607a1b8c43603c3f59693484baf917885cd4b7841 |
Hashes for ragged_buffer-0.2.14-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c3f3eb315579b399d11941cf49f49800379a001cd3c29f585b78a1e89a2a4602 |
|
MD5 | 531e1fc766958667641bda98e43e3336 |
|
BLAKE2b-256 | e8e4a56488777e53c7c349debd6603e89fd85602904286eee3e63aa1ae52b5be |
Hashes for ragged_buffer-0.2.14-cp38-cp38-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1d170f1bff02c2b714a6e774824da2fd5be5cd30ac850b535807a014a6d1a92a |
|
MD5 | aa23999a08dd7e3bb624d889dd6aedb7 |
|
BLAKE2b-256 | 70999d5b192f69bb1516be7a34cdb376865d624c8f5c212d54b09f11b1d5bba1 |
Hashes for ragged_buffer-0.2.14-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9259792108fffebb262dede94534ddd79f323ce9eba1caef91c132e6ccacad33 |
|
MD5 | c19a7dc74c2a5c2f84b0e8d76de52da4 |
|
BLAKE2b-256 | d505b93b0accb3cdbf9157a4f2fc3d82daf103defff2e31f34c218248e65b158 |
Hashes for ragged_buffer-0.2.14-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 682ff34f199350b684e331f00f384898fb258fe80cbcc58b1654b706fdd69c85 |
|
MD5 | ebf18c9b9deb95c6d20d39dbc9ed558d |
|
BLAKE2b-256 | c877e8d5e475d4f7a12e78f0f3579b28d1403345c1b3ee3830405ab277f5043c |
Hashes for ragged_buffer-0.2.14-cp37-cp37m-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 540b73b3c5132cb0203be06f835c5520dbede3f7fe68d30a1043adc68d7941b8 |
|
MD5 | ac65fc1b0b0454b9ab1f7b0acb0782bc |
|
BLAKE2b-256 | 16c52e6279d9533dc61a7460f7fabc28af8bb6712e9c29eab654709fdd1fdeb8 |
Hashes for ragged_buffer-0.2.14-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4947a3c41127e8c14d3f7aa09e7339df9fc5f8f9a3113656c081d5ff30636223 |
|
MD5 | 8a3589f8b747fe43991c84ede9c9f30e |
|
BLAKE2b-256 | 24f5ddb5e6bcb3d31d806b1c0eb854ac804b548bbb14146dbe8ce8483d1f0530 |
Hashes for ragged_buffer-0.2.14-cp36-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f7e3523f20f6596988a4964c81070338ff0ca2d42ff57d644f175c674d48a1c7 |
|
MD5 | bb1f10315c92efa4110fab92804a2aa4 |
|
BLAKE2b-256 | 047d7ffb2f5039eb8b4c6b8b7f097edeadb8dcde0fc1fe35f7ad914d79517057 |