No project description provided
Project description
ENN Ragged Buffer
This Python package implements an efficient RaggedBuffer
datatype that is similar to
a 3D numpy array, but which allows for variable sequence length in the second
dimension. It was created primarily for use in ENN-PPO
and currently only supports a small selection of the numpy array methods.
User Guide
Install the package with pip install ragged-buffer
.
The package currently supports three RaggedBuffer
variants, RaggedBufferF32
, RaggedBufferI64
, and RaggedBufferBool
.
Creating a RaggedBuffer
There are three ways to create a RaggedBuffer
:
RaggedBufferF32(features: int)
creates an emptyRaggedBuffer
with the specified number of features.RaggedBufferF32.from_flattened(flattened: np.ndarray, lenghts: np.ndarray)
creates aRaggedBuffer
from a flattened 2D numpy array and a 1D numpy array of lengths.RaggedBufferF32.from_array
creates aRaggedBuffer
(with equal sequence lenghts) from a 3D numpy array.
Creating an empty buffer and pushing each row:
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create an empty RaggedBuffer with a feature size of 3
buffer = RaggedBufferF32(3)
# Push sequences with 3, 5, 0, and 1 elements
buffer.push(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]], dtype=np.float32))
buffer.push(np.array([[10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24]], dtype=np.float32))
buffer.push(np.array([], dtype=np.float32)) # Alternative: `buffer.push_empty()`
buffer.push(np.array([[25, 25, 27]], dtype=np.float32))
Creating a RaggedBuffer from a flat 2D numpy array which combines the first and second dimension, and an array of sequence lengths:
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], [13, 14, 15], [16, 17, 18], [19, 20, 21], [22, 23, 24], [25, 25, 27]], dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
Creating a RaggedBuffer from a 3D numpy array (all sequences have the same length):
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
Get size
The size0
, size1
, and size2
methods return the number of sequences, the number of elements in a sequence, and the number of features respectively.
import numpy as np
from ragged_buffer import RaggedBufferF32
buffer = RaggedBufferF32.from_flattened(
np.zeros((9, 64), dtype=np.float32),
np.array([3, 5, 0, 1], dtype=np.int64))
)
# Get size of the first/batch dimension.
assert buffer.size0() == 10
# Get size of individual sequences.
assert buffer.size1(1) == 5
assert buffer.size1(2) == 0
# Get size of the last/feature dimension.
assert buffer.size2() == 64
Convert to numpy array
as_aray
converts a RaggedBuffer
to a flat 2D numpy array that combines the first and second dimension.
import numpy as np
from ragged_buffer import RaggedBufferI64
buffer = RaggedBufferI64(1)
buffer.push(np.array([[1], [1], [1]], dtype=np.int64))
buffer.push(np.array([[2], [2]], dtype=np.int64))
assert np.all(buffer.as_array(), np.array([[1], [1], [1], [2], [2]], dtype=np.int64))
Indexing
You can index a RaggedBuffer
with a single integer (returning a RaggedBuffer
with a single sequence), or with a numpy array of integers selecting/permuting multiple sequences.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create a new `RaggedBufferF32`
buffer = RaggedBufferF32.from_flattened(
np.arange(0, 40, dtype=np.float32).reshape(10, 4),
np.array([3, 5, 0, 1], dtype=np.int64)
)
# Retrieve the first sequence.
assert np.all(
buffer[0].as_array() ==
np.array([[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11]], dtype=np.float32)
)
# Get a RaggedBatch with 2 randomly selected sequences.
buffer[np.random.permutation(4)[:2]]
Addition
You can add two RaggedBuffer
s with the +
operator if they have the same number of sequences, sequence lengths, and features. You can also add a RaggedBuffer
where all sequences have a length of 1 to a RaggedBuffer
with variable length sequences, broadcasting along each sequence.
import numpy as np
from ragged_buffer import RaggedBufferF32
# Create ragged buffer with dimensions (3, [1, 3, 2], 1)
rb3 = RaggedBufferI64(1)
rb3.push(np.array([[0]], dtype=np.int64))
rb3.push(np.array([[0], [1], [2]], dtype=np.int64))
rb3.push(np.array([[0], [5]], dtype=np.int64))
# Create ragged buffer with dimensions (3, [1, 1, 1], 1)
rb4 = RaggedBufferI64.from_array(np.array([0, 3, 10], dtype=np.int64).reshape(3, 1, 1))
# Add rb3 and rb4, broadcasting along the sequence dimension.
rb5 = rb3 + rb4
assert np.all(
rb5.as_array() == np.array([[0], [3], [4], [5], [10], [15]], dtype=np.int64)
)
Concatenation
The extend
method can be used to mutate a RaggedBuffer
by appending another RaggedBuffer
to it.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb1 = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb2 = RaggedBufferF32.from_array(np.zeros((2, 5, 3), dtype=np.float32))
rb1.extend(r2)
assert rb1.size0() == 6
Clear
The clear
method removes all elements from a RaggedBuffer
without deallocating the underlying memory.
import numpy as np
from ragged_buffer import RaggedBufferF32
rb = RaggedBufferF32.from_array(np.zeros((4, 5, 3), dtype=np.float32))
rb.clear()
assert rb.size0() == 0
License
ENN Ragged Buffer dual-licensed under Apache-2.0 and MIT.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ragged_buffer-0.3.2-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 012347e0a72b461f47d609fee74c0e1f1fabb207739eea4f53839e639e06dd82 |
|
MD5 | 93cfc6941c056adef6b750f2dcd1636f |
|
BLAKE2b-256 | 0f8e259c1e5a79334cfdf9df39a386692b13d8991de211923742350bef2ea134 |
Hashes for ragged_buffer-0.3.2-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e5f95d9dbe21a361b5b243ed23b2577cfcf456c374cfcdb085e8235b7a3cdec |
|
MD5 | e78cd709414ee3b5962f8288e05fca24 |
|
BLAKE2b-256 | 9477a4244d7f0cfd0b979534cec2c65e1c8001b3a243b22b0793f3b3fa7a3bea |
Hashes for ragged_buffer-0.3.2-cp39-cp39-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7b6496bfabfdeb48e8d573c0d425dfca297e6da43e64c121a282d6d6deef5a77 |
|
MD5 | 492a28469be891dc14f94a8aa6095539 |
|
BLAKE2b-256 | 1b5b32b427349435d31c80ec8b23d1170f8df5f009dc6181c987e26840fea75c |
Hashes for ragged_buffer-0.3.2-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a97da8d7b53a9991337bc00b8310fffb80ea0f20b5cd034408bf9c2dc249019 |
|
MD5 | d6bbd8844f986987612345bdba9b1848 |
|
BLAKE2b-256 | c201390d9ca5cb437b56a87333256160751289d40f611b90d50a68fba48e68b1 |
Hashes for ragged_buffer-0.3.2-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1be06f8637533408fe94aa628cd946336df222222e98412127cc108ff5d1dba9 |
|
MD5 | d151887945db9fe8333ce2f1d937d358 |
|
BLAKE2b-256 | dab60ed5db778ea963738c0c3946b96fde6c8880cb609722f1b0b383edd7c1ec |
Hashes for ragged_buffer-0.3.2-cp38-cp38-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f0937418f658ec65292b76a0044ddac0925e7281401fd0a6eed0d09e7a9bcb0 |
|
MD5 | 0cff283ed623db99adaab5605fa23489 |
|
BLAKE2b-256 | 5f7bcba9a908548734bc23deb7faff03f6aab8205f3764fe714d1527dcdb6653 |
Hashes for ragged_buffer-0.3.2-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 90b465c50e55f9e2a27423f4cc6cc00c229fa2221f10a68ffdd424ed141b8e2a |
|
MD5 | 28d7c392c21e78ae9de1b7bf17e27a78 |
|
BLAKE2b-256 | dd91053f029e57d8cb5057295eaf83dac1e6598cbc572c9407c805c34ad6736b |
Hashes for ragged_buffer-0.3.2-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 219e635c83559a402e361580380b3b3541c591b24d99340ceb672239152daf16 |
|
MD5 | 55d8d5170cea1881b3cdc9cf41d94e82 |
|
BLAKE2b-256 | 87345e67e5140c16549992a086162657237008d55b9dbccb1423b29f9dd81be0 |
Hashes for ragged_buffer-0.3.2-cp37-cp37m-manylinux_2_24_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09ed7580edc43dc8375dc3ec53352e8b643c9971c7c5201353b2aa27a92f959f |
|
MD5 | 4bba867c7a33bf2e246bb5fc4fe1f555 |
|
BLAKE2b-256 | e757ad868bc478a6f01b15b325d758a427a54a0a1683372cca543ebe78a2106e |
Hashes for ragged_buffer-0.3.2-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3fdc5c09bdad1cd499d6f1916d49637a824022f0357093d161736c71bdf6ee97 |
|
MD5 | 04e90aee944b7f2c8b9a3b927a20ecb1 |
|
BLAKE2b-256 | ae438ca900d680858031326bfb85497c5c9df4f2feb3e926346547d0ab0da0fa |
Hashes for ragged_buffer-0.3.2-cp36-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b55f48c3f13a17d10253588a527b9e08ad89618dab6f93a69be45d12c2fe8dfe |
|
MD5 | 3a2c8b3df16e6faa7a3373bce105746d |
|
BLAKE2b-256 | ffcf1df3196671cbd6a8e6cba33bc84eaaa612fab73fc06819b5261a63c1aaf6 |