ra-pickles 0.8.4

A simple package that allows creating a pickles dataset and fast random access

RA Pickles

A simple package to create and access datasets based on python objects using pickles. A dataset will be represented by a directory. And the files in the directory contain the data. As file identifiers are randomly generated, files from two dataset folders can be added to each other create a larger dataset.

Example

First import all the packages

from ra_pickles import RandomAccessPicklesReader, RandomAccessPicklesWriter
import shutil
import os
import numpy as np

So to create a dataset in test_folder, use the following code. 100 here specifies number of dataset samples in each file. If there are more than 100 files, multiple files will be created.

fold = 'test_folder'
writer = RandomAccessPicklesWriter(100, fold)

Now entries to the dataset can be added as follows:

for i in range(30):
    d = np.ones(4) * i
    writer.add(d)
writer.close()

Here, 30 numpy arrays have been added.

To access contents in the dataset, use the following code:

reader = RandomAccessPicklesReader(fold)
print("Total", reader.get_total())

print("Trying to retreive")
for i in range(30):
    print("Retrieving",i)
    a = reader.get_element(i)
    print(i, a[0])

Multiple elements can also be retrieved in parallel. To do so, first retrieval threads must be started (and closed at the end):

reader.start_parallel_retrieval_threads(n_theads=5)
data = reader.get_multi_in_parallel([1,2,3])
reader.close_parallel_retrieval_threads()
reader.close()

Good luck!