DocSet (Document Dataset)
The project contains writer and reader utilities to store and extract documents (python dict) by a single ".ds" file as follow:
graph LR
classDef nobg fill:none,stroke:none;
classDef box fill:none,stroke:#000;
ds[(person.ds)]:::box
writer[DocSetWriter]:::box
reader[DocSetReader]:::box
doc1("{'name': 'Alice', 'age': 17, ...}<br>{'name': 'Bob', 'age': 18, ...}<br>...<br>{'name': 'Tony', 'age': 20, ...}"):::nobg
doc1_("{'name': 'Alice', 'age': 17, ...}<br>{'name': 'Bob', 'age': 18, ...}<br>...<br>{'name': 'Tony', 'age': 20, ...}"):::nobg
doc1 --> writer --> ds
ds --> reader --> doc1_
Installation
-
Install from PyPI repository.
pip3 install docset
Tutorial
Here, we give an example on a very simple machine learning task. Suppose we want to train a model to perform an image classification task, one important step is to construct a train dataset. To achieve this, we usually use a text file (e.g., csv, json) to store the "image path" and "label" information, and we store all the actual image files in another folder. This "text+folder" dataset can be fine in most of the situations, while it will suffer poor performance when the total number of samples is very large. The reason is that the file system is not good at reading/writing huge numbers of tiny files which are separately stored on the disk. So, storing the whole dataset in a single file is one possible way to solve this problem.
The following examples first show how to make a DocSet file based on the "text+folder" storage, and then show how to read or iterate this dataset.
Make a Dataset
import csv
import os
import cv2 as cv
import numpy as np
from docset import DocSet
csv_path = 'train.csv'
image_dir_path = 'images'
ds_path = 'train.ds'
with csv.DictReader(csv_path) as reader, DocSet(ds_path, 'w') as ds:
for row in reader:
image_path = os.path.join(image_dir_path, row['filename'])
image = cv.imread(image_path, cv.IMREAD_COLOR) # load the image as ndarray
doc = {
'feature': image.astype(np.float32) / 255.0, # convert the image into [0, 1] range
'label': row['label']
} # a data sample is represented by dict, ndarray can be used directly
ds.write(doc)
Read the Dataset
from docset import DocSet
ds_path = 'train.ds'
with DocSet(ds_path, 'r') as ds:
print(len(ds), 'samples')
doc = ds[0] # the sample can be access by subscript, a dict will be returned
print('feature shape', doc['feature'].shape)
print('feature dtype', doc['feature'].dtype)
print('label', doc['label'])
View the Dataset in Console
docset /path/to/the/file.ds
Integrate with Pytorch
from torch.utils.data import Dataset, DataLoader
from docset import DocSet
class MyDataset(Dataset):
def __init__(self, path):
self._impl = DocSet(path, 'r')
def __len__(self):
return len(self._impl)
def __getitem__(self, i: int):
doc = self._impl[i]
# perform any data processing
return doc
ds_path = 'train.ds'
train_loader = DataLoader(
MyDataset(ds_path),
batch_size=256,
shuffle=True,
num_workers=5,
pin_memory=True
)
for epoch in range(50):
for doc in train_loader:
feature = doc['feature']
label = doc['label']
# invoke the train function of the model
# model.train(feature, label)
Technical Details
To be added...
Metadata
Release files for docset 0.5.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| docset-0.5.5-py3-none-any.whl | Python 3 | none | any | Details for docset-0.5.5-py3-none-any.whl |
Release files / docset-0.5.5-py3-none-any.whl
| Download URL | docset-0.5.5-py3-none-any.whl |
|---|---|
| Size | 10.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
89bd7dbeb94e3781bafd6dfca5ac9fd6cd2534cbf253b13bb751bbf6036b5bca
|
|
BLAKE2b-256 checksum How to use checksums |
f89d8f6f6ef9fe38c2a18caaab6414d41518d137e518cb47c12390fb32ce9f82
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.6.3 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.63.0 CPython/3.8.10
|
Security for docset 0.5.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.