Dead simple standard for storing/loading datasets as lines of text. Supports zstd compression.
Project description
Lines Dataset
Dead simple standard for storing/loading datasets as lines of text. Supports zstd compression.
pip install lines-dataset
Format
A dataset folder looks like this:
my-dataset/
meta.json
my-inputs.txt
my-compressed-labels.txt.zst
other-labels.txt.zst
...
meta.json
:
{
"lines_dataset": {
"inputs": {
"file": "my-inputs.txt",
"num_lines": 3000 // optionally specify the number of lines
},
"labels": {
"file": "my-compressed-labels.txt.zst",
"compression": "zstd",
"num_lines": 3000
},
"other-labels": {
"file": "other-labels.txt.zst",
"compression": "zstd",
"num_lines": 2000 // not all files need to have the same number of lines, as long as samples match line by line. The shortest file will determine the length of the dataset.
},
},
// you can add other stuff if you want to
}
Usage
import lines_dataset as lds
ds = lds.Dataset.read('path/to/my-dataset')
num_samples = ds.len('inputs', 'labels') # int | None
for x in ds.samples('inputs', 'labels'):
x['inputs'] # "the first line of inputs.txt\n"
x['labels'] # "the decompressed first line of labels.txt.zst\n"
A common convenience to use is:
import lines_dataset as lds
datasets = lds.glob('path/to/datasets/*') # list[lds.Dataset]
for x in lds.chain(datasets, 'inputs', 'labels'):
...
And that's it! Simple.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
lines_dataset-0.2.7.tar.gz
(3.6 kB
view hashes)
Built Distribution
Close
Hashes for lines_dataset-0.2.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c749a289797aa0e8dd921332b1dd23f0ff8c44f7ee835b2f402800659fffdad |
|
MD5 | 9f1cbd88f61916ff10e4bc2f828bfad8 |
|
BLAKE2b-256 | 44d2b54d654a78269c70e0981e14fdaa7d38edf2c78fe9fd43644e1002568243 |