Lines Dataset

Dead simple standard for storing/loading datasets as lines of text. Supports zstd compression.

pip install lines-dataset

Format

A dataset folder looks like this:

my-dataset/
  meta.json
  my-inputs.txt
  my-compressed-labels.txt.zst
  other-labels.txt.zst
  ...

meta.json:

{
  "lines_dataset": {
    "inputs": {
      "file": "my-inputs.txt",
      "num_lines": 3000 // optionally specify the number of lines
    },
    "labels": {
      "file": "my-compressed-labels.txt.zst",
      "compression": "zstd",
      "num_lines": 3000
    },
    "other-labels": {
      "file": "other-labels.txt.zst",
      "compression": "zstd",
      "num_lines": 2000 // not all files need to have the same number of lines, as long as samples match line by line. The shortest file will determine the length of the dataset.
    },
  },
  // you can add other stuff if you want to
}

Usage

import lines_dataset as lds

ds = lds.Dataset.read('path/to/my-dataset')
num_samples = ds.len('inputs', 'labels') # int | None

for x in ds.samples('inputs', 'labels'):
  x['inputs'] # "the first line of inputs.txt\n"
  x['labels'] # "the decompressed first line of labels.txt.zst\n"

A common convenience to use is:

import lines_dataset as lds

datasets = lds.glob('path/to/datasets/*') # list[lds.Dataset]
for x in lds.chain(datasets, 'inputs', 'labels'):
  ...

And that's it! Simple.

Metadata

Release files for lines-dataset 0.2.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for lines-dataset 0.2.9
File	Size	Uploaded
lines_dataset-0.2.9.tar.gz	4.6 kB	Jul 17, 2024	Details

Built distribution (wheel)

Table of built distributions (wheels) for lines-dataset 0.2.9
File	Interpreter	ABI	Platform	Reset
lines_dataset-0.2.9-py3-none-any.whl 5.7 kB Jul 17, 2024	Python 3	none	any	Details

Total release size: 10.3 kB

Release files / lines_dataset-0.2.9.tar.gz

Download URL	lines_dataset-0.2.9.tar.gz
Size	4.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`2dfe2c30b129c87d9a17fa26a56d5e9c37bbb7491b9262e7c43f6e6c87d5c665`
BLAKE2b-256 checksum How to use checksums	`37278d34ba8fa7e6c30e367a1dcac005fb6a270bdd02eb998da37b5334fd7312`
Upload date	Jul 17, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.0 CPython/3.11.9`

Release files / lines_dataset-0.2.9-py3-none-any.whl

Download URL	lines_dataset-0.2.9-py3-none-any.whl
Size	5.7 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`f79a666028724421e8273557eee711d089503d4b333ccac3989e08dc91abff55`
BLAKE2b-256 checksum How to use checksums	`a41d40d07885c5da7979772d0c15aa5799bef4807494d3cd17086e45ff334b6f`
Upload date	Jul 17, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.0 CPython/3.11.9`