lm-datasets 0.0.1

A collection of datasets for language model training including scripts for downloading, preprocesssing, and sampling.

lm-datasets

lm-datasets is a collection of datasets for language model training including scripts for downloading, preprocesssing, and sampling.

Installation

pip install lm-datasets

Usage

To download and extract the plain-text of one or more datasets, run the following command:

python -m lm_datasets.extract_plaintext $DATASET_ID $OUTPUT_DIR

By default, output is saved as JSONL files. To change the output format, you can use the --output_format argument as below:

python -m lm_datasets.extract_plaintext $DATASET_ID $OUTPUT_DIR --output_format parquet  --output_compression zstd

Available datasets

A list or table with all available datasets can be print with the follow command:

python -m lm_datasets.print_stats --print_output md

Dataset viewer

We provide a Web-based application through streamlit to browse all datasets and their contained text content. To start the app, run the following command:

streamlit viewer/app.py $RAW_DATASETS_DIR $PROCESSED_DATASET_DIR

Development & Contributions

Setup environment

git clone [e-mail]:malteos/lm-datasets.git
cd lm-datasets

conda create -n lm-datasets python=3.10
conda activate lm-datasets

pip install -r requirements.txt

Install the pre-commit hooks

This repository uses git hooks to validate code quality and formatting.

pre-commit install
git config --bool flake8.strict true  # Makes the commit fail if flake8 reports an error

To run the hooks:

pre-commit run --all-files

Testing

The tests can be executed with:

pytest --doctest-modules --cov-report term --cov=lm_datasets

License

Apache 2.0