squeakily

This repository is heavily inspired by BigScience’s ROOTs project and EleutherAI’s The Pile.

The overall pipeline is as follows:

In this library, we define filtering as data instances being removed from the dataset based on some criteria and cleaning as data instances being modified in some way.

Install

pip install squeakily

How to use

Using the API

First, we need to define a datasource. squeakily accepts any Dataset object from the HuggingFace Datasets library. For example, we can use the wikitext dataset:

from datasets import load_dataset

ds = load_dataset("wikitext", "wikitext-103-v1", split="train[:1%]")

We simply need to wrap the Dataset object in a dictionary, with the key being the name of the datasource and the value being the Dataset object, the filter and cleaners. For example:

from squeakily.filter import check_char_repetition, check_flagged_words
from squeakily.clean import remove_empty_lines, normalize_whitespace

datasources = [
    {
        "dataset": ds,
        "columns": ["text"],
        "filters": [check_char_repetition, check_flagged_words],
        "cleaners": [remove_empty_lines, normalize_whitespace],
    },
    # ...
]

Finally, we can apply the filters and cleaners to the datasouces using a Pipeline object:

from squeakily.core import Pipeline

pipeline = Pipeline(datasources)
pipeline.run()

[11/16/22 04:32:57] INFO     Running datasource: wikitext                                                core.py:41

                    INFO     Running filter: check_char_repetition on text                               core.py:54

                    INFO     Running filter: check_flagged_words on text                                 core.py:54

                    INFO     Running cleaner: remove_empty_lines on text                                 core.py:57

[11/16/22 04:32:59] INFO     Running cleaner: normalize_whitespace on text                               core.py:57

If you need to run a filter or cleaner at the dataset level rather than the example level, you can pass global_filters or global_cleaners to the Pipeline.run function. For example:

from squeakily.filter import minhash_dedup

pipeline.run(global_filters=[minhash_dedup])

Additionally, you can run the pipeline in a dry run mode by passing dry_run=True to the run function. This will make no modifications to the datasets’ documents, but will add additional columns to the datasets with the results of the filters and cleaners. For example, if you if you ran the pipeline with the check_char_repetition filter, you would get a new column called check_char_repetition with a float value between 0 and 1 indicating the percentage of characters that are repeated in the document.

::: {.cell}
``` {.python .cell-code}
pipeline = Pipeline(datasources)
pipeline.run(dry_run=True)
pipeline.datasources[0]["dataset"].features

:::