smashed

Colorful logo of smashed. It is the word smashed written in a playful font that vaguely looks like pipes.

Sequential MAppers for Sequences of HEterogeneous Dictionaries is a set of Python interfaces designed to apply transformations to samples in datasets, which are often implemented as sequences of dictionaries. To start, run

pip install smashed

Example of Usage

Mappers are initialized and then applied sequentially. In the following example, we create a mapper that is applied to a samples, each containing a sequence of strings. The mappers are responsible for the following operations.

Tokenize each sequence, cropping it to a maximum length if necessary.
Stride sequences together to a maximum length or number of samples.
Add padding symbols to sequences and attention masks.
Concatenate all sequences from a stride into a single sequence.

import transformers
from smashed.mappers import (
    TokenizerMapper,
    MultiSequenceStriderMapper,
    TokensSequencesPaddingMapper,
    AttentionMaskSequencePaddingMapper,
    SequencesConcatenateMapper,
)

tokenizer = transformers.AutoTokenizer.from_pretrained(
    pretrained_model_name_or_path='bert-base-uncased',
)

mappers = [
    TokenizerMapper(
        input_field='sentences',
        tokenizer=tokenizer,
        add_special_tokens=False,
        truncation=True,
        max_length=80
    ),
    MultiSequenceStriderMapper(
        max_stride_count=2,
        max_length=512,
        tokenizer=tokenizer,
        length_reference_field='input_ids'
    ),
    TokensSequencesPaddingMapper(
        tokenizer=tokenizer,
        input_field='input_ids'
    ),
    AttentionMaskSequencePaddingMapper(
        tokenizer=tokenizer,
        input_field='attention_mask'
    ),
    SequencesConcatenateMapper()
]

dataset = [
    {
        'sentences': [
            'This is a sentence.',
            'This is another sentence.',
            'Together, they make a paragraph.',
        ]
    },
    {
        'sentences': [
            'This sentence belongs to another sample',
            'Overall, the dataset is made of multiple samples.',
            'Each sample is made of multiple sentences.',
            'Samples might have a different number of sentences.',
            'And that is the story!',
        ]
    }
]

for mapper in mappers:
    dataset = mapper.map(dataset)

print(len(dataset))

# >>> 5

print(dataset[0])

# >>> {
#    'input_ids': [
#        101,
#        2023,
#        2003,
#        1037,
#        6251,
#        1012,
#        102,
#        2023,
#        2003,
#        2178,
#        6251,
#        1012,
#        102
#    ],
#    'attention_mask': [
#        1,
#        1,
#        1,
#        1,
#        1,
#        1,
#        1,
#        1,
#        1,
#        1,
#        1,
#        1,
#        1
#    ]
# }

Building a Pipeline

Mappers can also be composed into a pipeline using the >> (or <<) operator. For example, the code above can be rewritten as follows:

pipeline = TokenizerMapper(
    input_field='sentences',
    tokenizer=tokenizer,
    add_special_tokens=False,
    truncation=True,
    max_length=80
) >> MultiSequenceStriderMapper(
    max_stride_count=2,
    max_length=512,
    tokenizer=tokenizer,
    length_reference_field='input_ids'
) >> TokensSequencesPaddingMapper(
    tokenizer=tokenizer,
    input_field='input_ids'
) >> AttentionMaskSequencePaddingMapper(
    tokenizer=tokenizer,
    input_field='attention_mask'
) >> SequencesConcatenateMapper()

dataset = ...

# apply the full pipeline to the dataset
pipeline.map(dataset)

Dataset Interfaces Available

The initial version of SMASHED supports two interfaces for dataset:

interfaces.simple.Dataset: A simple dataset representation that is just a list of python dictionaries with some extra convenience methods to make it work with SMASHED. You can crate a simple dataset by passing a list of dictionaries to interfaces.simple.Dataset.
HuggingFace datasets library. SMASHED mappers work with any datasets from HuggingFace, whether it is a regular or iterable dataset.

Developing SMASHED

To contribute to SMASHED, make sure to:

(If you are not part of AI2) Fork the repository on GitHub.
Clone it locally.
Create a new branch in for the new feature.
Install development dependencies with pip install -r dev-requirements.txt.
Add your new mapper or feature.
Add unit tests.
Run tests, linting, and type checking from the root directory of the repo:
1. Style: black . (Should format for you)
2. Style: flake8 . (Should return no error)
3. Style: isort . (Should sort imports for you)
4. Static type check: mypy . (Should return no error)
5. Tests: pytest -v --color=yes tests/ (Should return no error)
Commit, push, and create a pull request.
Tag soldni to review the PR.

A note about versioning

SMASHED follows Semantic Versioning. In short, this means that the version number is MAJOR.MINOR.PATCH, where:

MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards compatible manner; adding a mapper typically falls under this category, and
PATCH version when you make backwards compatible bug fixes.

Metadata

Release files for smashed 0.21.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for smashed 0.21.5
File	Size	Uploaded
smashed-0.21.5.tar.gz	95.9 kB	Sep 22, 2023	Details

Built distribution (wheel)

Table of built distributions (wheels) for smashed 0.21.5
File	Interpreter	ABI	Platform	Reset
smashed-0.21.5-py3-none-any.whl 95.4 kB Sep 22, 2023	Python 3	none	any	Details

Total release size: 191.3 kB

Release files / smashed-0.21.5.tar.gz

Download URL	smashed-0.21.5.tar.gz
Size	95.9 kB
Tags	Source
SHA-256 checksum How to use checksums	`bf4a96a22e127a7d69378c5c6701aafa710b65a7c9b2605df4fa4a54d7896737`
BLAKE2b-256 checksum How to use checksums	`2f8b0717eb69dc7ad85b89fdf46f1a9a2c2982a8a3a82756b561e0dfedfd087f`
Upload date	Sep 22, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.10.12`

Release files / smashed-0.21.5-py3-none-any.whl

Download URL	smashed-0.21.5-py3-none-any.whl
Size	95.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`90ed1b5b0fc83d4b213cdcf417bf424e391cc7a335706da2d6c0a4e590b79d1c`
BLAKE2b-256 checksum How to use checksums	`05ced9f7390f548c52662cbdb0ad43b5935706220d6f01d5288e8fa329075bdc`
Upload date	Sep 22, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.10.12`