unitxt 1.0.20

Load any mixture of text to text data in one line of code

Unitxt is a python library for getting data fired up and set for utilization. In one line of code, it preps a dataset or mixtures-of-datasets into an input-output format for training and evaluation. We aspire to be simple, adaptable and transperant.

Unitxt builds on separation. Separation allows adding a dataset, without knowing anything about the models using it. Separation allows training without caring for preprocessing, switching models without loading the data differently and changing formats (instruction\ICL\etc.) without changing anything else.

Where to start? 🦄

Why Unitxt? 🦄

🦄 Simplicity

Everything is unitxt is simple and designed to feel natural and self explenatory.

🦄 Adaptability

Adding new datasets, loading recpepies, instructions and formattors is possible and encoureged!

🦄 Transperancy

The reosurces and formators of Unitxt are stored as shared datasets and therfore can easily reviewed by the crowed. Moreover, when assembling dataset with Unitxt it is very clear to others whats in it.