llm_formats

llm_formats is a Python package designed to provide data structures for various JSONL formats used for training and fine-tuning Large Language Models (LLMs). It includes predefined schemas for formats such as OpenAI prompts, chat messages, Alpaca instructions, Dolly responses, preference data, and more. This package is useful for validating, generating, and analyzing data in different LLM training formats.

Installation

To install llm_formats, use pip:

pip install llm_formats

Usage

Once installed, you can import the main function to access the format definitions:

from llm_formats import get_llm_jsonl_formats

formats = get_llm_jsonl_formats()
for fmt in formats:
    print(fmt['name'])

This will list all supported LLM data formats.

Example

Here's an example demonstrating how to retrieve and print the supported formats:

from llm_formats import get_llm_jsonl_formats

formats = get_llm_jsonl_formats()
for format in formats:
    print(f"Format: {format['name']}")
    print(f"Aliases: {', '.join(format['aliases'])}")
    print(f"Example: {format['example']}\n")

License

llm_formats is licensed under the MIT License.

Author

Eugene Evstafev
LinkedIn