Instructify 📝
Instructify is a Python library designed to convert CSV files or Hugging Face datasets into Hugging Face Dataset objects, specifically formatted for fine-tuning large language models (LLMs). Inspired by the instruction-based dataset approach described in OpenAI's InstructGPT paper (2203.02155), this package helps prepare your data for instruction-based tasks using a chat-like format.
Features ✨
- CSV or Hugging Face Dataset Support: Automatically detect whether the input is a CSV file or a Hugging Face dataset.
- Customizable Message Formatting: Supports user, assistant, and system messages with flexible column names.
- Tokenizer Integration: Automatically integrates with a pre-trained tokenizer to format messages.
- Custom Templates: Apply a custom template or use the tokenizer's default chat format.
- Tokenizer Visualization: You can use the tool below to understand how a piece of text might be tokenized by a language model, and the total count of tokens in that piece of text.
- Easy Fine-Tuning Preparation: Prepares data for instruction tuning, similar to the InstructGPT format.
Installation 📦
pip install instructify
Usage 🚀
CSV Input
import pandas as pd
from instructify import to_train_dataset
# Example custom template
custom_template = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
{}
### Input:
{}
### Response:
{}"""
# Example data
data = {
"input": ["When was the Library of Alexandria burned down?", "What is the capital of France?"],
"output": ["I-I think that was in 48 BC, b-but I'm not sure.", "The capital of France is Paris."],
"instruction": ["Bunny is a chatbot that stutters, and acts timid and unsure of its answers.", None]
}
# Convert data to CSV
df = pd.DataFrame(data)
df.to_csv("data.csv", index=False)
# Generate Hugging Face dataset for fine-tuning
train_dataset = to_train_dataset("data.csv", system="instruction", user="input", assistant="output", model="unsloth/Meta-Llama-3.1-8B-Instruct", custom_template=custom_template)
# Inspect the formatted dataset
print(train_dataset["text"])
Hugging Face Dataset Input
from instructify import to_train_dataset
# Example custom template
alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
{}
### Input:
{}
### Response:
{}"""
# Using a Hugging Face dataset
train_dataset = to_train_dataset("yahma/alpaca-cleaned", system="instruction", user="input", assistant="output", model="unsloth/Meta-Llama-3.1-8B-Instruct", custom_template=alpaca_prompt)
# Inspect the formatted dataset
print(train_dataset["text"])
Tokenizer Visualization 👁️
In addition to converting datasets, you can now visualize how different tokenizers process chat messages. The visualization displays the tokenized text with the following special symbols:
- 🤜 (Right-Facing Fist): Represents spaces between words.
- 💧 (Droplet): Represents newline characters.
- 💔 (Broken Heart): Marks token boundaries.
from instructify import compare_tokenizers
# Compare tokenizers from different models
compare_tokenizers(["unsloth/Meta-Llama-3.1-8B-Instruct", "unsloth/gemma-2-9b-it"])
This will help you understand how a piece of text might be tokenized by a language model, with the total count of tokens displayed.
Output Example 📄
The function formats CSV files or Hugging Face datasets into a structured template ready for fine-tuning:
| instruction | input | output |
|---|---|---|
| Bunny is a chatbot that stutters, and acts timid and unsure of its answers. | When was the Library of Alexandria burned down? | I-I think that was in 48 BC, b-but I'm not sure. |
| None | What is the capital of France? | The capital of France is Paris. |
Default Output Format
The train_dataset["text"] will output the following instruction-style dataset format when using the default tokenizer template:
[
"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 Jul 2024\n\nBunny is a chatbot that stutters, and acts timid and unsure of its answers.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhen was the Library of Alexandria burned down?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nI-I think that was in 48 BC, b-but I'm not sure.<|eot_id|>",
"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 Jul 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nThe capital of France is Paris.<|eot_id|>"
]
Custom Template Output
The train_dataset["text"] will output the following format when using a custom template:
[
"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\nBunny is a chatbot that stutters, and acts timid and unsure of its answers.\n\n### Input:\nWhen was the Library of Alexandria burned down?\n\n### Response:\nI-I think that was in 48 BC, b-but I'm not sure.<|eot_id|>",
"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n\n\n### Input:\nWhat is the capital of France?\n\n### Response:\nThe capital of France is Paris.<|eot_id|>"
]
Functionality Overview 🔍
to_train_dataset
This function is the core of the library, enabling both CSV and Hugging Face dataset conversion for LLM fine-tuning.
Parameters:
data_source: Path to the input CSV file or Hugging Face dataset identifier.system(optional): Column name for system messages (e.g., instructions for the model).user: Column name for user messages (default:'user').assistant: Column name for assistant messages (default:'assistant').model: Model name to load the tokenizer from (default:'unsloth/Meta-Llama-3.1-8B-Instruct').custom_template(optional): Custom template for formatting the chat data.
Returns:
Dataset: A Hugging Face Dataset, ready for LLM fine-tuning.
License ⚖️
This project is licensed under the Apache 2.0 License. See the LICENSE file for details.
Contributing 🤝
We welcome contributions! Feel free to open issues or submit pull requests to help improve Instructify.
Metadata
Release files for instructify 0.0.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| instructify-0.0.4.tar.gz | 9.8 kB | Without provenance | Details for instructify-0.0.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| instructify-0.0.4-py3-none-any.whl | Python 3 | none | any | Details for instructify-0.0.4-py3-none-any.whl |
Total release size: 20.3 kB
Release files / instructify-0.0.4.tar.gz
| Download URL | instructify-0.0.4.tar.gz |
|---|---|
| Size | 9.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c6126e827b4d62040986f896442b121351d706a9673b9de53af6bd209c752bde
|
|
BLAKE2b-256 checksum How to use checksums |
9afcce9da8947cb0b2c8a578621e146150d3e8175175ac398c55f5903e7e4e0b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.5
|
Release files / instructify-0.0.4-py3-none-any.whl
| Download URL | instructify-0.0.4-py3-none-any.whl |
|---|---|
| Size | 10.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
33db29797b5f041522dcbcb23ec3536eab91c5150dd388a5dc1ae0ad8f598111
|
|
BLAKE2b-256 checksum How to use checksums |
b54dbeb3acfd6a90e9c17fbbab59dd00b763eed0ed652febbf7c78cbc9bdf869
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.5
|
Security for instructify 0.0.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.