tweaktune 0.0.1a1

A Python package for syntesize datasets for training and fine-tuning AI models.

tweaktune

tweaktune is a Rust-powered, Python-facing library designed to synthesize datasets for training and fine-tuning AI models, especially LMs (Language Models).
It allows you to easily build data pipelines, generate new examples using LLM APIs, and create structured datasets from a variety of sources.

---

Features

• Flexible Data Sources:
  Supports datasets from:

  • Parquet files
  • CSV files
  • JSONL files
  • Arrow datasets
  • OpenAPI specifications (for function calling datasets)
  • Lists of tools (Python functions for function calling datasets)
  • Pydantic models (for structured output datasets)

• LLM Integration:
  Connects to any LLM API to generate synthetic text or structured JSON.

• Dynamic Prompting:
  Supports Jinja templates for highly customizable prompts.

• Parallel Processing:
  Configure multiple workers to run your pipeline steps in parallel.

• Easy Pipeline Building:
  Compose steps like sampling, generating, writing, or debugging into a seamless pipeline.

---

Quick Example

Here's how you can build a dataset from a Parquet file and synthesize new data using an LLM API:

from tweaktune import Pipeline
import os

persona_template = """
Na podstawie poniższego fragmentu tekstu opisz personę która jest z nim związana.
Dla opisywanej osoby wymyśl fikcyjne imię i nazwisko.
Napisz dwa zdania na temat tej osoby, opis zwróć w formacie json, nie dodawaj nic więcej:
{"persona":"opis osoby"}

---
FRAGMENT TEKSTU:

{{article[0].text}}
"""

url = "http://localhost:8000/"
api_key = os.environ["API_KEY"]
model = "model"

p = Pipeline()\
    .with_workers(5)\
    .with_parquet_dataset("web_articles", "../../datasets/articles.pq")\
    .with_openai_llm("bielik", url, api_key, model)\
    .with_template("persona", persona_template)\
    .with_template("output", """{"persona": {{persona|jstr}} }""")\
    .iter_range(10000)\
        .sample(dataset="web_articles", size=1, output="article")\
        .generate_json(template="persona", llm="bielik", output="persona", json_path="persona")\
        .write_jsonl(path="../../datasets/personas.jsonl", template="output")\
    .run()

---

Pipeline Steps

You can easily chain together multiple steps:

• sample() – sample items from a dataset
• read() – read entire dataset
• generate_text() – generate text using an LLM
• generate_json() – generate JSON output and extract a specific field
• write_jsonl() – write output to a JSONL file
• write_csv() – write output to a CSV file
• print() – print outputs
• debug() – enable detailed debugging
• log() – set log level
• python step – add custom Python-defined step classes

---

Why tweaktune?

• Build synthetic datasets faster for fine-tuning models.
• Automate text, JSON, or structured data generation.
• Stay flexible: plug your own LLM API or use existing OpenAI-compatible ones.
• Rust speed, Python usability.

---

📦 Installation

pip install tweaktune