fine_tuning_dataset_preparation

Comprehensive toolkit for turning codebases and Markdown documentation into fine-tuning datasets for LLMs. It ships a code pipeline (tree-sitter extraction, instruction generation, optional paraphrasing), a document pipeline (Markdown segmentation, exhaustive Q/A generation, multi-file support), shared exporters for Gemini/OpenAI/Q/A JSONL, and configurable prompts. Tests run on pytest to keep changes safe.

Installation

pip install -e .

Python 3.10+ is recommended. Before running pipelines, set your LLM credentials:

• GOOGLE_API_KEY for Gemini (default)
• or OPENAI_API_KEY / ANTHROPIC_API_KEY if you switch providers

Code pipeline

Create instruction datasets from a repository (single or multiple projects).

CLI (see examples/run_code_pipeline.py):

python examples/run_code_pipeline.py

Programmatic use:

from fine_tuning_dataset_preparation.code_dataset import PromptConfig
from fine_tuning_dataset_preparation.code_dataset.pipeline import code_pipeline

code_pipeline(
    project_path="path/to/repo",
    multi_project=True,                # treat subfolders as projects
    dataset_path="dataset.jsonl",
    llm_provider="gemini",
    model_name="gemini-2.5-flash",
    instruction_concurrency=8,
    instruction_temperature=0.7,
    prompt_config=PromptConfig(
        instruction_hint="Keep instructions concise and actionable.",
        paraphrase_hint="Return one alternative phrasing.",
    ),
    paraphrase_variations=1,           # optional paraphrasing
    paraphrase_temperature=0.9,
    exports=[
        {"target": "gemini", "output_path": "gemini_dataset.jsonl", "options": {"jsonl": True}},
        {"target": "openai", "output_path": "openai_dataset.jsonl", "options": {"jsonl": True}},
    ],
)

Key arguments: project_path, multi_project, instruction_concurrency, instruction_temperature, prompt_config, optional paraphrase_*, and exports with targets gemini or openai.

Document pipeline

Generate Q/A datasets from Markdown. You can point to a single file, a directory, or a list of files.

CLI (see examples/run_document_pipeline.py):

python examples/run_document_pipeline.py

Programmatic use:

from fine_tuning_dataset_preparation.document_dataset import run_document_pipeline
from fine_tuning_dataset_preparation.document_dataset.dataset import DocumentPromptConfig

run_document_pipeline(
    markdown_dir="docs",               # or markdown_path="file.md" or markdown_paths=["a.md", "b.md"]
    output_path="document_dataset.json",
    min_total_pairs=1,
    llm_provider="gemini",
    model_name="gemini-2.5-pro",
    prompt_config=DocumentPromptConfig(
        system_message="Use only the provided docs; end with attribution.",
        instructions=[
            "Use only supplied documentation fragments.",
            "If missing, say it is not specified.",
            "End every answer with the attribution line.",
        ],
        attribution="Information sourced from ACME Docs © 2025.",
    ),
    exports=[
        {"target": "qa_openai", "output_path": "doc_openai.jsonl", "options": {"jsonl": True}},
        {"target": "qa_gemini", "output_path": "doc_gemini.jsonl", "options": {"jsonl": True}},
        {"target": "qa_jsonl", "output_path": "doc_pairs.jsonl"},
    ],
)

Key arguments: markdown_path or markdown_dir or markdown_paths, min_total_pairs, prompt_config for tone/attribution, and exports with Q/A targets qa_openai, qa_gemini, qa_jsonl.

Exporters

All exporters live in fine_tuning_dataset_preparation/common/exporters. Use them directly or via the pipeline exports argument.

from fine_tuning_dataset_preparation.common.exporters import export_dataset

export_dataset(
    target="qa_openai",                 # gemini | openai | qa_openai | qa_gemini | qa_jsonl
    output_path="out.jsonl",
    pairs=[{"question": "...", "answer": "..."}],  # or instruction records when using instruction targets
    options={"jsonl": True},
)

Instruction targets: gemini, openai. Q/A targets: qa_openai, qa_gemini, qa_jsonl.

Project structure

• fine_tuning_dataset_preparation/code_dataset: tree-sitter extraction, instruction generation, paraphrasing
• fine_tuning_dataset_preparation/document_dataset: Markdown ingestion, Q/A generation, prompt helpers
• fine_tuning_dataset_preparation/common: LLM utilities, text helpers, exporters
• examples/: runnable scripts for code and document pipelines, plus export helper
• tests/: pytest suite organized by domain

Testing

pytest
pytest --cov=. --cov-report=term-missing

Tips

• Match provider extras to the model you choose; the pinned requirements already bring tree-sitter grammars.
• Export your API key(s) before running examples to avoid partial or templated outputs.
• Pick models by task: fast models (Gemini 2.5 Flash, small OpenAI tiers) for bulk coverage/paraphrases; higher-fidelity models (Gemini 2.5 Pro, GPT-4.1) for final passes or sensitive Q/A. Raise temperature (0.7–0.9) for paraphrasing; keep it lower (0.2–0.5) for deterministic instruction/Q/A generation.