convmerge

Convert Alpaca, ShareGPT, tool-calling, preference, and mixed chat datasets into one training-ready JSONL.
Fetch from HuggingFace or GitHub, normalize messy Parquet / JSON / JSONL, convert 30+ popular dataset layouts into messages (SFT) or {prompt, chosen, rejected} (DPO) rows, weighted-mix multiple domain sources, deduplicate, filter, and decontaminate — one command each, or the whole pipeline from a reproducible recipe.

convmerge is a data-preparation CLI and library for LLM supervised fine-tuning (SFT). It takes heterogeneous instruction-tuning datasets — Alpaca, ShareGPT, raw chat JSONL, template-rendered text columns, tool-calling data (OpenAI, Hermes, Glaive, xLAM, LLaMA-Factory), preference data (UltraFeedback, HH-RLHF, Orca DPO pairs, Arena), Parquet dumps — and produces a single clean JSONL file in the standard messages format (or alpaca shape, or DPO preference pairs) that TRL, LLaMA-Factory, axolotl, and other fine-tuning frameworks consume directly.

It is intentionally scoped to the pre-training-loop step: no model loading, no inference, no labeling, no training orchestration. See Out of scope below.

Repository: github.com/snowmuffin/convmerge
Status: pre-1.0; APIs and CLI may change between minor versions until 1.0.

Install

pip install convmerge                    # core: convert, dedupe, filter, decontam; normalize for .json/.jsonl
pip install "convmerge[all]"             # full CLI: fetch (HF+GitHub), parquet, YAML presets, tokens, near dedupe

Granular extras:

pip install "convmerge[fetch]"           # YAML manifests + GitHub (PyYAML)
pip install "convmerge[fetch-all]"       # fetch + HuggingFace (``datasets``)
pip install "convmerge[fetch-hf]"        # same dependencies as ``fetch-all`` (backward-compatible name)
pip install "convmerge[parquet]"         # Parquet input for ``normalize``
pip install "convmerge[preset]"          # YAML convert presets (`--preset`, `preset validate`)
pip install "convmerge[tokens]"          # `tokens`: lengths + chat-template checks (transformers, no PyTorch)
pip install "convmerge[quality]"         # `dedupe --near`: near-duplicate removal (datasketch)

Or from a clone:

git clone https://github.com/snowmuffin/convmerge.git
cd convmerge
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,all]"

From datasets to a training run

Each guide is one recipe — fetch, convert, mix, dedupe, a tokens filter for the target model's chat template and length, and a train/validation split — plus the trainer config. All three were run end to end (SFT and DPO). Korean data: docs/guides/korean.md (한국어 가이드).

Tested datasets

Every dataset below is converted in the test suite from a record with its exact layout, and python scripts/datasets.py check runs the same conversions on real rows from the Hub. The Datasets workflow runs that check weekly and on pull requests that touch the catalog or the adapters. For example:

convmerge fetch hf://NousResearch/hermes-function-calling-v1 --config func_calling --max-rows 1000 -o raw
convmerge convert -i raw/NousResearch_hermes-function-calling-v1.jsonl -o tools.jsonl --from auto --format messages

convmerge fetch hf://HuggingFaceH4/ultrafeedback_binarized --split train_prefs -o raw
convmerge convert -i raw/HuggingFaceH4_ultrafeedback_binarized.jsonl -o dpo.jsonl --from auto --format preference

A preference dataset can also feed SFT: --format messages --preference chosen trains on the chosen answers.

Commands

1. fetch — pull raw data from HF + GitHub via a YAML manifest

HuggingFace entries delegate to datasets.load_dataset(...).to_json(...), i.e. the output is a JSONL dump of the selected split. GitHub entries support a single raw URL, recursive Trees API fetch with an extension filter, or git clone (with optional git lfs pull). fetch is a reproducible downloader, not a mirror of HuggingFace's Arrow cache.

# manifest.yaml
version: 1
defaults: { output_root: ./raw, resume: true }
auth:     { hf_token_env: HF_TOKEN, github_token_env: GITHUB_TOKEN }
datasets:
  - { name: alpaca-ko, hf: MarkrAI/KoCommercial-Dataset, split: train }
  - { name: orca-raw,
      url: https://raw.githubusercontent.com/org/repo/main/data/train.jsonl }
  - { name: repo-tree,
      url: https://github.com/org/example-repo, ext: [".jsonl"] }
  - { name: big-lfs,
      url: https://github.com/org/big-lfs-repo, mode: clone, lfs: true }

convmerge fetch manifest.yaml -o ./raw
# or one-shot shortcuts:
convmerge fetch hf://org/dataset -o ./raw --split train
convmerge fetch https://github.com/org/repo -o ./raw --ext .jsonl

Tokens resolve in order CLI flag → file → env var, and are redacted from logs. See docs/fetch.md for the full schema.

2. normalize — reshape parquet / messy JSON into clean JSONL

convmerge normalize -i ./raw -o ./jsonl

Handles parquet (streamed via pyarrow), top-level JSON arrays, concatenated single-line JSON ({...}{...}{...}), JSONL whose lines are arrays (wrapped as {"conversation": [...]}), and already-valid JSONL. A directory input is walked recursively and mirrored under the output directory.

3. convert — adapter + emitter pipeline

convmerge convert -i ./jsonl/alpaca.jsonl -o ./train/alpaca.messages.jsonl \
  --from alpaca --format messages

convmerge convert -i ./jsonl/mixed.jsonl -o ./train/mixed.messages.jsonl \
  --from auto --format messages         # auto-detecting chat adapter

# Optional: YAML preset (pip install "convmerge[preset]")
convmerge preset init -o convert_preset.yaml
convmerge preset validate convert_preset.yaml
convmerge convert -i ./jsonl/mixed.jsonl -o ./out.jsonl --preset convert_preset.yaml

Adapters: alpaca, sharegpt, chat (alias auto), and map for any other layout (dotted paths such as dialogue[].utterances[], no code; see docs/format.md).
Output formats: messages, alpaca, preference (DPO pairs), sharegpt and sharegpt-preference (LLaMA-Factory / Unsloth).

Tool calling and multimodal: OpenAI tool_calls / tools, LLaMA-Factory function_call / observation turns, Hermes <tool_call> tags, Glaive and xLAM layouts all come out as standard tool_calls / tools, and image / audio / video references (image_url parts, images columns with <image> tokens) are preserved in the messages output. Media is kept by reference only — convmerge never downloads or decodes it. --from sharegpt keeps whole conversations since 0.6.0 (turn_mode: pairs restores the old split). See docs/format.md.

Preference (DPO / reward) datasets: --format preference writes {prompt, chosen, rejected} pairs for TRL's DPOTrainer; --preference chosen trains SFT on the chosen answer instead (UltraFeedback, HH-RLHF, Orca DPO pairs, LLaMA-Factory ranking, TRL, and Chatbot Arena shapes).

Rendered text columns (ChatML, Llama 2/3, Gemma, ### Human:, HH, the Alpaca prompt) are split back into turns by --from auto.

Reasoning data: traces stored inline (<think>...</think>) or in a reasoning_content / thinking field are kept; --reasoning moves them to the field your model's chat template reads (reasoning_content for Qwen3 / DeepSeek, thinking for gpt-oss), inline, or drops them. See docs/guides/reasoning.md.

Strict chat templates: --system fold (no system role), --merge-consecutive ("roles must alternate"), --reasoning-turns last / --split-turns (templates that render reasoning only after the last user turn). Tool-call turns get "content": "", which every common template accepts. Symptom → fix table: docs/guides/troubleshooting.md.

Large files: --workers N converts with N processes (same output and stats as a single process; ~3.8x faster with 4 workers in our benchmark).

Every example is validated before it is written; ones with no user turn, empty messages, or unmatched tool results are dropped and counted by reason (--on-invalid keep|fail to change that, --report PATH for details, convmerge validate -i FILE to check an existing file). See docs/format.md.

Presets and team-specific tuning: docs/custom_presets.md.

chat / auto is a heuristic adapter: it inspects the keys of each input record (messages, conversation(s), text, conversation_a/_b, instruction/input/output, …) and routes to the right branch with a configurable role map. For unusual schemas, pin an explicit adapter (alpaca, sharegpt) or override keys programmatically — see docs/format.md.

4. mix — domain-controlled weighted merge

# Inline weights
convmerge mix \
  -i ./train/code.messages.jsonl:0.4 \
     ./train/math.messages.jsonl:0.3 \
     ./train/general.messages.jsonl:0.3 \
  -o ./train/mixed.jsonl --total 100000 --seed 42

# Or via a config file (YAML requires convmerge[preset])
convmerge mix mix.yaml

# mix.yaml
seed: 42
total: 100000
output: ./train/mixed.jsonl
sources:
  - { path: ./train/code.messages.jsonl,    weight: 0.4 }
  - { path: ./train/math.messages.jsonl,    weight: 0.3 }
  - { path: ./train/general.messages.jsonl, weight: 0.3 }

Weights are normalized automatically and need not sum to 1.0. When a source has fewer records than its allocation it is clipped; pass --oversample to repeat records instead. mix streams: it reads each source twice and shuffles through temporary files next to the output, so memory stays small even when merging multi-GB sources (--sampler v1 reproduces mixes made before 0.7). A sidecar .mix.json is written alongside the output recording the exact seed, weights, and per-source counts for full reproducibility. Omit --total to merge all records from every source.

5. dedupe / filter / decontam / tokens / split — ready for training

convmerge dedupe -i ./train/mixed.jsonl -o ./train/mixed.dedup.jsonl
# --near also drops near-copies (MinHash LSH; needs convmerge[quality])

# Rule-based quality filter: refusals and "as an AI language model" disclaimers
# (English, Korean), empty answers, looping answers or reasoning traces, and
# preference pairs whose answers are the same or whose rejected side is empty.
# Optional: --enable slop, --min-chars/--max-chars, --min-script hangul=0.3.
# Prints a JSON report with example rows per rule; -o keeps the rows that pass.
convmerge filter -i ./train/mixed.dedup.jsonl -o ./train/clean.jsonl --rejects rejected.jsonl

# Drop rows that share a 13-word n-gram with a benchmark (prompts; --check all
# for answers too). hf: sources need convmerge[fetch-all].
convmerge decontam -i ./train/clean.jsonl --against hf:openai/gsm8k:main \
  --against hf:cais/mmlu:all -o ./train/decontam.jsonl

# Render every row with the model's chat template: length percentiles, rows the
# template rejects (with line numbers), double-encoded tool arguments, answers
# that start beyond --max-tokens, answers not followed by a stop token, reasoning
# the template drops, and {% generation %} support -- with the convert flag that
# fixes each ("hints"). -o keeps the rows that render and fit.
# Needs convmerge[tokens] (transformers, no PyTorch).
convmerge tokens -i ./train/decontam.jsonl --tokenizer Qwen/Qwen2.5-7B-Instruct \
  --max-tokens 4096 -o ./train/fit.jsonl

# Train/validation split by content hash: reproducible, order-independent,
# duplicates never straddle the two sides. --val-rows N for an exact count.
convmerge split -i ./train/fit.jsonl -o ./train/train.jsonl --val 0.02   # + train.val.jsonl

# LLaMA-Factory: add the dataset_info.json entry for a --format sharegpt file.
convmerge llamafactory-info -i ./data/sft.jsonl --name my_sft --info ./data/dataset_info.json

# axolotl: the datasets: block whose field names match a converted file.
convmerge axolotl-config -i ./train/train.jsonl --val ./train/train.val.jsonl

# Single-turn vs multi-turn report (and split)
convmerge turns -i ./train/train.jsonl --single-out single.jsonl --multi-out multi.jsonl

See docs/quality.md for the filter rules, decontam, and near dedupe, docs/format.md for adapter / emitter schemas, docs/fetch.md for manifest details, and docs/api.md for the Python API and writing plugins (custom adapters / output formats via entry points). docs/stability.md lists what stays compatible across releases (API, CLI flags and exit codes, file formats).

6. run — the whole pipeline from one recipe

# recipe.yaml
version: 1
output: train/mixed.jsonl
sources:
  alpaca: { path: data/alpaca_data.json, convert: { from: alpaca } }
  tools:
    fetch: { url: https://raw.githubusercontent.com/org/repo/main/tools.jsonl }
    convert: { from: sharegpt }
mix: { total: 100000, seed: 42, weights: { alpaca: 0.7, tools: 0.3 } }
dedupe: true
filter: true                                                         # optional
decontam: { against: [hf:openai/gsm8k:main] }                        # optional
tokens: { tokenizer: Qwen/Qwen2.5-7B-Instruct, max_tokens: 4096 }   # optional
split: { val: 0.02 }                                                 # optional

convmerge run recipe.yaml --plan     # what would run, and why
convmerge run recipe.yaml            # fetch → normalize → convert → mix → dedupe → filter → decontam → tokens → split
convmerge run recipe.yaml --frozen   # CI: fail unless the lock file is current

Each step is the same command you would type by hand, so the result is identical. recipe.lock.json records options, convmerge version, and input/output digests; the next run repeats only the steps whose inputs or options changed. See docs/recipes.md.

Out of scope

To keep the package lean and dependency-free at its core, convmerge does not include — and has no plans to include — the following:

• Model loading / inference / training. No PyTorch, Transformers, vLLM, or similar runtime is imported by the core or any shipped extra.
• Model-based labeling or classification of samples (e.g. topic tagging, LLM-as-judge or classifier quality scores, safety classification). These are left to upstream tools or private pipelines; filter covers the deterministic, rule-based checks.
• RLHF / DPO / preference-dataset construction beyond passing through existing pairwise rows via the chat adapter's pairwise_mode.
• Training-job orchestration (SkyPilot, RunPod, Modal, K8s operators).
• Prompt templating / chat-template rendering for specific model families. Output JSONL uses the standard messages / alpaca shapes; downstream trainers apply their own template.
• Tokenizer-aware length filtering, packing, or curriculum scheduling. Those live in the training stack, not here.
• Downloading, decoding, or transforming media. Images, audio, and video are carried through as references (URLs or paths) exactly as the source gave them; fetching and preprocessing the files is the trainer's job.
• Scraping HTML pages or running browser automation. Structured JSON / JSONL / Parquet inputs only.

If any of these are important to your workflow, wire convmerge in as one step of a larger pipeline rather than expecting it to grow into those areas.

Development

See CONTRIBUTING.md for the full guide — setup, local checks, code conventions, and a walkthrough for adding a new adapter / emitter. CI runs Ruff, mypy, and pytest on Python 3.10 – 3.12.

pip install -e ".[dev,all]"
ruff check src tests
ruff format --check src tests
mypy
pytest -q

Participation in this project is governed by the Contributor Covenant Code of Conduct.

Good first PRs: new adapters / emitters for public dataset schemas, new fetch backends (GitLab / Zenodo / Kaggle), recipe examples under examples/, and docs improvements. Browse the good first issue label for concrete starting points.

PyPI release (maintainers)

Releases run from .github/workflows/publish.yml on pushing a v* tag. Publishing authenticates via the PYPI_API_TOKEN GitHub Actions secret.

1. Create an API token on pypi.org.
   • If the project already exists on PyPI, scope the token to the convmerge project (principle of least privilege).
   • For the very first upload (project not yet registered), PyPI does not allow project-scoped tokens — use Entire account scope for the first release, then rotate to a project-scoped token afterwards and revoke the original.
2. In the GitHub repo, Settings → Secrets and variables → Actions → New repository secret, add PYPI_API_TOKEN with the token value.
3. Tag and push: git tag vX.Y.Z && git push origin vX.Y.Z.

Changelog

CHANGELOG.md · upgrading: to 1.0 (check on 0.9), from 0.6, from 0.5

License

MIT