mintdim 0.1.12

Token unit pipeline for building low-padding binary token shards from JSONL datasets.

MintDim

MintDim is a token unit pipeline for building low-padding binary token shards from JSONL datasets.

The first public API focuses on one task:

stream JSONL batches → render templates → tokenize exactly → assign by token length → write binary token unit shards

Install

pip install mintdim

For local development:

pip install -e .

Quick start

from mintdim import pipeline


def build_instruction_units():
    job = (
        pipeline("unit-build")
        .source.jsonl(
            files=[
                "./data/alpaca.jsonl",
            ],
            fields=[
                ["instruction", "input", "output"],
            ],
            templates=[
                "{instruction}\n\n{input}\n\n{output}",
            ],
        )
        .tokenizer.sentencepiece(
            files=["./tokenizer/tokenizer.model"],
        )
        .units(
            sizes=[[128, 192, 256, 320]],
            build_batch=[4096],
        )
        .output.dir(
            "./artifacts/instruction_units",
            samples_per_shard=[10000],
        )
    )

    return job.run()

API flow

pipeline("unit-build")
→ source
→ tokenizer
→ units
→ output
→ run

MintDim uses a fluent pipeline API. Each step declares one part of the build job. Nothing is executed until job.run() is called.

Batch build flow:

stream JSONL sequentially
→ collect source batch
→ render text batch from template
→ hash rendered text batch
→ tokenizer.encode_batch(texts)
→ assign unit by exact token_count inside the batch
→ group records by unit_size
→ writer[unit].write_batch(token_ids)
→ write_many sample/UNK/duplicate index rows
→ update stats and histogram in batch

Path rules

All relative paths are resolved from the current working directory.

Example:

.source.jsonl(
    files=["./data/train.jsonl"],
    fields=[["text"]],
    templates=["{text}"],
)

If the script is run from the project root, MintDim resolves the file as:

./data/train.jsonl

MintDim does not scan the repository or search the filesystem automatically. Users must provide explicit paths.

JSONL source

.source.jsonl(
    files=["./data/train.jsonl"],
    fields=[["text"]],
    templates=["{text}"],
)

Three arguments are required:

files      → list of JSONL paths
fields     → list-of-list; fields[i] is the JSONL keys MintDim reads from files[i]
templates  → list of user-defined templates; templates[i] formats samples of files[i]

Multiple files are supported:

.source.jsonl(
    files=[
        "./data/alpaca.jsonl",
        "./data/sharegpt.jsonl",
    ],
    fields=[
        ["instruction", "output"],
        ["prompt", "response"],
    ],
    templates=[
        "{instruction}\n\n{output}",
        "{prompt}\n{response}",
    ],
)

Each JSONL sample MUST contain every key listed in fields[i]. Extra JSONL keys are allowed and ignored.

Templates

Templates are user-defined strings. MintDim does not ship preset templates (no chatml, alpaca, etc.). The user controls every character.

Template syntax:

- placeholders use {field_name}
- placeholder names must match fields[i] exactly as a set
- any other character is treated as a literal (\n, \t, custom separators, etc.)
- placeholders are substituted by the JSONL sample's field values

Example:

templates=[
    "{instruction}\n\n{input}\n\n{output}",
]

Rules:

1. set(placeholders in templates[i]) == set(fields[i])
2. JSONL samples must contain every declared field in fields[i]
3. Declared fields that do not appear in the template are invalid
4. Static parts of templates[i] (every character outside placeholders)
   must encode cleanly with the tokenizer (no UNK)
5. Template rendering is a pure substitution; no extra tokens are injected

A user may use any character their tokenizer supports. If the tokenizer encodes \t, then "{a}\t{b}" is valid. If not, MintDim fails pre-flight.

Invalid example:

fields=[
    ["prompt", "answer", "separator", "source"],
]
templates=[
    "{prompt}{separator}{answer}",
]

source is declared but not consumed by the template, so MintDim raises TemplateFieldMismatchError.

Tokenizer

V1 supports:

- SentencePiece tokenizer: .model
- Hugging Face tokenizer: .json

Each tokenizer step takes a files list of paths — one path shared across all input files, or one path per input file. The user declares only the tokenizer file path. MintDim loads unk_id and pad_id from the tokenizer library itself.

SentencePiece:

.tokenizer.sentencepiece(
    files=["./tokenizer/tokenizer.model"],
)

Hugging Face tokenizer JSON:

.tokenizer.hf_json(
    files=["./tokenizer/tokenizer.json"],
)

Multiple tokenizer files (one per input file):

.tokenizer.sentencepiece(
    files=[
        "./tokenizer/alpaca.model",
        "./tokenizer/sharegpt.model",
    ],
)

Required argument:

files → list of tokenizer file paths

Validation rules:

- each file path is a string and exists on disk
- the tokenizer library exposes a valid UNK id
- the tokenizer library exposes a valid PAD id

Why UNK and PAD are required:

UNK
→ used for tracking unknown tokens and dataset integrity statistics

PAD
→ used to pad samples to assigned unit size for fixed-width binary shard layout

The tokenizer is never modified by MintDim. UNK and PAD ids come from the loaded tokenizer object, not from user-declared values or MintDim-side vocab parsing.

Tokenizer contract

MintDim treats the tokenizer file as the single source of truth for vocabulary, normalization, encoding behavior, and special-token ids. The pipeline does not maintain its own token whitelist, preset template, or hand-curated character set.

What MintDim does:

- load the tokenizer file you point at (.model for SentencePiece, .json for HF)
- delegate encoding to that tokenizer's vocabulary and model as-is
- read unk_id and pad_id directly from the tokenizer library
- treat every character outside {placeholders} in a template as a literal
- accept any template literal whose tokenization does not produce UNK
- record samples whose field values produce UNK in unk_index.jsonl

What MintDim does not do:

- prepend or append BOS / EOS / CLS / SEP or any other control symbols
- inject newlines, separators, role tokens, or chat wrappers around fields
- maintain a built-in allow-list of characters that templates may use
- ship preset templates such as chatml, alpaca, or llama-style wrappers
- shrink, truncate, remap, or reinterpret the tokenizer's vocabulary
- modify the tokenizer file or its in-memory state

What this means for templates:

- "{a}\n\n{b}", "<s>{prompt}</s>{answer}", "{q}\t{a}",
  emoji separators, CJK punctuation, and custom delimiters are valid
  as long as your tokenizer encodes the literal characters without UNK
- literal characters between placeholders are tokenized by the same
  tokenizer that runs on field values; there is no separate encoder,
  normalizer, sanitizer, or escape layer

Tokenization is delegated entirely to the tokenizer library:

SentencePiece: SentencePieceProcessor.encode(text, out_type=int)
Hugging Face : Tokenizer.encode_batch(texts, add_special_tokens=False)

If you want a <bos> marker at the start of every sample, put that marker directly in your template. MintDim will pass it to the tokenizer as literal text; whether it becomes a special token, a normal token sequence, or UNK depends entirely on your tokenizer.

Tokenizer-side normalization still applies. SentencePiece and Hugging Face tokenizers may apply NFKC, lowercasing, dummy prefix, byte-level remapping, or other normalization declared inside the tokenizer file. That normalization runs on every text MintDim hands to the tokenizer, including template literals during preflight and field values at runtime. MintDim does not change this behavior in either direction.

Units

Shared unit config:

.units(
    sizes=[[128, 192, 256, 320]],
    build_batch=[4096],
)

Mapped unit config:

.units(
    sizes=[
        [128, 192, 256, 320],
        [96, 128, 160, 192, 224, 320],
    ],
    build_batch=[
        4096,
        2048,
    ],
)

Meaning:

sizes
→ target shard unit sizes

build_batch
→ number of raw samples processed per tokenizer batch

MintDim always tokenizes samples exactly. Unit assignment is based on exact token count.

Example:

token_count = 143
sizes = [128, 192, 256, 320]
target unit = 192

If a sample's sum(segment token lengths) exceeds max(sizes), MintDim pauses the build at that sample and asks the user how to recover:

[unit-build] Sample exceeds max unit_size
  source       <file>:<line>
  sample_id    <id>
  token_count  <n>
  max(sizes)   <m>

  EN: How should this sample be handled?
      [1] truncate  drop trailing tokens to fit max(sizes)
      [2] extend    declare a new unit_size that fits this sample
      [3] skip      drop this sample (recorded in overflow_index.jsonl)
  VN: Xử lý sample này thế nào?
      [1] truncate  cắt token cuối cho vừa max(sizes)
      [2] extend    khai báo thêm unit_size đủ chứa sample
      [3] skip      bỏ qua sample (ghi vào overflow_index.jsonl)

  [1/2/3] >

Choosing extend triggers a follow-up asking for an integer new_unit_size >= token_count. After picking an action, MintDim asks the scope:

  EN: Apply this choice to:
      [1] only this sample
      [2] all subsequent overflow samples in this run
  VN: Áp dụng lựa chọn này cho:
      [1] chỉ sample này
      [2] mọi sample overflow tiếp theo trong lần build này

All three actions ask for scope. With all:

truncate / all → every subsequent overflow is truncated to max(sizes)
skip / all     → every subsequent overflow is dropped
extend / all   → every subsequent overflow auto-adds a new unit_size = token_count
                 (no further prompts; sizes grow as needed to fit each oversized sample)

The prompt can be replaced with a fixed policy via .run(on_overflow=...):

"prompt"    (default) ask interactively per overflow sample; abort if stdin is not a TTY
"abort"     raise UnitOverflowError immediately, no prompt
"truncate"  drop trailing tokens to max(sizes) for every overflow sample
"skip"      drop every overflow sample

Every overflow event (truncate / extend / skip) writes one row to overflow_index.jsonl for audit.

MintDim V1 does not:

• continue partial builds
• cache build progress

UNK handling

MintDim distinguishes two kinds of UNK and treats them differently.

Tier 1 — template/tokenizer contract

Before any data is processed, every literal segment of every template is tokenized once. If any literal produces UNK, MintDim prompts the user before building:

[unit-build] Template UNK detected
  file index <i>
  literal segments with UNK <n>

  EN: Static-text tokens will be UNK in every encoded sample.
      Continue building?
  VN: Token của static-text sẽ là UNK ở mọi sample.
      Tiếp tục build?

  [y/N] >

Default = N. The decision can be made non-interactive via .run(on_template_unk=...):

"prompt"   (default) prompt the user; abort if stdin is not a TTY
"abort"    raise TemplateTokenizerError immediately, no prompt
"continue" proceed silently — every sample carries the same template UNK at fixed positions

When the build continues with template UNK, the literal UNK positions are recorded in unk_index.jsonl like any other UNK — each entry carries unk_field_indices so readers can tell which segment the UNK came from.

Tier 2 — data content UNK

After the template contract is resolved, samples are processed. If a sample's field values produce UNK tokens, MintDim records an entry in unk_index.jsonl. Each entry includes:

• unk_positions — positions within the encoded payload
• unk_field_indices — which segment of sequence_template each UNK came from (0-base)
• unk_chars — the original substring that produced the UNK

Tier 2 UNK does not fail the build — it is a dataset/tokenizer coverage signal for audit.

Summary:

template/tokenizer mismatch
→ prompt user (default), abort, or continue per .run(on_template_unk=...)

data content produces UNK
→ record in unk_index.jsonl with segment/field index

Token storage dtype

MintDim writes tokenized shards as binary token arrays. Every slot on disk — magic marker, segment lengths, and token payload — uses the same dtype.

V1 supports:

uint16
uint32

Dtype selection:

vocab_size <= 0xFFFF   (65535)
→ uint16, magic = 0xFFFF

vocab_size <= 0xFFFFFFFF
→ uint32, magic = 0xFFFFFFFF

The magic value is reserved for the per-record marker at offset 0; it must not collide with any real token id. If vocab_size > magic for the chosen dtype, MintDim raises TokenDTypeCapacityError at tokenizer load (the dtype selector promotes uint16 → uint32 automatically when needed).

Selected dtype and magic are written to manifest.json and length_field.json.

Example:

{
  "token_dtype": "uint16",
  "magic": 65535
}

Output

.output.dir(
    "./artifacts/instruction_units",
    samples_per_shard=[10000],
)

Two arguments:

path / paths         → output directory path(s)
samples_per_shard    → list of positive ints; samples_per_shard[i] applies to files[i]

The output directory must be new or empty. MintDim V1 does not support overwrite. Use a new output path for a new build.

Sharding

Shards are rolled purely by sample count:

shard_000000  →  samples_per_shard[i] samples
shard_000001  →  samples_per_shard[i] samples
shard_xxxxxx  →  remainder (may be smaller)

This guarantees deterministic shard cardinality for the train loader. Shard byte size depends on unit_size × bytes_per_token × samples_per_shard[i] and varies per unit_xxx/ directory.

Shard observability

MintDim emits one line per closed shard:

[unit-build] shard=000012 samples=100000 bytes=241.8 MiB unit=512

If a closed shard exceeds an internal recommended size (512 MiB), MintDim prints a warning but does not stop the build:

[unit-build:warning] shard=000012 size=684.3 MiB exceeds recommended 512 MiB.
Build continues because sharding is controlled by samples_per_shard.

The threshold is an internal heuristic, not a user-facing config. To produce smaller shards, lower samples_per_shard[i].

Multiple files, fields, templates, tokenizers, units, and outputs

MintDim resolves mapping per config block.

For N input files, each list-style config may be either:

len == 1
→ shared by every input file

len == N
→ mapped by input file index

anything else
→ fail fast

This means fields, templates, tokenizer, sizes, build_batch, output.dir, and samples_per_shard can each be shared or mapped independently.

Fully shared

job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/alpaca.jsonl",
            "./data/sharegpt.jsonl",
        ],
        fields=[
            ["instruction", "output"],
        ],
        templates=[
            "{instruction}\n\n{output}",
        ],
    )
    .tokenizer.sentencepiece(
        files=["./tokenizer/tokenizer.model"],
    )
    .units(
        sizes=[[128, 192, 256, 320]],
        build_batch=[4096],
    )
    .output.dir(
        "./artifacts/instruction_units",
        samples_per_shard=[10000],
    )
)

All files share one schema, one template, one tokenizer, one unit config, and one output directory. The files are written into the same output dataset.

Fully mapped

job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/alpaca.jsonl",
            "./data/sharegpt.jsonl",
        ],
        fields=[
            ["instruction", "input", "output"],
            ["prompt", "response"],
        ],
        templates=[
            "{instruction}\n\n{input}\n\n{output}",
            "{prompt}\n{response}",
        ],
    )
    .tokenizer.sentencepiece(
        files=[
            "./tokenizer/alpaca.model",
            "./tokenizer/sharegpt.model",
        ],
    )
    .units(
        sizes=[
            [128, 192, 256, 320],
            [96, 128, 160, 192, 224, 320],
        ],
        build_batch=[
            4096,
            2048,
        ],
    )
    .output.dir(
        [
            "./artifacts/alpaca_units",
            "./artifacts/sharegpt_units",
        ],
        samples_per_shard=[10000, 5000],
    )
)

Everything is mapped by input order:

files[i]
↔ fields[i]
↔ templates[i]
↔ tokenizer[i]
↔ sizes[i]
↔ build_batch[i]
↔ output_dir[i]
↔ samples_per_shard[i]

Mixed shared and mapped

You may map only the parts that need separate outputs while sharing the rest:

tokenizer_path = "./tokenizer/fineweb_edu_32k.model"

job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/train_a.jsonl",
            "./data/train_b.jsonl",
        ],
        fields=[
            ["prompt", "answer", "separator"],
        ],
        templates=[
            "{prompt}{separator}{answer}",
        ],
    )
    .tokenizer.sentencepiece(
        files=[tokenizer_path, tokenizer_path],
    )
    .units(
        sizes=[[32, 64, 128, 256]],
        build_batch=[64],
    )
    .output.dir(
        [
            "./output1",
            "./output2",
        ],
        samples_per_shard=[256],
    )
)

Resolved order for this example:

file 0 → fields[0], templates[0], tokenizer[0], sizes[0], build_batch[0], output1, samples_per_shard[0]
file 1 → fields[0], templates[0], tokenizer[1], sizes[0], build_batch[0], output2, samples_per_shard[0]

Passing a single-entry files list shares one tokenizer across every input file:

.tokenizer.sentencepiece(files=[tokenizer_path])

Passing N entries maps tokenizers by file index, even if the entries are identical:

.tokenizer.sentencepiece(files=[tokenizer_path, tokenizer_path])

message errors

[english] PipelineValidationError: Pipeline configuration is incomplete.

Missing:

• source.files
• source.fields
• source.templates
• tokenizer
• units.sizes
• units.build_batch
• output.dir
• output.samples_per_shard

[vietnam] PipelineValidationError: Pipeline chưa được cấu hình đầy đủ.

Thiếu:

• source.files
• source.fields
• source.templates
• tokenizer
• units.sizes
• units.build_batch
• output.dir
• output.samples_per_shard

Thiếu cấu hình nào thì báo thiếu cấu hình đó.

[english] MintDimFileNotFoundError: Required file was not found.

Context: source

Missing file: ./data/train.jsonl

[vietnam] MintDimFileNotFoundError: Không tìm thấy file cần dùng.

Ngữ cảnh: source

File thiếu: ./data/train.jsonl

[english] SourceParseError: JSONL line could not be parsed.

File: ./data/alpaca.jsonl

Line: 18291

Reason: invalid JSON syntax

[vietnam] SourceParseError: Không parse được dòng JSONL.

File: ./data/alpaca.jsonl

Dòng: 18291

Lý do: invalid JSON syntax

[english] SampleSchemaError: JSONL sample is missing declared fields.

File: ./data/alpaca.jsonl

Line: 18291

Declared fields: instruction input output

Missing fields: input

Available keys: instruction output metadata

[vietnam] SampleSchemaError: Sample JSONL thiếu field đã khai báo.

File: ./data/alpaca.jsonl

Dòng: 18291

Field đã khai báo: instruction input output

Field bị thiếu: input

Key hiện có: instruction output metadata

TokenizerValidationError reports structural problems with the tokenizer step. Special-token ids are loaded from the tokenizer library; users do not declare unk_id or pad_id.

Components:

files → structural problem in the tokenizer.files list: entry is not a path,
        path is not a string, missing path, or unknown keys leaked through

Full ruleset:

tokenizer.files must be a list of tokenizer file paths
each tokenizer file path must be a string
unk_id and pad_id are loaded from the tokenizer library

Example — a non-string path was passed to tokenizer.files:

[english] TokenizerValidationError: Invalid tokenizer configuration.

Component: files

Problem: tokenizer file path must be str, got int

Rules:

• tokenizer.files must be a list of tokenizer file paths
• each tokenizer file path must be a string
• unk_id and pad_id are loaded from the tokenizer library

[vietnam] TokenizerValidationError: Cấu hình tokenizer không hợp lệ.

Thành phần: files

Vấn đề: tokenizer file path must be str, got int

Quy tắc:

• tokenizer.files phải là list các đường dẫn file tokenizer
• mỗi đường dẫn file tokenizer phải là string
• unk_id và pad_id được lấy từ thư viện tokenizer

Example — tokenizer library does not expose a required special token:

[english] TokenizerSpecialTokenError: Tokenizer library did not expose a required special token.

Tokenizer: ./tokenizer/tokenizer.model

Special token: pad_id

Problem: missing pad_id

Rule: MintDim uses tokenizer-provided special token ids only; do not declare unk_id/pad_id manually.

[vietnam] TokenizerSpecialTokenError: Thư viện tokenizer không cung cấp special token bắt buộc.

Tokenizer: ./tokenizer/tokenizer.model

Special token: pad_id

Vấn đề: missing pad_id

Quy tắc: MintDim chỉ dùng special token id do tokenizer cung cấp; không khai báo unk_id/pad_id thủ công.

[english] TokenizerValidationError: Unsupported tokenizer format.

Supported formats:

• sentencepiece (.model)
• huggingface tokenizer (.json)

[vietnam] TokenizerValidationError: Định dạng tokenizer chưa được hỗ trợ.

Các định dạng hiện hỗ trợ:

• sentencepiece (.model)
• huggingface tokenizer (.json)

[english] TokenizerMappingError: Received 3 input files but 2 tokenizers.

MintDim supports:

• 1 tokenizer shared across all files
• N tokenizers for N files

[vietnam] TokenizerMappingError: Nhận 3 file đầu vào nhưng có 2 tokenizer.

MintDim hỗ trợ:

• 1 tokenizer dùng chung cho tất cả file
• N tokenizer cho N file

[english] FieldMappingError: Received 3 input files but 2 field configs.

MintDim supports:

• 1 field config shared across all files
• N field configs for N files

[vietnam] FieldMappingError: Nhận 3 file đầu vào nhưng có 2 cấu hình fields.

MintDim hỗ trợ:

• 1 cấu hình fields dùng chung cho tất cả file
• N cấu hình fields cho N file

[english] TemplateMappingError: Received 3 input files but 2 templates.

MintDim supports:

• 1 template shared across all files
• N templates for N files

[vietnam] TemplateMappingError: Nhận 3 file đầu vào nhưng có 2 template.

MintDim hỗ trợ:

• 1 template dùng chung cho tất cả file
• N template cho N file

Example — template uses a placeholder that is not declared in fields:

[english] TemplateFieldMismatchError: Source file: ./data/train.jsonl

API declaration for this file: fields = ['instruction', 'input', 'output'] template = '{instruction}\n\n{prompt}\n\n{output}'

Field causing template mismatch: prompt Reason: template uses {prompt}, but fields does not declare it

[vietnam] TemplateFieldMismatchError: File nguồn: ./data/train.jsonl

Khai báo API cho file này: fields = ['instruction', 'input', 'output'] template = '{instruction}\n\n{prompt}\n\n{output}'

Field gây lệch template: prompt Lý do: template dùng {prompt}, nhưng fields không khai báo field này

Example — fields declares a field that the template does not use:

[english] TemplateFieldMismatchError: Source file: ./data/train.jsonl

API declaration for this file: fields = ['instruction', 'input', 'output'] template = '{instruction}\n\n{output}'

Field causing template mismatch: input Reason: fields declares this field, but template does not use {input}

[vietnam] TemplateFieldMismatchError: File nguồn: ./data/train.jsonl

Khai báo API cho file này: fields = ['instruction', 'input', 'output'] template = '{instruction}\n\n{output}'

Field gây lệch template: input Lý do: fields khai báo field này, nhưng template không dùng {input}

[english] TemplateTokenizerError: Template literal segments produce UNK when encoded by the tokenizer.

File index: 0

Template: '{a}\n\n{b}'

Problem: template literal characters are not fully covered by tokenizer vocabulary

Fix: adjust template literal characters, use a tokenizer that covers them, or run with an explicit template-UNK policy.

[vietnam] TemplateTokenizerError: Các đoạn literal của template tạo UNK khi được tokenizer encode.

Vị trí file: 0

Template: '{a}\n\n{b}'

Vấn đề: ký tự literal trong template không được tokenizer vocabulary bao phủ đầy đủ

Cách sửa: chỉnh ký tự literal trong template, dùng tokenizer bao phủ các ký tự đó, hoặc chạy với template-UNK policy rõ ràng.

[english] UnitMappingError: Received 3 input files but 2 unit configs.

MintDim supports:

• 1 unit config shared across all files
• N unit configs for N input files

[vietnam] UnitMappingError: Nhận 3 file đầu vào nhưng có 2 cấu hình unit.

MintDim hỗ trợ:

• 1 cấu hình unit dùng chung cho tất cả file
• N cấu hình unit cho N file đầu vào

[english] OutputMappingError: Received 3 input files but 2 output directories.

MintDim supports:

• 1 shared output directory
• N output directories for N input files

[vietnam] OutputMappingError: Nhận 3 file đầu vào nhưng có 2 thư mục output.

MintDim hỗ trợ:

• 1 thư mục output dùng chung
• N thư mục output cho N file đầu vào

[english] UnitValidationError: Invalid unit config.

Rules:

• sizes must be positive integers
• sizes must be sorted ascending
• build_batch must be a positive integer

[vietnam] UnitValidationError: Cấu hình unit không hợp lệ.

Quy tắc:

• sizes phải là các số nguyên dương
• sizes phải được sắp xếp tăng dần
• build_batch phải là số nguyên dương

[english] UnitOverflowError: Sample token length exceeds max unit size.

Sample: ./data/alpaca.jsonl:18291

token_count: 417

max_unit_size: 320

Fix:

• increase unit sizes
• or run with an overflow policy that truncates, skips, or prompts

[vietnam] UnitOverflowError: Độ dài token của sample vượt quá unit size lớn nhất.

Sample: ./data/alpaca.jsonl:18291

token_count: 417

max_unit_size: 320

Cách sửa:

• tăng unit sizes
• hoặc chạy với overflow policy để truncate, skip, hoặc prompt

[english] OutputValidationError: Output directory already exists and is not empty.

Directory: ./artifacts/token_shards

Use a new output directory.

[vietnam] OutputValidationError: Thư mục output đã tồn tại và không rỗng.

Thư mục: ./artifacts/token_shards

Hãy dùng thư mục output mới.

[english] TokenDTypeCapacityError: Tokenizer vocabulary size is too large for the token dtypes supported by MintDim.

Tokenizer vocab_size: 5,000,000,000

Supported token dtypes:

• uint16: max vocab_size 65,535
• uint32: max vocab_size 4,294,967,295

Fix:

• use a tokenizer with a smaller vocabulary
• or use a MintDim version that supports a wider token dtype

[vietnam] TokenDTypeCapacityError: Kích thước vocabulary của tokenizer quá lớn so với token dtype MintDim hỗ trợ.

Tokenizer vocab_size: 5.000.000.000

Token dtype được hỗ trợ:

• uint16: vocab_size tối đa 65.535
• uint32: vocab_size tối đa 4.294.967.295

Cách sửa:

• dùng tokenizer có vocabulary nhỏ hơn
• hoặc dùng phiên bản MintDim hỗ trợ token dtype rộng hơn

SamplesPerShardError reports value-level and arity-level problems for samples_per_shard.

Rules:

each samples_per_shard[i] is a positive int
len(samples_per_shard) == 1 or len(samples_per_shard) == file_count

Example — non-positive value:

[english] SamplesPerShardError: Invalid samples_per_shard.

Problem: samples_per_shard[0] must be a positive int, got 0

Rules:

• each samples_per_shard[i] is a positive int
• len(samples_per_shard) == 1 or len(samples_per_shard) == file_count

[vietnam] SamplesPerShardError: samples_per_shard không hợp lệ.

Vấn đề: samples_per_shard[0] phải là số nguyên dương, hiện tại là 0

Quy tắc:

• mỗi samples_per_shard[i] phải là số nguyên dương
• len(samples_per_shard) == 1 hoặc len(samples_per_shard) == số lượng file

Example — arity mismatch:

[english] SamplesPerShardError: Invalid samples_per_shard.

Problem: Received 3 input files but 2 samples_per_shard entries

Rules:

• each samples_per_shard[i] is a positive int
• len(samples_per_shard) == 1 or len(samples_per_shard) == file_count

[vietnam] SamplesPerShardError: samples_per_shard không hợp lệ.

Vấn đề: Nhận 3 file đầu vào nhưng có 2 entries samples_per_shard

Quy tắc:

• mỗi samples_per_shard[i] phải là số nguyên dương
• len(samples_per_shard) == 1 hoặc len(samples_per_shard) == số lượng file

Repository layout

mintdim/
├─ pyproject.toml
├─ README.md
├─ LICENSE
├─ .gitignore
│
├─ mintdim/
│  ├─ __init__.py
│  ├─ pipeline.py
│  ├─ registry.py
│  │
│  ├─ cli/
│  │  ├─ __init__.py
│  │  └─ main.py
│  │
│  ├─ shared/
│  │  ├─ __init__.py
│  │  ├─ paths.py
│  │  ├─ errors.py
│  │  └─ types.py
│  │
│  └─ apis/
│     ├─ __init__.py
│     │
│     └─ unit_build/
│        ├─ __init__.py
│        ├─ api.py
│        ├─ config.py
│        ├─ contracts.py
│        ├─ validate.py
│        ├─ runtime.py
│        │
│        ├─ cli/
│        │  ├─ main.py
│        │  └─ prompts.py
│        │
│        ├─ shared/
│        │  ├─ __init__.py
│        │  ├─ errors.py
│        │  ├─ types.py
│        │  └─ constants.py
│        │
│        ├─ source/
│        │  ├─ __init__.py
│        │  ├─ jsonl.py
│        │  ├─ template.py
│        │  └─ validate.py
│        │
│        ├─ tokenizer/
│        │  ├─ __init__.py
│        │  ├─ hf_json.py
│        │  ├─ sentencepiece.py
│        │  └─ validate.py
│        │
│        ├─ units/
│        │  ├─ __init__.py
│        │  ├─ overflow.py
│        │  ├─ planner.py
│        │  ├─ shard_writer.py
│        │  └─ validate.py
│        │
│        └─ output/
│           ├─ __init__.py
│           ├─ dir.py
│           ├─ manifest.py
│           ├─ length_field.py
│           ├─ sample_index.py
│           ├─ duplicate_index.py
│           ├─ unk_index.py
│           ├─ overflow_index.py
│           ├─ histogram.py
│           ├─ stats.py
│           └─ validate.py
│
├─ tests/
│  └─ apis/
│     └─ unit_build/
│        ├─ conftest.py
│        ├─ test_api_chain.py
│        ├─ test_indices.py
│        ├─ test_jsonl_source.py
│        ├─ test_loaders.py
│        ├─ test_output.py
│        ├─ test_shard_writer.py
│        ├─ test_smoke.py
│        ├─ test_template.py
│        ├─ test_tokenizer_validate.py
│        ├─ test_units.py
│        └─ test_validate.py
│
└─ examples/
   └─ unit_build_instruction.py

Output layout

Shared output layout

./artifacts/instruction_units/
├─ manifest.json
├─ length_field.json
├─ histogram.json
├─ stats.json
├─ sample_index.jsonl
├─ duplicate_index.jsonl
├─ unk_index.jsonl
├─ overflow_index.jsonl
│
├─ unit_128/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
├─ unit_192/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
├─ unit_256/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
└─ unit_320/
   ├─ shard_000000.bin
   ├─ shard_000001.bin
   └─ ...

Meaning:

manifest.json
→ pipeline metadata, build configuration, tokenizer metadata, token dtype, magic, sequence_template

length_field.json
→ per-record header schema readers use to decode shards (magic, sequence_template, dtype, pad_token_id)

histogram.json
→ sample distribution by assigned unit size

stats.json
→ summarized token, shard, duplicate, and UNK statistics

sample_index.jsonl
→ per-sample hash, token count, unit assignment, shard location, empty_fields

duplicate_index.jsonl
→ grouped duplicate samples by hash

unk_index.jsonl
→ samples containing UNK tokens with positions, field indices, and original chars

overflow_index.jsonl
→ samples whose token_count exceeded max(sizes) and how the policy resolved them
  (truncate / extend / skip)

unit_xxx/
→ binary token shards grouped by unit size

unit_xxx/EMPTY_UNIT.md
→ generated when a declared unit receives no assigned samples

unit_xxx/UNIT_BUILD_FAILED.md
→ generated when the build fails before the unit output is finalized

manifest.json

Example:

{
  "pipeline": "unit-build",
  "mintdim_version": "0.1.8",
  "tokenizer": {
    "type": "sentencepiece",
    "path": "./tokenizer/tokenizer.model",
    "vocab_size": 32000,
    "unk_id": 0,
    "pad_id": 1
  },
  "token_storage": {
    "dtype": "uint16",
    "bytes_per_token": 2,
    "magic": 65535
  },
  "sequence_template": [
    "token_template",
    "instruction",
    "token_template",
    "input",
    "token_template",
    "output"
  ],
  "length_field_metadata": "length_field.json",
  "units": {
    "sizes": [128, 192, 256, 320],
    "build_batch": 4096
  },
  "source": {
    "files": ["./data/alpaca.jsonl"],
    "fields": [["instruction", "input", "output"]],
    "templates": ["{instruction}\n\n{input}\n\n{output}"]
  },
  "output": {
    "samples_per_shard": 10000
  }
}

If the overflow policy extends sizes mid-build, the persisted list includes the new size — manifest reflects the layout that actually exists on disk.

length_field.json

Example:

{
  "magic": 65535,
  "sequence_template": [
    "token_template",
    "instruction",
    "token_template",
    "input",
    "token_template",
    "output"
  ],
  "token_dtype": "uint16",
  "pad_token_id": 0
}

Readers use this file to:

• verify the magic marker at slot 0 of every record
• decode the N length headers that follow (N = len(sequence_template))
• know which segment (field or literal) each header refers to

"token_template" is the reserved name for a literal segment of the template. Field segments use their declared field name.

histogram.json

Example:

{
  "128": 30291,
  "192": 82191,
  "256": 120882,
  "320": 43121
}

Meaning:

histogram.json counts how many samples were assigned to each unit size.

stats.json

Example:

{
  "total_samples": 1823811,
  "total_tokens": 482918221,
  "token_dtype": "uint16",
  "bytes_per_token": 2,
  "duplicate_groups": 821,
  "duplicate_samples": 1942,
  "samples_with_unk": 321,
  "unk_tokens": 1821,
  "unit_distribution": {
    "128": 582111,
    "192": 891221,
    "256": 301882,
    "320": 48291
  }
}

sample_index.jsonl

Each line describes one source sample after tokenization and unit assignment.

Example:

{
  "sample_id": 91822,
  "hash": "8f2c1b7d9a9d4a0d...",
  "token_count": 143,
  "unit_size": 192,
  "source_file": "./data/alpaca.jsonl",
  "source_line": 18291,
  "shard_path": "unit_192/shard_000001.bin",
  "empty_fields": []
}

empty_fields lists the field names whose values rendered to an empty string for this sample (the field stays in sequence_template with len_field = 0). It's a fast filter for finding samples missing actual content in a slot.

Purpose:

sample_index.jsonl enables:
- dataset audit
- sample-level reproducibility
- deduplication
- corruption detection
- shard rebuild/debug
- token count verification
- missing-content detection (via empty_fields)

duplicate_index.jsonl

Each line groups all duplicated samples sharing the same content hash.

Example:

{
  "hash": "8f2c1b7d9a9d4a0d...",
  "count": 3,
  "samples": [
    {
      "sample_id": 12031,
      "source_file": "./data/alpaca.jsonl",
      "source_line": 991
    },
    {
      "sample_id": 91822,
      "source_file": "./data/alpaca.jsonl",
      "source_line": 18291
    },
    {
      "sample_id": 140992,
      "source_file": "./data/sharegpt.jsonl",
      "source_line": 2012
    }
  ]
}

Purpose:

duplicate_index.jsonl enables:
- duplicate audit
- dataset cleanup
- repeated sample detection
- duplicate filtering
- corruption investigation

unk_index.jsonl

Each line describes one sample whose tokenization produced UNK. This includes both UNK from field values (Tier 2) and, when the build was allowed to continue past Tier 1, UNK from template literals.

Example:

{
  "sample_id": 91822,
  "hash": "8f2c1b7d9a9d4a0d...",
  "source_file": "./data/alpaca.jsonl",
  "source_line": 18291,
  "unk_count": 2,
  "unk_positions": [17, 89],
  "unk_field_indices": [1, 3],
  "unk_chars": ["𓀀", "🜲"]
}

Field meaning:

unk_positions
→ positions of each UNK within the encoded payload (0-base, payload-relative)

unk_field_indices
→ segment index in sequence_template (0-base) where the UNK originated;
  aligned 1:1 with unk_positions

unk_chars
→ original substring from the rendered text that produced each UNK,
  aligned 1:1 with unk_positions

Purpose:

unk_index.jsonl enables:
- tokenizer quality audit
- unknown token analysis (by source segment)
- tokenizer coverage inspection
- dataset cleanup
- encoding issue detection

overflow_index.jsonl

Each line records one sample whose sum(segment token lengths) exceeded max(sizes) and how the policy resolved it.

Example:

{
  "sample_id": 91822,
  "source_file": "./data/alpaca.jsonl",
  "source_line": 18291,
  "token_count": 412,
  "max_unit_size": 320,
  "action": "extend",
  "new_unit_size": 512
}

action is one of "truncate", "extend", or "skip". new_unit_size appears only when action == "extend".

Purpose:

overflow_index.jsonl enables:
- visibility into samples that didn't fit the original sizes plan
- audit trail for the overflow policy resolution
- post-hoc analysis to decide whether to grow `sizes` in the next build

Binary token shards

Each record in unit_xxx/shard_*.bin is laid out as:

[magic] [len_seg_0] [len_seg_1] ... [len_seg_{n-1}] [payload_0 .. payload_{unit_size-1}]
└──1──┘ └──────────── n length headers ────────────┘ └──── payload of length unit_size ────┘

• All slots use token_dtype (uint16 → magic 0xFFFF, uint32 → magic 0xFFFFFFFF).
• n = len(sequence_template). Segment names live in length_field.json.
• unit_size counts payload slots only. The first 1 + n slots are header metadata, not training tokens.
• The payload concatenates each segment's tokens in sequence_template order, then pads with pad_id to unit_size.
• Total record size in slots = 1 + n + unit_size; in bytes = that × bytes_per_token.
• The k-th record in a shard starts at byte k * (1 + n + unit_size) * bytes_per_token.

To slice a segment back out:

payload_offset    = 1 + n
segment_i_start   = payload_offset + sum(len_seg_0..i-1)
segment_i_end     = segment_i_start + len_seg_i

magic is reserved — readers can verify it at slot 0 of every record to catch shard corruption.

Multi-output layout

./artifacts/
├─ alpaca_units/
│  ├─ manifest.json
│  ├─ histogram.json
│  ├─ stats.json
│  ├─ sample_index.jsonl
│  ├─ duplicate_index.jsonl
│  ├─ unk_index.jsonl
│  │
│  ├─ unit_128/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  ├─ unit_192/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  ├─ unit_256/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  └─ unit_320/
│     ├─ shard_000000.bin
│     └─ ...
│
└─ sharegpt_units/
   ├─ manifest.json
   ├─ histogram.json
   ├─ stats.json
   ├─ sample_index.jsonl
   ├─ duplicate_index.jsonl
   ├─ unk_index.jsonl
   │
   ├─ unit_96/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_128/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_160/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_192/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_224/
   │  ├─ shard_000000.bin
   │  └─ ...
   └─ unit_320/
      ├─ shard_000000.bin
      └─ ...

Each output directory is isolated and mapped by input order when output.dir has one path per input file. Other blocks may still be shared:

files[i]
↔ fields[0] or fields[i]
↔ templates[0] or templates[i]
↔ tokenizer[0] or tokenizer[i]
↔ sizes[0] or sizes[i]
↔ build_batch[0] or build_batch[i]
↔ output_dir[i]
↔ samples_per_shard[0] or samples_per_shard[i]

Design principles

small public API
explicit paths
explicit user-defined templates (no presets)
required-field sample validation (extra JSONL keys are ignored)
fail fast validation
two-tier UNK contract (template prompt vs data record)
interactive overflow resolution (truncate / extend / skip)
exact per-segment tokenization
direct unit assignment by token length
fixed-width binary shards with [magic][lengths][payload] framing
length_field.json describes per-record header schema for readers
sequence_template preserves field/literal boundaries on disk
automatic token dtype selection with magic-collision guard
no tokenizer mutation
special token ids loaded from tokenizer library
no automatic repo scanning
no overwrite in V1
no build resume/cache
pipeline-specific internals
sample-level reproducibility
dataset quality audit support (sample_index, unk_index, overflow_index, duplicate_index)

Version

Initial public version:

0.1.8