mintdim 0.1.5

Token unit pipeline for building low-padding binary token shards from JSONL datasets.

MintDim

MintDim is a token unit pipeline for building low-padding binary token shards from JSONL datasets.

The first public API focuses on one task:

stream JSONL batches → render templates → tokenize exactly → assign by token length → write binary token unit shards

Install

pip install mintdim

For local development:

pip install -e .

Quick start

from mintdim import pipeline


def build_instruction_units():
    job = (
        pipeline("unit-build")
        .source.jsonl(
            files=[
                "./data/alpaca.jsonl",
            ],
            fields=[
                ["instruction", "input", "output"],
            ],
            templates=[
                "{instruction}\n\n{input}\n\n{output}",
            ],
        )
        .tokenizer.sentencepiece(
            {
                "path": "./tokenizer/tokenizer.model",
            },
        )
        .units(
            sizes=[[128, 192, 256, 320]],
            build_batch=[4096],
        )
        .output.dir(
            "./artifacts/instruction_units",
            samples_per_shard=[10000],
        )
    )

    return job.run()

API flow

pipeline("unit-build")
→ source
→ tokenizer
→ units
→ output
→ run

MintDim uses a fluent pipeline API. Each step declares one part of the build job. Nothing is executed until job.run() is called.

Batch build flow:

stream JSONL sequentially
→ collect source batch
→ render text batch from template
→ hash rendered text batch
→ tokenizer.encode_batch(texts)
→ assign unit by exact token_count inside the batch
→ group records by unit_size
→ writer[unit].write_batch(token_ids)
→ write_many sample/UNK/duplicate index rows
→ update stats and histogram in batch

Path rules

All relative paths are resolved from the current working directory.

Example:

.source.jsonl(
    files=["./data/train.jsonl"],
    fields=[["text"]],
    templates=["{text}"],
)

If the script is run from the project root, MintDim resolves the file as:

./data/train.jsonl

MintDim does not scan the repository or search the filesystem automatically. Users must provide explicit paths.

JSONL source

.source.jsonl(
    files=["./data/train.jsonl"],
    fields=[["text"]],
    templates=["{text}"],
)

Three arguments are required:

files      → list of JSONL paths
fields     → list-of-list; fields[i] is the JSONL keys MintDim reads from files[i]
templates  → list of user-defined templates; templates[i] formats samples of files[i]

Multiple files are supported:

.source.jsonl(
    files=[
        "./data/alpaca.jsonl",
        "./data/sharegpt.jsonl",
    ],
    fields=[
        ["instruction", "output"],
        ["prompt", "response"],
    ],
    templates=[
        "{instruction}\n\n{output}",
        "{prompt}\n{response}",
    ],
)

Each JSONL sample MUST contain every key listed in fields[i]. Extra JSONL keys are allowed and ignored.

Templates

Templates are user-defined strings. MintDim does not ship preset templates (no chatml, alpaca, etc.). The user controls every character.

Template syntax:

- placeholders use {field_name}
- placeholder names must match fields[i] exactly as a set
- any other character is treated as a literal (\n, \t, custom separators, etc.)
- placeholders are substituted by the JSONL sample's field values

Example:

templates=[
    "{instruction}\n\n{input}\n\n{output}",
]

Rules:

1. set(placeholders in templates[i]) == set(fields[i])
2. JSONL samples must contain every declared field in fields[i]
3. Declared fields that do not appear in the template are invalid
4. Static parts of templates[i] (every character outside placeholders)
   must encode cleanly with the tokenizer (no UNK)
5. Template rendering is a pure substitution; no extra tokens are injected

A user may use any character their tokenizer supports. If the tokenizer encodes \t, then "{a}\t{b}" is valid. If not, MintDim fails pre-flight.

Invalid example:

fields=[
    ["prompt", "answer", "separator", "source"],
]
templates=[
    "{prompt}{separator}{answer}",
]

source is declared but not consumed by the template, so MintDim raises TemplateFieldMismatchError.

Tokenizer

V1 supports:

- SentencePiece tokenizer: .model
- Hugging Face tokenizer: .json

Each tokenizer is configured with a dict, not a bare path. The user declares only the tokenizer file path. MintDim loads unk_id and pad_id from the tokenizer library itself.

SentencePiece:

.tokenizer.sentencepiece(
    {
        "path": "./tokenizer/tokenizer.model",
    },
)

Hugging Face tokenizer JSON:

.tokenizer.hf_json(
    {
        "path": "./tokenizer/tokenizer.json",
    },
)

Required keys:

path → tokenizer file path

Validation rules:

- path is a string
- the tokenizer library exposes a valid UNK id
- the tokenizer library exposes a valid PAD id

Why UNK and PAD are required:

UNK
→ used for tracking unknown tokens and dataset integrity statistics

PAD
→ used to pad samples to assigned unit size for fixed-width binary shard layout

The tokenizer is never modified by MintDim. UNK and PAD ids come from the loaded tokenizer object, not from user-declared values or MintDim-side vocab parsing.

Units

Shared unit config:

.units(
    sizes=[[128, 192, 256, 320]],
    build_batch=[4096],
)

Mapped unit config:

.units(
    sizes=[
        [128, 192, 256, 320],
        [96, 128, 160, 192, 224, 320],
    ],
    build_batch=[
        4096,
        2048,
    ],
)

Meaning:

sizes
→ target shard unit sizes

build_batch
→ number of raw samples processed per tokenizer batch

MintDim always tokenizes samples exactly. Unit assignment is based on exact token count.

Example:

token_count = 143
sizes = [128, 192, 256, 320]
target unit = 192

If a sample exceeds the largest configured unit size, MintDim fails immediately with UnitOverflowError.

MintDim V1 does not:

• truncate samples
• skip overflow samples
• continue partial builds
• cache build progress

UNK handling

MintDim distinguishes two kinds of UNK and treats them differently.

Tier 1 — template/tokenizer contract

Before any data is processed, every static part of every template is tokenized once. If any static part produces UNK, MintDim fails immediately with TemplateTokenizerError. This means the template's literal characters cannot be cleanly represented by the tokenizer, which is a configuration error, not a data issue.

Tier 2 — data content UNK

After the template/tokenizer contract is validated, samples are processed. If a sample's field values produce UNK tokens when encoded, MintDim records an entry in unk_index.jsonl. Tier 2 UNK does not fail the build — it is a dataset/tokenizer coverage signal for audit.

Summary:

template/tokenizer mismatch
→ fail fast (TemplateTokenizerError)

data content produces UNK
→ record in unk_index.jsonl

Token storage dtype

MintDim writes tokenized shards as binary token arrays.

V1 supports:

uint16
uint32

Dtype selection:

vocab_size <= 65535
→ uint16

vocab_size > 65535
→ uint32

The selected dtype is written to manifest.json.

Example:

{
  "token_dtype": "uint16"
}

Output

.output.dir(
    "./artifacts/instruction_units",
    samples_per_shard=[10000],
)

Two arguments:

path / paths         → output directory path(s)
samples_per_shard    → list of positive ints; samples_per_shard[i] applies to files[i]

The output directory must be new or empty. MintDim V1 does not support overwrite. Use a new output path for a new build.

Sharding

Shards are rolled purely by sample count:

shard_000000  →  samples_per_shard[i] samples
shard_000001  →  samples_per_shard[i] samples
shard_xxxxxx  →  remainder (may be smaller)

This guarantees deterministic shard cardinality for the train loader. Shard byte size depends on unit_size × bytes_per_token × samples_per_shard[i] and varies per unit_xxx/ directory.

Shard observability

MintDim emits one line per closed shard:

[unit-build] shard=000012 samples=100000 bytes=241.8 MiB unit=512

If a closed shard exceeds an internal recommended size (512 MiB), MintDim prints a warning but does not stop the build:

[unit-build:warning] shard=000012 size=684.3 MiB exceeds recommended 512 MiB.
Build continues because sharding is controlled by samples_per_shard.

The threshold is an internal heuristic, not a user-facing config. To produce smaller shards, lower samples_per_shard[i].

Multiple files, fields, templates, tokenizers, units, and outputs

MintDim resolves mapping per config block.

For N input files, each list-style config may be either:

len == 1
→ shared by every input file

len == N
→ mapped by input file index

anything else
→ fail fast

This means fields, templates, tokenizer, sizes, build_batch, output.dir, and samples_per_shard can each be shared or mapped independently.

Fully shared

job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/alpaca.jsonl",
            "./data/sharegpt.jsonl",
        ],
        fields=[
            ["instruction", "output"],
        ],
        templates=[
            "{instruction}\n\n{output}",
        ],
    )
    .tokenizer.sentencepiece(
        {
            "path": "./tokenizer/tokenizer.model",
        },
    )
    .units(
        sizes=[[128, 192, 256, 320]],
        build_batch=[4096],
    )
    .output.dir(
        "./artifacts/instruction_units",
        samples_per_shard=[10000],
    )
)

All files share one schema, one template, one tokenizer, one unit config, and one output directory. The files are written into the same output dataset.

Fully mapped

job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/alpaca.jsonl",
            "./data/sharegpt.jsonl",
        ],
        fields=[
            ["instruction", "input", "output"],
            ["prompt", "response"],
        ],
        templates=[
            "{instruction}\n\n{input}\n\n{output}",
            "{prompt}\n{response}",
        ],
    )
    .tokenizer.sentencepiece([
        {
            "path": "./tokenizer/alpaca.model",
        },
        {
            "path": "./tokenizer/sharegpt.model",
        },
    ])
    .units(
        sizes=[
            [128, 192, 256, 320],
            [96, 128, 160, 192, 224, 320],
        ],
        build_batch=[
            4096,
            2048,
        ],
    )
    .output.dir(
        [
            "./artifacts/alpaca_units",
            "./artifacts/sharegpt_units",
        ],
        samples_per_shard=[10000, 5000],
    )
)

Everything is mapped by input order:

files[i]
↔ fields[i]
↔ templates[i]
↔ tokenizer[i]
↔ sizes[i]
↔ build_batch[i]
↔ output_dir[i]
↔ samples_per_shard[i]

Mixed shared and mapped

You may map only the parts that need separate outputs while sharing the rest:

tokenizer_cfg = {
    "path": "./tokenizer/fineweb_edu_32k.model",
}

job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/train_a.jsonl",
            "./data/train_b.jsonl",
        ],
        fields=[
            ["prompt", "answer", "separator"],
        ],
        templates=[
            "{prompt}{separator}{answer}",
        ],
    )
    .tokenizer.sentencepiece([
        tokenizer_cfg,
        tokenizer_cfg,
    ])
    .units(
        sizes=[[32, 64, 128, 256]],
        build_batch=[64],
    )
    .output.dir(
        [
            "./output1",
            "./output2",
        ],
        samples_per_shard=[256],
    )
)

Resolved order for this example:

file 0 → fields[0], templates[0], tokenizer[0], sizes[0], build_batch[0], output1, samples_per_shard[0]
file 1 → fields[0], templates[0], tokenizer[1], sizes[0], build_batch[0], output2, samples_per_shard[0]

Passing a single tokenizer dict shares it across files:

.tokenizer.sentencepiece(tokenizer_cfg)

Passing a list maps by file index, even if the entries contain the same values:

.tokenizer.sentencepiece([tokenizer_cfg, tokenizer_cfg])

message errors

[english] PipelineValidationError: Pipeline configuration is incomplete.

Missing:

• source.files
• source.fields
• source.templates
• tokenizer
• units.sizes
• units.build_batch
• output.dir
• output.samples_per_shard

[vietnam] PipelineValidationError: Pipeline chưa được cấu hình đầy đủ.

Thiếu:

• source.files
• source.fields
• source.templates
• tokenizer
• units.sizes
• units.build_batch
• output.dir
• output.samples_per_shard

Thiếu cấu hình nào thì báo thiếu cấu hình đó.

[english] MintDimFileNotFoundError: Required file was not found.

Context: source

Missing file: ./data/train.jsonl

[vietnam] MintDimFileNotFoundError: Không tìm thấy file cần dùng.

Context: source

File thiếu: ./data/train.jsonl

[english] SourceParseError: JSONL line could not be parsed.

File: ./data/alpaca.jsonl

Line: 18291

Reason: invalid JSON syntax

[vietnam] SourceParseError: Không đọc được dòng JSONL.

File: ./data/alpaca.jsonl

Dòng: 18291

Lý do: JSON không hợp lệ

[english] SampleSchemaError: JSONL sample is missing declared fields.

File: ./data/alpaca.jsonl

Line: 18291

Declared fields: instruction input output

Missing fields: input

Available keys: instruction output metadata

[vietnam] SampleSchemaError: Sample JSONL thiếu fields đã khai báo.

File: ./data/alpaca.jsonl

Dòng: 18291

Fields đã khai báo: instruction input output

Fields bị thiếu: input

Keys hiện có: instruction output metadata

TokenizerValidationError reports structural tokenizer config problems. Special-token ids are loaded from the tokenizer library; users do not declare unk_id or pad_id.

Components:

config → structural problem (not a dict, missing path, path not string, unknown keys)

Full ruleset (each component reports only its own rules):

config must be a dict with key: path
path must be a string
unk_id and pad_id are loaded from the tokenizer library

Example — manual pad_id is declared:

[english] TokenizerValidationError: Invalid tokenizer config.

Component: config

Problem: tokenizer config has unknown keys: ['pad_id']. unk_id and pad_id are loaded from the tokenizer library.

Rules:

• config must be a dict with key: path
• path must be a string
• unk_id and pad_id are loaded from the tokenizer library

[vietnam] TokenizerValidationError: Cấu hình tokenizer không hợp lệ.

Thành phần: config

Vấn đề: config tokenizer có key không hợp lệ: ['pad_id']. unk_id và pad_id được lấy từ thư viện tokenizer.

Quy tắc:

• config phải là dict có key: path
• path phải là string
• unk_id và pad_id được lấy từ thư viện tokenizer

[english] TokenizerSpecialTokenError: Tokenizer library did not expose a required special token.

Special token: pad_id

[vietnam] TokenizerSpecialTokenError: Thư viện tokenizer không cung cấp special token bắt buộc.

Special token: pad_id

[english] TokenizerValidationError: Unsupported tokenizer format.

Supported formats:

• sentencepiece (.model)
• huggingface tokenizer (.json)

[vietnam] TokenizerValidationError: Định dạng tokenizer chưa được hỗ trợ.

Các định dạng hiện hỗ trợ:

• sentencepiece (.model)
• huggingface tokenizer (.json)

[english] TokenizerMappingError: Received 3 input files but 2 tokenizers.

MintDim supports:

• 1 tokenizer shared across all files
• N tokenizers for N files

[vietnam] TokenizerMappingError: Nhận 3 file đầu vào nhưng chỉ có 2 tokenizer.

MintDim hỗ trợ:

• 1 tokenizer dùng chung cho tất cả file
• N tokenizer cho N file

[english] FieldMappingError: Received 3 input files but 2 field configs.

MintDim supports:

• 1 field config shared across all files
• N field configs for N files

[vietnam] FieldMappingError: Nhận 3 file đầu vào nhưng chỉ có 2 cấu hình fields.

MintDim hỗ trợ:

• 1 fields dùng chung cho tất cả file
• N fields cho N file

[english] TemplateMappingError: Received 3 input files but 2 templates.

MintDim supports:

• 1 template shared across all files
• N templates for N files

[vietnam] TemplateMappingError: Nhận 3 file đầu vào nhưng chỉ có 2 template.

MintDim hỗ trợ:

• 1 template dùng chung cho tất cả file
• N template cho N file

[english] TemplateFieldMismatchError: Source file: ./data/train.jsonl

API declaration for this file: fields = ['instruction', 'input', 'output'] template = "{instruction}\n\n{prompt}\n\n{output}"

Field causing template mismatch: prompt Reason: template uses {prompt}, but fields does not declare it

[vietnam] TemplateFieldMismatchError: File nguồn: ./data/train.jsonl

Khai báo API cho file này: fields = ['instruction', 'input', 'output'] template = "{instruction}\n\n{prompt}\n\n{output}"

Field gây lệch template: prompt Lý do: template dùng {prompt}, nhưng fields không khai báo field này

[english] TemplateTokenizerError: Template static parts produce UNK when encoded by the tokenizer.

File index: 0

Template: "{a}\n\n{b}"

Problem: template literal characters are not fully covered by tokenizer vocabulary

Fix: adjust template literal characters or use a tokenizer that covers them

[vietnam] TemplateTokenizerError: Phần literal của template tạo UNK khi tokenize.

Vị trí file: 0

Template: "{a}\n\n{b}"

Vấn đề: ký tự literal trong template không được tokenizer bao phủ

Cách sửa: chỉnh ký tự literal hoặc dùng tokenizer bao phủ được

[english] UnitMappingError: Received 3 input files but 2 unit configs.

MintDim supports:

• 1 unit config shared across all files
• N unit configs for N input files

[vietnam] UnitMappingError: Nhận 3 file đầu vào nhưng chỉ có 2 unit config.

MintDim hỗ trợ:

• 1 unit config dùng chung cho tất cả file
• N unit config cho N file

[english] OutputMappingError: Received 3 input files but 2 output directories.

MintDim supports:

• 1 shared output directory
• N output directories for N input files

[vietnam] OutputMappingError: Nhận 3 file đầu vào nhưng chỉ có 2 thư mục output.

MintDim hỗ trợ:

• 1 output dùng chung
• N output cho N file

[english] UnitValidationError: Invalid unit config.

Rules:

• sizes must be positive integers
• sizes must be sorted ascending
• build_batch must be a positive integer

[vietnam] UnitValidationError: Unit config không hợp lệ.

Quy tắc:

• sizes phải là các số nguyên dương
• sizes phải tăng dần
• build_batch phải là số nguyên dương

[english] UnitOverflowError: Sample token length exceeds max unit size.

Sample: ./data/alpaca.jsonl:18291

token_count: 417

max_unit_size: 320

Increase unit sizes and run again.

[vietnam] UnitOverflowError: Độ dài token của mẫu vượt quá unit lớn nhất.

Mẫu: ./data/alpaca.jsonl:18291

token_count: 417

max_unit_size: 320

Hãy tăng unit sizes rồi chạy lại.

[english] OutputValidationError: Output directory already exists and is not empty.

Directory: ./artifacts/token_shards

Use a new output directory.

[vietnam] OutputValidationError: Thư mục output đã tồn tại và không rỗng.

Thư mục: ./artifacts/token_shards

Hãy dùng thư mục output mới.

[english] TokenDTypeValidationError: Tokenizer vocabulary is too large for supported token dtypes.

Supported dtypes:

• uint16
• uint32

[vietnam] TokenDTypeValidationError: Vocab của tokenizer quá lớn so với dtype token được hỗ trợ.

Các dtype hiện hỗ trợ:

• uint16
• uint32

SamplesPerShardError reports value-level and arity-level problems for samples_per_shard.

Example — non-positive value:

[english] SamplesPerShardError: Invalid samples_per_shard.

Problem: samples_per_shard[0] must be a positive int, got 0

Rules:

• each samples_per_shard[i] is a positive int
• len(samples_per_shard) == 1 or len(samples_per_shard) == file_count

[vietnam] SamplesPerShardError: samples_per_shard không hợp lệ.

Vấn đề: samples_per_shard[0] phải là số nguyên dương, hiện tại là 0

Quy tắc:

• mỗi samples_per_shard[i] phải là số nguyên dương
• len(samples_per_shard) == 1 hoặc len(samples_per_shard) == số file

Example — arity mismatch:

[english] SamplesPerShardError: Invalid samples_per_shard.

Problem: Received 3 input files but 2 samples_per_shard entries

Rules:

• each samples_per_shard[i] is a positive int
• len(samples_per_shard) == 1 or len(samples_per_shard) == file_count

[vietnam] SamplesPerShardError: samples_per_shard không hợp lệ.

Vấn đề: Nhận 3 file đầu vào nhưng chỉ có 2 samples_per_shard

Quy tắc:

• mỗi samples_per_shard[i] phải là số nguyên dương
• len(samples_per_shard) == 1 hoặc len(samples_per_shard) == số file

Repository layout

mintdim/
├─ pyproject.toml
├─ README.md
├─ LICENSE
├─ .gitignore
│
├─ mintdim/
│  ├─ __init__.py
│  ├─ pipeline.py
│  ├─ registry.py
│  │
│  ├─ shared/
│  │  ├─ __init__.py
│  │  ├─ paths.py
│  │  ├─ errors.py
│  │  └─ types.py
│  │
│  └─ pipelines/
│     ├─ __init__.py
│     │
│     └─ unit_build/
│        ├─ __init__.py
│        ├─ api.py
│        ├─ config.py
│        ├─ contracts.py
│        ├─ validate.py
│        ├─ runtime.py
│        │
│        ├─ source/
│        │  ├─ __init__.py
│        │  ├─ jsonl.py
│        │  ├─ template.py
│        │  └─ validate.py
│        │
│        ├─ tokenizer/
│        │  ├─ __init__.py
│        │  ├─ sentencepiece.py
│        │  ├─ hf_json.py
│        │  └─ validate.py
│        │
│        ├─ units/
│        │  ├─ __init__.py
│        │  ├─ planner.py
│        │  ├─ shard_writer.py
│        │  └─ validate.py
│        │
│        └─ output/
│           ├─ __init__.py
│           ├─ dir.py
│           ├─ manifest.py
│           ├─ sample_index.py
│           ├─ duplicate_index.py
│           ├─ unk_index.py
│           ├─ histogram.py
│           ├─ stats.py
│           └─ validate.py
│
├─ tests/
│  └─ unit_build/
│     ├─ test_api_chain.py
│     ├─ test_validate.py
│     ├─ test_jsonl_source.py
│     ├─ test_template.py
│     ├─ test_tokenizer.py
│     ├─ test_units.py
│     ├─ test_sample_index.py
│     ├─ test_duplicate_index.py
│     ├─ test_unk_index.py
│     └─ test_output.py
│
└─ examples/
   └─ unit_build_instruction.py

Output layout

Shared output layout

./artifacts/instruction_units/
├─ manifest.json
├─ histogram.json
├─ stats.json
├─ sample_index.jsonl
├─ duplicate_index.jsonl
├─ unk_index.jsonl
│
├─ unit_128/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
├─ unit_192/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
├─ unit_256/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
└─ unit_320/
   ├─ shard_000000.bin
   ├─ shard_000001.bin
   └─ ...

Meaning:

manifest.json
→ pipeline metadata, build configuration, tokenizer metadata, and token dtype

histogram.json
→ sample distribution by assigned unit size

stats.json
→ summarized token, shard, duplicate, and UNK statistics

sample_index.jsonl
→ per-sample hash, token count, unit assignment, and shard location

duplicate_index.jsonl
→ grouped duplicate samples by hash

unk_index.jsonl
→ samples containing UNK tokens and their token positions

unit_xxx/
→ binary token shards grouped by unit size

unit_xxx/EMPTY_UNIT.md
→ generated when a declared unit receives no assigned samples

unit_xxx/UNIT_BUILD_FAILED.md
→ generated when the build fails before the unit output is finalized

manifest.json

Example:

{
  "pipeline": "unit-build",
  "mintdim_version": "0.1.0",
  "tokenizer": {
    "type": "sentencepiece",
    "path": "./tokenizer/tokenizer.model",
    "vocab_size": 32000,
    "unk_id": 0,
    "pad_id": 1
  },
  "token_storage": {
    "dtype": "uint16",
    "bytes_per_token": 2
  },
  "units": {
    "sizes": [128, 192, 256, 320],
    "build_batch": 4096
  },
  "source": {
    "files": ["./data/alpaca.jsonl"],
    "fields": [["instruction", "input", "output"]],
    "templates": ["{instruction}\n\n{input}\n\n{output}"]
  },
  "output": {
    "samples_per_shard": 10000
  }
}

histogram.json

Example:

{
  "128": 30291,
  "192": 82191,
  "256": 120882,
  "320": 43121
}

Meaning:

histogram.json counts how many samples were assigned to each unit size.

stats.json

Example:

{
  "total_samples": 1823811,
  "total_tokens": 482918221,
  "token_dtype": "uint16",
  "bytes_per_token": 2,
  "duplicate_groups": 821,
  "duplicate_samples": 1942,
  "samples_with_unk": 321,
  "unk_tokens": 1821,
  "unit_distribution": {
    "128": 582111,
    "192": 891221,
    "256": 301882,
    "320": 48291
  }
}

sample_index.jsonl

Each line describes one source sample after tokenization and unit assignment.

Example:

{
  "sample_id": 91822,
  "hash": "8f2c1b7d9a9d4a0d...",
  "token_count": 143,
  "unit_size": 192,
  "source_file": "./data/alpaca.jsonl",
  "source_line": 18291,
  "shard_path": "unit_192/shard_000001.bin"
}

Purpose:

sample_index.jsonl enables:
- dataset audit
- sample-level reproducibility
- deduplication
- corruption detection
- shard rebuild/debug
- token count verification

duplicate_index.jsonl

Each line groups all duplicated samples sharing the same content hash.

Example:

{
  "hash": "8f2c1b7d9a9d4a0d...",
  "count": 3,
  "samples": [
    {
      "sample_id": 12031,
      "source_file": "./data/alpaca.jsonl",
      "source_line": 991
    },
    {
      "sample_id": 91822,
      "source_file": "./data/alpaca.jsonl",
      "source_line": 18291
    },
    {
      "sample_id": 140992,
      "source_file": "./data/sharegpt.jsonl",
      "source_line": 2012
    }
  ]
}

Purpose:

duplicate_index.jsonl enables:
- duplicate audit
- dataset cleanup
- repeated sample detection
- duplicate filtering
- corruption investigation

unk_index.jsonl

Each line describes one sample whose field values produced UNK tokens. UNK from template literals never reaches this file — that case fails pre-flight with TemplateTokenizerError.

Example:

{
  "sample_id": 91822,
  "hash": "8f2c1b7d9a9d4a0d...",
  "source_file": "./data/alpaca.jsonl",
  "source_line": 18291,
  "unk_count": 2,
  "unk_positions": [17, 89]
}

Purpose:

unk_index.jsonl enables:
- tokenizer quality audit
- unknown token analysis
- tokenizer coverage inspection
- dataset cleanup
- encoding issue detection

Binary token shards

Unit shard files are stored as fixed-width binary token arrays.

Each record in unit_xxx/shard_*.bin occupies exactly unit_size token slots:

slots[0 : token_count]        = encoded tokens
slots[token_count : unit_size] = PAD token (from tokenizer.pad_id)

This guarantees deterministic reader offsets: the k-th sample in a shard starts at byte k * unit_size * bytes_per_token.

Example:

unit_192/shard_000001.bin

If token_dtype is uint16, each token uses 2 bytes (record size = unit_size * 2).

If token_dtype is uint32, each token uses 4 bytes (record size = unit_size * 4).

The shard dtype and pad_id are recorded in manifest.json. Readers may rely on sample_index.jsonl.token_count to know the boundary between encoded tokens and padding.

Multi-output layout

./artifacts/
├─ alpaca_units/
│  ├─ manifest.json
│  ├─ histogram.json
│  ├─ stats.json
│  ├─ sample_index.jsonl
│  ├─ duplicate_index.jsonl
│  ├─ unk_index.jsonl
│  │
│  ├─ unit_128/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  ├─ unit_192/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  ├─ unit_256/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  └─ unit_320/
│     ├─ shard_000000.bin
│     └─ ...
│
└─ sharegpt_units/
   ├─ manifest.json
   ├─ histogram.json
   ├─ stats.json
   ├─ sample_index.jsonl
   ├─ duplicate_index.jsonl
   ├─ unk_index.jsonl
   │
   ├─ unit_96/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_128/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_160/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_192/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_224/
   │  ├─ shard_000000.bin
   │  └─ ...
   └─ unit_320/
      ├─ shard_000000.bin
      └─ ...

Each output directory is isolated and mapped by input order when output.dir has one path per input file. Other blocks may still be shared:

files[i]
↔ fields[0] or fields[i]
↔ templates[0] or templates[i]
↔ tokenizer[0] or tokenizer[i]
↔ sizes[0] or sizes[i]
↔ build_batch[0] or build_batch[i]
↔ output_dir[i]
↔ samples_per_shard[0] or samples_per_shard[i]

Design principles

small public API
explicit paths
explicit user-defined templates (no presets)
required-field sample validation (extra JSONL keys are ignored)
fail fast validation
two-tier UNK contract (template fail vs data record)
exact tokenization
direct unit assignment by token length
fixed-width padded binary shards
automatic token dtype selection
no tokenizer mutation
special token ids loaded from tokenizer library
no automatic repo scanning
no overwrite in V1
no build resume/cache
pipeline-specific internals
sample-level reproducibility
dataset quality audit support

Version

Initial public version:

0.1.5