Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

Pixel Reading (pixr)

OCR-free, end-to-end document understanding models trained on open data: pixels in, text out. pixr covers the models (image encoder + text decoder families), the training tasks (page reading pretraining, DocVQA fine-tuning), generation (KV cached greedy / beam search) and a lightweight PyTorch training framework (DDP / FSDP2, compile, EMA). Data pipelines are provided by chugr.

pixr is the continuation of the original pixparse project (huggingface/pixparse, 2023), renamed and revived.

Status

Alpha, expect breaking changes. The current focus is reproducing decent OCR / DocVQA results w/ open data and modern (NaFlex, native aspect ratio) image encoders.

Changes since pixparse

  • Ported to chugr 0.2, NaFlexVit (native aspect / variable resolution) image encoders, and self-contained text decoders (BART, Ettin, Qwen3) w/ three model families (see below)
  • FSDP2, torch.compile, EMA, NaFlex batching, chunked LM loss, KV cached greedy / beam search generation
  • Fixes for issues that hampered the original DocVQA results:
    • DocVQA fine-tune loss was applied to question + answer tokens (prompt masking used the task start token)
    • train / eval image preprocessing mismatched (aspect preserving resize + pad vs stretch, long axis rotation)
    • DocVQA eval decoded by re-tokenizing the generated string each step (BPE round trip is not idempotent)
    • scale_embedding was forced on for BART decoders (copied from Donut's mBART, BART is trained w/o)
    • ModelArgs command line overrides (image size, etc) were silently dropped
    • pretrained BART cross-attention weights significantly slow learning w/ an image encoder, cross-attention is now re-initialized w/ zero init output projections by default (decoder starts as the pretrained LM)

Installation

pip install --pre pixr   # pulls in chugr
pip install --pre "pixr[wandb]"   # optional logging extras: wandb, tensorboard

NOTE the NaFlexVit attention backends (attn_backend) and pretrained patch embedding channel adaptation used by the NaFlex models need timm changes that are not in a release yet, until they are, install timm from source w/ them.

From a source checkout (the training / eval scripts and the test launcher live in the repository):

pip install -e ./chugr -e ./pixr

Models

Three model families share the same tasks, data pipelines, loss masking and (greedy / beam search, KV cached) generation:

  • Cruller -- image encoder + autoregressive text decoder w/ cross-attention (Donut style)
  • Fritter -- image encoder -> (optional m x m token merge) -> projector -> image token prefix for a decoder-only LM w/ prefix-LM attention (PaLI / LLaVA style)
  • Cronut -- native decoder-only transformer over image patches and text (GenLIP style single trunk w/ 2D/1D interleaved MRoPE, qk-norm, gated attention, SwiGLU), trained from scratch w/ a tiktoken tokenizer

Model configs live in src/pixr/models/configs:

config image encoder text decoder
cruller_naflex_clip_base NaFlexVit B/16, CLIP style (pre-norm, cls token), OpenAI CLIP init, 4096 patches BART base (6L)
cruller_naflex_clip_base_8k as above, 8192 patches BART base (6L)
cruller_naflex_clip_base_ettin150m as above, 4096 patches Ettin (ModernBERT) decoder 150M + cross-attn
cruller_naflex_siglip2_base NaFlexVit B/16 SigLIP2 BART base (6L)
cruller_naflex_clip_large NaFlexVit L/14 CLIP style, OpenAI CLIP init, 6144 patches BART large (8L)
cruller_naflex_clip_large_ettin400m as above Ettin decoder 400M + cross-attn
cruller_naflex_siglip2_base_ettin150m NaFlexVit B/16 SigLIP2 Ettin decoder 150M + cross-attn
cruller_naflex_siglip2_so400m NaFlexVit SO400M/16 SigLIP2 BART large (8L)
fritter_naflex_siglip2_base_ettin150m NaFlexVit B/16 SigLIP2, 2x2 token merge Ettin decoder 150M (prefix-LM)
fritter_naflex_siglip2_base_ettin400m as above Ettin decoder 400M (prefix-LM)
fritter_naflex_siglip2_so400m_ettin1b NaFlexVit SO400M/16 SigLIP2, 2x2 token merge Ettin decoder 1B (prefix-LM)
cruller_naflex_siglip2_base_qwen3_0.6b NaFlexVit B/16 SigLIP2 Qwen3 0.6B + zero init cross-attn every N layers
fritter_naflex_siglip2_base_qwen3_0.6b NaFlexVit B/16 SigLIP2, 2x2 token merge Qwen3 0.6B (prefix-LM)
cronut_s16_tiktoken / cronut_b16_tiktoken -- (patch embed in trunk) -- (single trunk), cl100k_base
cruller_vit_clip_base_1280 fixed size ViT B/16 CLIP @ 1280x960 BART base (6L)
cruller_hybrid_r50_base_1280 fixed size hybrid R50 + ViT B/16 @ 1280x960 BART base (6L)

NaFlex models consume pages at their native aspect ratio, resized to fit a patch budget (--task.model.max-seq-len), the budget can be changed between pretraining and fine-tuning.

Ettin (ModernBERT) decoders use 64 token sliding window attention in 2 of every 3 layers, a poor fit for a long image prefix (Fritter), Qwen3 (full attention) is the recommended prefix-LM decoder.

Attention backends

Padded NaFlex sequences can use a dense mask or a variable length kernel, selected per model config field:

field options
image_encoder.attn_backend sdpa (dense mask), varlen (packed, CUDA), flex (FlexAttention block mask), batches w/o padding skip masking for all -- requires the timm naflex-attn-backends branch
text_decoder.cross_attn_backend sdpa, varlen, flex
text_decoder.self_attn_backend sdpa (dense prefix-LM mask), flex (prefix-LM block mask), Ettin / Qwen3
cronut.attn_backend sdpa, flex

Override from the command line w/ --task.model.image-attn-backend, --task.model.cross-attn-backend, and --task.model.self-attn-backend (decoder self-attention, or the Cronut trunk), e.g. --task.model.image-attn-backend flex.

Tokenizers

HF tokenizers by name (facebook/bart-base, Qwen/Qwen3-0.6B, ...) or tiktoken encodings via tiktoken:<encoding> (tiktoken:cl100k_base, tiktoken:o200k_base, ...), w/ eos / pad / bos and task tokens appended above the base vocab.

Usage

Training / evaluation apps

The apps are installed as pixr-train / pixr-eval (or python -m pixr.app.train / python -m pixr.app.eval). Distributed runs use torchrun (or python -m torch.distributed.run):

torchrun --nproc_per_node 2 -m pixr.app.train --task read --task.model.name cruller_naflex_siglip2_base \
  --train-data.source "<idl-wds shards>" --train-data.num-samples 200000 --trainer.num-intervals 40 ...
pixr-eval --task docvqa --checkpoint-path <checkpoint.pt> \
  --eval-data.source pixparse/docvqa-single-page-questions --eval-data.split validation --eval-data.format hfds

Scripts in scripts/ (source checkout) cover the main workflow w/ tuned defaults, see each script for env var options:

# 1. pretrain (read all text on a page) on IDL, streamed from the HF hub
NUM_GPUS=2 MODEL=cruller_naflex_siglip2_base ./scripts/pretrain_idl.sh

# held out reading CER / WER
CHECKPOINT=output/pretrain_cruller_naflex_siglip2_base_idl/checkpoints/checkpoint-latest.pt ./scripts/eval_ocr.sh

# 2. fine-tune on DocVQA (ANLS on validation each eval interval)
CHECKPOINT=output/pretrain_cruller_naflex_siglip2_base_idl/checkpoints/checkpoint-latest.pt ./scripts/finetune_docvqa.sh

# 3. evaluate (validation ANLS, or SPLIT=test for an eval server submission file)
CHECKPOINT=output/finetune_docvqa/checkpoints/checkpoint-latest.pt ./scripts/eval_docvqa.sh

Command line / config

Apps are configured by dataclasses parsed w/ jsonargparse:

  • --task read|docvqa selects the task (and the type of the task config section), default read for train
  • nested fields map to dotted args, written w/ dashes: --trainer.opt.learning-rate 1e-4 (underscores also accepted)
    • --task.* -- task: model, tokenizer, image transforms, dtype / amp, max text length, eval generation
    • --trainer.* -- optimization: intervals, warmup, optimizer / schedule (--trainer.opt.*), EMA, compile, grad ckpt, FSDP
    • --train-data.* / --eval-data.* -- chugr data args (source, format, split, batch size, workers, NaFlex batching)
  • --config file.yaml loads a config (CLI args override), --print-config dumps the resolved config, --help lists all
  • lists are passed as --train-data.naflex-seq-lens [1024,2048,4096]

The full resolved config is written to <output_dir>/<experiment>/config.yaml and can be passed back via --config.

Distributed / compile

Distributed runs (torch.distributed.run) use DDP by default, --trainer.fsdp true shards params, grads and optimizer state w/ FSDP2 (fully_shard per encoder block / decoder layer + root, models list their blocks in fsdp_shard_modules()). The handling follows the OpenCLIP TrainingTask (prepare_distributed / prepare_fsdp / compile on the task):

  • FSDP mixed precision is a MixedPrecisionPolicy (bf16 params, fp32 gradient reduction), autocast is suppressed
  • --trainer.grad-checkpointing w/ FSDP applies composable activation checkpointing per block (inline otherwise)
  • --trainer.compile true --trainer.compile-strategy task|model|blocks: task compiles the task training forward (model + loss), model the model before DDP, blocks the blocks in place before DDP (checkpoint recompute stays in eager autograd, bounding compiled backward memory). W/ FSDP, blocks are always compiled inside prepare_fsdp(), ordered compile -> composable checkpointing -> fully_shard (checkpoint hooks on the compiled wrapper so recompute re-enters the FSDP all-gather hooks), --compile-mode / --compile-dynamic / --compile-backend pass through
  • eval and train metrics run w/ the params gathered (task.full_params()), so ranks decode independently
  • checkpoints are gathered full state dicts (model, EMA, optimizer), loadable w/o FSDP, model / EMA weights are interchangeable between FSDP and DDP runs (optimizer state is not)

scripts/check_distributed.py checks single / DDP / FSDP parity (loss trajectories, final params), grad checkpointing, accumulation, compile, EMA, resume and generation through the Trainer on synthetic batches.

Generation

pixr.generation provides batched, KV cached greedy (generate_greedy) and beam search (generate_beam) decoding, --task.eval-num-beams selects beam search for eval. Cruller, Fritter and Cronut share one incremental decoding interface:

memory = model.encode_image(image_input)                         # per image, computed once
logits, state = model.prefill(memory, prompt_ids, prompt_mask)   # left padded prompts
logits, state = model.step(token_ids, state)                     # one token per hypothesis
state.reorder(parent_idx)                                        # select / duplicate hypotheses (beam search)

The per image memory and the per hypothesis sequence state are separate (pixr/models/kv_cache.py). Cruller's cross-attention K / V are projected once inside each decoder layer and shared by all hypotheses of an image (several prompts or beams per image, image-major order), only the self-attention caches are reordered. Under FSDP, generation must run inside task.full_params() (ranks then decode independently).

LR scaling

Params are grouped by component via the model's param_layer_map(): encoder (image encoder), bridge (cross-attention / prefix projector), decoder (text decoder / LM trunk). Each has an LR multiplier (--trainer.opt.encoder-lr-scale, --trainer.opt.bridge-lr-scale, --trainer.opt.decoder-lr-scale) and the encoder / decoder support layer-wise LR decay (--trainer.opt.encoder-layer-decay 0.75, --trainer.opt.decoder-layer-decay). --trainer.opt.lr-scale-overrides '{"embed_tokens": 2.0}' adds multipliers by param name regex. Training pretrained encoders at full LR from the start wrecks their features before the decoder learns to use them (reading never takes off), scripts/pretrain_idl.sh uses ENC_LR_SCALE=0.1.

LM loss

The LM head + cross-entropy is fused and chunked over tokens by default (--task.loss-chunk-size 4096, 0 to materialize full logits), padding is dropped before the head GEMM and each chunk's logits are recomputed in backward, so the [tokens, vocab] logits (~150k vocab for Qwen3) are never materialized. --task.z-loss-weight (e.g. 1e-4) adds a z-loss (mean squared logsumexp), --task.label-smoothing is supported by both paths.

NaFlex batching

For NaFlex models the train loader can draw a sequence length per batch and size the batch to a token budget, w/ samples resized to the batch's patch budget (rank-identical schedule, from OpenCLIP's streaming NaFlex loader):

--train-data.naflex-seq-lens [1024,2048,4096] --train-data.naflex-seq-len-probs [0.25,0.25,0.5] \
--train-data.naflex-max-tokens 65536 --train-data.naflex-text-tokens 512 --trainer.opt.loss-batch-scale sqrt

--train-data.length-bucketing true groups samples of similar target text length. Neither requires resampled shards.

Checkpoints include the resolved model config and the tokens added to the tokenizer, so fine-tuning and evaluation only need --checkpoint-path, task tokens are appended and their embeddings initialized from the mean embedding.

Code Organization

Within src/pixr:

  • app/ - CLI applications for training and evaluation
    • app/train.py - main training CLI entrypoint, will attempt to keep useable across tasks
    • app/eval.py - main evaluation CLI entrypoint
  • data/ - data loaders, image and text preprocessing
  • framework/ - lightweight train & evaluation scaffolding on top of canonical PyTorch
    • framework/task.py - TrainingTask base (model + objective + eval)
    • framework/trainer.py - Trainer (optimizer, LR schedule, AMP, grad accumulation, DDP / FSDP2 / compile setup, EMA, train state)
  • models/ - modelling code with associated factory methods and helpers
  • task/ - tasks (read pretraining, docvqa fine-tune), registered in task/task_registry.py
  • tokenizers/ - tokenizer creation and task token handling
  • generation.py - batched, KV cached greedy and beam search decoding (--task.eval-num-beams)
  • utils/ - misc utils that don't have a home (utils/cli.py -- jsonargparse config / CLI parsing)

Concepts & Terminology

Some terms and concepts used in this project that may be a bit unfamiliar.

Task

A key organization concept in this project. Package the model with its loss, pre/post-processing, and optimization setup together for a given objective.

Examples of tasks conceptually:

  • Pretraining a Donut style (image enc + text dec) model on supervised (OCR annotation) doc-text dataset
  • Pretraining a Pix2Struct style (image enc + text dec) model w/ a dataset of webpage/screenshots and structured, simplified HTML
  • Pretraining a PaLI style (image enc + text enc-dec) model w/ prefix & masked-token completion on datasets as above
  • Fine-tuning any of the above pretrained models on a possibly large variety of downstream tasks
    • Semi-structured doc parsing - receipt, invoice, business cards, etc.
    • VQA
    • Captioning
    • ... and more

With the Task concept, the data pipeline exists outside the task. The data pipeline is coupled to the task by passing the pre-processing functions created within the task to the data pipeline on creation (task.train_data_cfg() / task.eval_data_cfg()).

A task is a TrainingTask (an nn.Module wrapping the model) implementing training_forward(batch) -> (outputs, losses), optional train_metrics(), and evaluate(loader). The generic Trainer owns everything else (optimizer, schedule, AMP, accumulation, setup ordering for DDP / FSDP2 / compile, EMA, checkpointing state). Tasks are agnostic to the model family.

Interval

You'll see the term 'interval' in the code, sometimes next to epoch. It's related, but an epoch means 'one complete pass of the dataset', an interval may be an epoch, but it may not. Interval is a span of training between checkpoints, ideally meaningful enough in duration to warrant evaluating and archiving each interval checkpoint.

In OpenCLIP development the term arose when using shard sampling with replacement, where the intervals between checkpoints were determined by limitations on job durations or likelihood of crashes.

Development

Tests: pytest tests (tests/test_distributed.py runs FSDP2 / DDP on a single rank NCCL process group, skipped w/o CUDA).

For short test / train / eval runs on two GPUs (source checkout), use the scenario launcher from an environment with editable pixr, chugr, and the local timm installed:

python scripts/run_review.py --list
python scripts/run_review.py all --validate                 # parse app args, no GPU jobs
python scripts/run_review.py smoke --dry-run                # inspect commands
python scripts/run_review.py smoke --devices 0,1
python scripts/run_review.py fsdp_compile naflex_repeat prefix_flex
python scripts/run_review.py data_ranks                     # check rank streams are distinct
python scripts/run_review.py tests parity
python scripts/run_review.py all

The launcher runs scenarios sequentially through pixr.app.train, pixr.app.eval, pytest and scripts/check_distributed.py. Scenarios cover DDP / FSDP2, gradient accumulation, compile, checkpointing, EMA, NaFlex batching, all three model families, attention backends, checkpoint resume and DocVQA fine-tuning. data_ranks checks actual resampled WDS sample identities across ranks, without loading a model or decoding PDFs. It fails if any two ranks receive an identical stream. App runs also fail if no updates or non-finite losses are logged. parity compares FP32 DDP / FSDP2 model and EMA parameters and fails above a relative L2 difference of 1e-5; its resumed losses and generated outputs are also recorded for inspection.

Defaults are short execution checks: random initialization, 64 global samples per interval, batch 2 per GPU, accumulation 2, 256 image patches and 128 text tokens. --pretrained enables hub weight initialization for the app runs; tokenizers and Qwen3 / Ettin configs can still require downloads without it. pretrained_tests downloads HF reference weights; docvqa uses the HF DocVQA dataset. --samples, --batch-size, --accum, --patches, --text-length, and --workers adjust the workload. These short random-weight runs do not measure convergence.

The default data paths are sibling ../data/idl/idl-train-000{01..07}.tar for training and shard 00000 for eval; override with --train-source / --eval-source. naflex_repeat needs at least GPUs * workers training shards. Logs, exact commands, resolved configs, checkpoints and summary.json go into a new ../runs/review-<timestamp> directory (--output overrides it). Each command has a 20-minute timeout (--timeout in seconds). Failed scenarios are logged and the remaining scenarios continue; --fail-fast stops at the first failure. The launcher exits nonzero on failure and prints a tail -f command for the active log.

Acknowledgements

Model and training design draws on Donut, Pix2Struct, PaLI, LLaVA and GenLIP. Image encoders come from timm, the training task / FSDP2 handling and NaFlex batching follow OpenCLIP, datasets are on the Hugging Face Hub under the pixparse organization.

License

Apache 2.0, see LICENSE.

Metadata

Release files for pixr 0.1.0.dev0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pixr 0.1.0.dev0
File Size Uploaded Provenance Actions
pixr-0.1.0.dev0.tar.gz 105.8 kB Without provenance Details for pixr-0.1.0.dev0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pixr 0.1.0.dev0
File Interpreter ABI Platform Actions
pixr-0.1.0.dev0-py3-none-any.whl Python 3 none any Details for pixr-0.1.0.dev0-py3-none-any.whl

Total release size: 221.9 kB

Release files / pixr-0.1.0.dev0.tar.gz

Download URL pixr-0.1.0.dev0.tar.gz
Size 105.8 kB
Tags Source
SHA-256 checksum
How to use checksums
1506cb7f4049fce53be4197fb6f962bd899328c65aed865428f4ce7a04039221
BLAKE2b-256 checksum
How to use checksums
a39a9ac98d7e892ac93862a732c058da75427b104ee6c5f41968cfb0e5a74455
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / pixr-0.1.0.dev0-py3-none-any.whl

Download URL pixr-0.1.0.dev0-py3-none-any.whl
Size 116.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
52ca8a23c2bea14cb166931bdb402acaf2584a23f1187d07d39c3cdb141b5143
BLAKE2b-256 checksum
How to use checksums
6b18b89fbb4c260fd979ccaf42a5538333a1ae6972bfc4bd05bde3ebeed5c168
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for pixr 0.1.0.dev0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page