Skip to main content

NAT action tokenizer

Turns robot action chunks into discrete tokens for a vision-language-action model, and tokens back into actions. Every 32-step chunk of up to 7 action columns becomes 14 tokens, in time order, so a model can stream actions before it has generated the whole chunk.

Quick start

pip install nat-tokenizer

or from source: pip install "git+https://github.com/ahadjawaid/nat.git".

The weights and config download from the Hugging Face repository ahad-j/nat-action-tokenizer on first use.

import numpy as np
from nat_tokenizer import NATActionTokenizer

tokenizer = NATActionTokenizer.from_pretrained()   # downloads ahad-j/nat-action-tokenizer

# 1. Fit the normalization once, on your dataset's actions (any shape ending in action_dim).
tokenizer.fit(dataset_actions)

# 2. Tokenize 32-step chunks: (batch, 32, action_dim) or one chunk (32, action_dim).
tokens = tokenizer(action_chunks)            # list (one per chunk) of lists of ints in [0, 25600)

# 3. Decode back to actions in your original units.
actions = tokenizer.decode(tokens, action_dim=action_chunks.shape[-1])   # (batch, 32, action_dim)

# 4. Save the fitted tokenizer (keeps your min and max) and load it later the same way.
tokenizer.save_pretrained("my-nat-tokenizer")
tokenizer = NATActionTokenizer.from_pretrained("my-nat-tokenizer")

Normalization (on by default)

The tokenizer was trained on actions in [-1, 1]. With normalize=True (the default) every column is mapped from its min and max to [-1, 1] before encoding (values outside are clipped) and decoded actions are mapped back, so you work in your own units throughout. Set the min and max with fit(dataset_actions), or pass them directly:

tokenizer = NATActionTokenizer.from_pretrained(action_min=[...], action_max=[...])

If your actions are already in [-1, 1], pass normalize=False to both calls: tokenizer(chunks, normalize=False) and tokenizer.decode(tokens, action_dim=d, normalize=False). A column whose min equals its max is encoded as 0 and decoded back to its constant value.

Streaming (causal decoding)

Within a group the 14 tokens are time ordered: the first 4 decode the chunk's first 8 steps, the first 7 its first 16, the first 11 its first 24, and all 14 the whole chunk. A prefix decodes exactly the same first steps as the full chunk, so a model can execute actions while it is still generating:

for steps in (8, 16, 24, 32):
    prefix = [chunk[:tokenizer.prefix_token_count(steps)] for chunk in tokens]
    first_steps = tokenizer.decode(prefix, action_dim=7)      # (batch, steps, 7)

Streaming works for actions of up to 7 columns (one group).

Action dimensions

Columns are split into consecutive groups of up to 7 (8 columns = 7 + 1, 14 = 7 + 7, 21 = 7 + 7 + 7). Each group gets its own 14 tokens and a chunk's tokens are the groups' tokens one after another: tokenizer.tokens_per_chunk(action_dim) gives the count (14 per group). Chunks must be exactly 32 steps long.

GPU and CUDA graphs

tokenizer = tokenizer.to("cuda")                      # eager on the GPU
tokenizer = tokenizer.to("cuda", cuda_graphs=True)    # captures the encoder and decoder once per batch shape

Inputs and outputs stay NumPy. GPU results match the CPU up to floating point (rarely a token on a quantization boundary differs).

API

call does
NATActionTokenizer.from_pretrained(repo_or_folder, action_min=None, action_max=None) load (optionally with your min and max)
tokenizer.fit(actions) per-column min and max of your actions (returns the tokenizer)
tokenizer(chunks, normalize=True) (batch, 32, d) or (32, d) actions to a list of token lists
tokenizer.decode(tokens, action_dim, normalize=True) token lists (whole chunks, or one group's prefix) to (batch, steps, d) actions
tokenizer.prefix_token_count(steps) tokens that decode the first steps (8, 16, 24 or 32)
tokenizer.tokens_per_chunk(action_dim) 14 per group of 7 columns
tokenizer.to(device, cuda_graphs=False) run on a device, optionally with CUDA graphs
tokenizer.save_pretrained(directory) save, with the fitted min and max
tokenizer.vocab_size 25,600 (4 scales x 6,400 codes)

Model

4 latents per chunk, 4 FSQ scales of 6,400 codes (one coarse token per two latents, three finer tokens per latent), trained on a large corpus of robot and human action data (every window once, batch 4,096). The repository holds only the weights (model.safetensors) and a JSON config; the network is plain PyTorch in the nat-tokenizer package.

Reconstruction on robot benchmarks, and speed

Random 32-step chunks of each benchmark's demonstrations, each normalized by its own min and max (MetaWorld's actions clipped to the [-1, 1] its simulator executes). Errors are mean absolute differences between the input and its reconstruction. Speed is in milliseconds per call on an RTX 4090 (shared with another job while measured) and 4 CPU threads, mean and SD over 20 calls.

benchmark chunks x shape MAE in [-1, 1] units MAE max error
LIBERO-10 1000 x (32, 7) 0.0304 0.0210 1.9762
MetaWorld mt4 (clipped) 1000 x (32, 4) 0.0651 0.0651 1.7578
RoboMimic mt4 1000 x (32, 7) 0.0466 0.0404 1.5712
RoboCasa mt4 1000 x (32, 12) 0.0130 0.0107 0.9970
device batch of 32 x 7 chunks encode ms decode ms
cpu, eager 1 11.57 ± 0.39 29.86 ± 4.06
cpu, eager 64 322.06 ± 1.76 263.23 ± 6.67
cuda, eager 1 6.54 ± 1.67 11.07 ± 1.18
cuda, eager 64 7.86 ± 2.41 12.67 ± 1.80
cuda, CUDA graphs 1 0.71 ± 0.57 2.44 ± 0.39
cuda, CUDA graphs 64 1.05 ± 0.35 5.18 ± 2.09

Development

pip install -e ".[test]"
pytest

License

MIT, see LICENSE.

Metadata

Release files for nat-tokenizer 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for nat-tokenizer 0.1.0
File Size Uploaded Provenance Actions
nat_tokenizer-0.1.0.tar.gz 10.5 kB Without provenance Details for nat_tokenizer-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for nat-tokenizer 0.1.0
File Interpreter ABI Platform Actions
nat_tokenizer-0.1.0-py3-none-any.whl Python 3 none any Details for nat_tokenizer-0.1.0-py3-none-any.whl

Total release size: 21.6 kB

Release files / nat_tokenizer-0.1.0.tar.gz

Download URL nat_tokenizer-0.1.0.tar.gz
Size 10.5 kB
Tags Source
SHA-256 checksum
How to use checksums
050f31cae33d4772b6c1507dd27e43e38937f578a573ebed2fb763370da6e438
BLAKE2b-256 checksum
How to use checksums
50942a361cbf00ca905dfc101766ec4f18b2905c196edaa6f87c0a94f8eb328a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.6.2

Release files / nat_tokenizer-0.1.0-py3-none-any.whl

Download URL nat_tokenizer-0.1.0-py3-none-any.whl
Size 11.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f4535ce9c0cdd3f235fd4e28fc98b26d10e653ef3ee94323ceb79b2b87cd6c1b
BLAKE2b-256 checksum
How to use checksums
2e26787a44f97b15f87b5b7bd40ad732007256edb6582fea9f0fe24f8a8e4feb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.6.2

Security for nat-tokenizer 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page