Skip to main content
Discord Email Hours

Low-level Guidance (llguidance)


Performance results from MaskBench



About

This library implements constrained decoding (also called constrained sampling or structured outputs) for Large Language Models (LLMs). It can enforce arbitrary context-free grammar on the output of LLM and is fast - on the order of 50μs of CPU time per token (for 128k tokenizer) with negligible startup costs.

Following grammar formats are supported:

The internal format is most powerful (though Lark-like format is catching up, and there are plans to convert the libraries to use it) and can be generated by the following libraries:

The library can be used from:

Integrations

The library is currently integrated in:

Technical details

See Making Structured Outputs Go Brrr for an overview of the library, including the design decisions, performance, and how it compares to other approaches.

Given a context-free grammar, a tokenizer, and a prefix of tokens, llguidance computes a token mask - a set of tokens from the tokenizer - that, when added to the current token prefix, can lead to a valid string in the language defined by the grammar. Mask computation takes approximately 50μs of single-core CPU time for a tokenizer with 128k tokens. While this timing depends on the exact grammar, it holds, for example, for grammars derived from JSON schemas. There is no significant startup cost.

The library implements a context-free grammar parser using Earley’s algorithm on top of a lexer based on derivatives of regular expressions. Mask computation is achieved by traversing the prefix tree (trie) of all possible tokens, leveraging highly optimized code.

Grammars can be also used to speed up decode via fast-forward tokens.

Comparison and performance

See MaskBench in JSON Schema Bench for detailed performance comparisons.

LM-format-enforcer and llama.cpp grammars are similar to llguidance in that they dynamically build token masks for every step of the decoding process. Both are significantly slower - the former due to clean Python code and the latter due to the lack of a lexer and use of a backtracking parser, which, while elegant, is inefficient.

Outlines builds an automaton from constraints and then pre-computes token masks for all automaton states, potentially making sampling fast but inherently limiting constraint complexity and introducing significant startup cost and memory overhead. Llguidance computes token masks on the fly and has essentially no startup cost. The lexer’s automata in llguidance are built lazily and are typically much smaller, as the context-free grammar imposes the top-level structure.

XGrammar follows an approach similar to llama.cpp (explicit stack-based, character-level parser) with additional pre-computation of certain token masks, similar to Outlines. The pre-computation often runs into seconds, and sometimes minutes. If the pre-computation works well for a given input, the masks are computed quickly (under 8μs in half of masks we tested), however if it doesn't fit the particular input, the mask computation times can run to tens or hundreds of milliseconds.

In llguidance, the full mask computation for a typical JSON schema takes about 1.5ms (for 128k tokenizer). However, very often the "slicer" optimization applies, and thus the avarage mask computation in JSON Schema Bench (2.5M tokens, 10k schemas) is under 50μs, with less than 1% of masks taking longer than 1ms, and 0.001% taking longer than 10ms (but still shorter than 30ms). The optimization doesn't involve any significant pre-computation.

Thus, with 16 cores and a 10ms forward pass, llguidance can handle batch sizes up to 3200 without slowing down the model. (Note that a 10ms forward pass for small batch sizes typically increases to 20ms+ for batch sizes of 100-200.)

Building

If you just need the C or Rust library (llguidance), check the parser directory.

For Python bindings:

  • install python 3.10 or later; very likely you'll need a virtual env/conda
  • run ./scripts/install-deps.sh
  • to build and after any changes, run ./scripts/test-guidance.sh

This builds the Python bindings for the library and runs the tests (which mostly live in the Guidance repo - it will clone it).

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

llguidance-1.8.0.tar.gz (1.2 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

llguidance-1.8.0-cp314-cp314t-win_arm64.whl (2.7 MB view details)

Uploaded CPython 3.14tWindows ARM64

llguidance-1.8.0-cp314-cp314t-win_amd64.whl (2.9 MB view details)

Uploaded CPython 3.14tWindows x86-64

llguidance-1.8.0-cp314-cp314t-win32.whl (2.6 MB view details)

Uploaded CPython 3.14tWindows x86

llguidance-1.8.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ x86-64

llguidance-1.8.0-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl (3.8 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ i686

llguidance-1.8.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.5 MB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ ARM64

llguidance-1.8.0-cp314-cp314t-macosx_11_0_arm64.whl (3.2 MB view details)

Uploaded CPython 3.14tmacOS 11.0+ ARM64

llguidance-1.8.0-cp314-cp314t-macosx_10_12_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.14tmacOS 10.12+ x86-64

llguidance-1.8.0-cp39-abi3-win_arm64.whl (2.8 MB view details)

Uploaded CPython 3.9+Windows ARM64

llguidance-1.8.0-cp39-abi3-win_amd64.whl (2.9 MB view details)

Uploaded CPython 3.9+Windows x86-64

llguidance-1.8.0-cp39-abi3-win32.whl (2.6 MB view details)

Uploaded CPython 3.9+Windows x86

llguidance-1.8.0-cp39-abi3-manylinux_2_39_riscv64.whl (3.6 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.39+ riscv64

llguidance-1.8.0-cp39-abi3-manylinux_2_34_i686.whl (3.3 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.34+ i686

llguidance-1.8.0-cp39-abi3-manylinux_2_31_x86_64.whl (3.1 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.31+ x86-64

llguidance-1.8.0-cp39-abi3-manylinux_2_31_aarch64.whl (2.9 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.31+ ARM64

llguidance-1.8.0-cp39-abi3-macosx_11_0_arm64.whl (3.2 MB view details)

Uploaded CPython 3.9+macOS 11.0+ ARM64

llguidance-1.8.0-cp39-abi3-macosx_10_12_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.9+macOS 10.12+ x86-64

File details

Details for the file llguidance-1.8.0.tar.gz.

File metadata

  • Download URL: llguidance-1.8.0.tar.gz
  • Upload date:
  • Size: 1.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for llguidance-1.8.0.tar.gz
Algorithm Hash digest
SHA256 18d1579eabb040e65c870d50c6df19a7bef140c5260d12ad35b7f0dc446312e0
MD5 d21165ee99286eacde1085200aafbb80
BLAKE2b-256 2027972de1ba4c93072fce816b967972e1d18bc48b04b145b5c77b5bd1dc9662

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0.tar.gz:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp314-cp314t-win_arm64.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp314-cp314t-win_arm64.whl
Algorithm Hash digest
SHA256 941667623a938cb79b9afcf6e94e77f2fea26debef5651c2728d777df5af6f0d
MD5 4a0f97586e815f22bddd068ea3c33bf4
BLAKE2b-256 4ec0677942b7ef8ed1708513d7089334fd623babd797c518718cb585de7e5a05

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp314-cp314t-win_arm64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp314-cp314t-win_amd64.whl.

File metadata

  • Download URL: llguidance-1.8.0-cp314-cp314t-win_amd64.whl
  • Upload date:
  • Size: 2.9 MB
  • Tags: CPython 3.14t, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for llguidance-1.8.0-cp314-cp314t-win_amd64.whl
Algorithm Hash digest
SHA256 fa5557c0e9abb56257ecbee13a4b8ae61be0529393caac0a7c7d91f5d1737759
MD5 be75227ad473e47d8e32fabf7a608a25
BLAKE2b-256 1ff38ade3f1065cee316e75a914f8b79f70d9223b70dd2a1ca18f7e6d12ed4c3

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp314-cp314t-win_amd64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp314-cp314t-win32.whl.

File metadata

  • Download URL: llguidance-1.8.0-cp314-cp314t-win32.whl
  • Upload date:
  • Size: 2.6 MB
  • Tags: CPython 3.14t, Windows x86
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for llguidance-1.8.0-cp314-cp314t-win32.whl
Algorithm Hash digest
SHA256 8e0cdf3822e1ab12c4aeaae06b2f9e059d6b62aff81e6bfc2600e1b234021f50
MD5 8cde88bb4b90e2fbd12e8acad568b134
BLAKE2b-256 6062d7363194797cd9408d76ef08264d1615a8af2b1b76265e7cf11c2cb6d3d2

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp314-cp314t-win32.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ccbacbb9269b853d51ccb6c47525509b9b9024cd314249eb78225781f1dad6f3
MD5 9cfb69602631bf874704e522f4e8b163
BLAKE2b-256 890b82addb5de9f9d344b6ae0b4f6a586a763d02bb3594b6be670da8fc7e58ca

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 0f1370ffe7e8b6c0c6e6232b60bdff90a04af3b4a2bad1ac77f3f687ec07b952
MD5 44cc7ce75a640aab956fad140f19076c
BLAKE2b-256 ddee9645bca5ea323cc85c53e1b69047a1938942288a3c1fd75d185571ec9bc8

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 46c18c993094617d87644befc9ed5c3287df2121fa1432f47004f7d62ef85663
MD5 cdf4b530d0cc529c962fb13c8be227ee
BLAKE2b-256 6bc05a6d7c0ac9e1003893fb334cfe763d9200aed7396890a5f3a03f8e9f838f

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp314-cp314t-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp314-cp314t-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 06aa5dc26e87de2d152ffb3f0131ec531620963b677f0d73ab05f49adc8c4133
MD5 e7a5d2b29aa36c05b0505aa530ba6873
BLAKE2b-256 b8c485bcfe5cee1daecba57f5a6ac4edfdfc8f1185b4ef2c46391bd87c67d850

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp314-cp314t-macosx_11_0_arm64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp314-cp314t-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp314-cp314t-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 155a83c2273a8b8b83a4d2c96b7a2a2549491d80fb9c8b64d75298f4287076f9
MD5 f85ea792a8fb47e4bea001b1844e5c33
BLAKE2b-256 20386a12b605f763b82afa1fdcd45ca164c539b902c80f34c95bd3308bea6de3

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp314-cp314t-macosx_10_12_x86_64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp39-abi3-win_arm64.whl.

File metadata

  • Download URL: llguidance-1.8.0-cp39-abi3-win_arm64.whl
  • Upload date:
  • Size: 2.8 MB
  • Tags: CPython 3.9+, Windows ARM64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for llguidance-1.8.0-cp39-abi3-win_arm64.whl
Algorithm Hash digest
SHA256 6bf3953d06e7f5e24bd02fa6a89a5b2b88f7e811c0fa0228487d8267ef7cec54
MD5 cf1992ecf98ec7f380818efc9467b20b
BLAKE2b-256 0aa3c34a975b37f686dfa27fa0798bca3432d8d93d2427db6674969b39903791

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp39-abi3-win_arm64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp39-abi3-win_amd64.whl.

File metadata

  • Download URL: llguidance-1.8.0-cp39-abi3-win_amd64.whl
  • Upload date:
  • Size: 2.9 MB
  • Tags: CPython 3.9+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for llguidance-1.8.0-cp39-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 a8837ac2b3bf4c46e1b6363012a22de043b7f8ef013b04d8689d471eb573b766
MD5 4e4e9af99920ee6d39b38b26c7dca103
BLAKE2b-256 574382ae24a80a6157b05cc988f3d15ffa188fe806d09c840c1351bb8d38c57a

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp39-abi3-win_amd64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp39-abi3-win32.whl.

File metadata

  • Download URL: llguidance-1.8.0-cp39-abi3-win32.whl
  • Upload date:
  • Size: 2.6 MB
  • Tags: CPython 3.9+, Windows x86
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for llguidance-1.8.0-cp39-abi3-win32.whl
Algorithm Hash digest
SHA256 b5e866d8a896e255f30ec952f5280c61a3d6f391a9dce575ce976dd58f0b7000
MD5 7a2a2ad6f681e61bdd8f6761ce9377d9
BLAKE2b-256 ccfec714cef8a976fcb8c68f366cb26120fad0bf6eff5beddbc227d8f3128ea9

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp39-abi3-win32.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp39-abi3-manylinux_2_39_riscv64.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp39-abi3-manylinux_2_39_riscv64.whl
Algorithm Hash digest
SHA256 0eb7be70bf822e54cd4021bb200cfe2b86e4f3379d251067dc9f7da327f3ceab
MD5 dc7fdad74f770172ac1b255efac8291c
BLAKE2b-256 751655527f58cbb2ea1aaff147b49362b331c502c25687d0934020b49f52d749

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp39-abi3-manylinux_2_39_riscv64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp39-abi3-manylinux_2_34_i686.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp39-abi3-manylinux_2_34_i686.whl
Algorithm Hash digest
SHA256 79b0576991b8fc7534456b65c41d43c3183c8ca974a17799359af969c1489c07
MD5 c8c3cf6e1d8a23a1ed5b3511ff16b8ae
BLAKE2b-256 a8c55cce5e3880f42b522d0d231244bda3604083b5deae2d4f582bfd14f48208

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp39-abi3-manylinux_2_34_i686.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp39-abi3-manylinux_2_31_x86_64.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp39-abi3-manylinux_2_31_x86_64.whl
Algorithm Hash digest
SHA256 39668c11396896e5f05f59b70c81e4afd060b3408f02c8518b7a6943bfbb8a5d
MD5 c5e536e0322b3cb0b4f78eedb6482858
BLAKE2b-256 d197208f22f995c64fd3232db41d0004aece45fecaf267b97cf5dd5943ef967e

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp39-abi3-manylinux_2_31_x86_64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp39-abi3-manylinux_2_31_aarch64.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp39-abi3-manylinux_2_31_aarch64.whl
Algorithm Hash digest
SHA256 6ae4343bd40b88d1dd824a17edcee11b5e5a000b16b6fedb9fcf7f58d019177c
MD5 5f74f40f1b017e7e1988bccdc320156d
BLAKE2b-256 32aadb393283f934e8a22e2b8f750924eb17ba6d320f160174fcdee0223a107c

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp39-abi3-manylinux_2_31_aarch64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 bb9a89e8cdd7c8b5cf4f84e45b04177e79acdcc4d5116fbc775e511f7314df44
MD5 5b7adb00f7c2607f45aff600efecb977
BLAKE2b-256 7898568132150d3f6f09200f389191f4f863b405f1c46429dd8d27a120c1518d

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp39-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file llguidance-1.8.0-cp39-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for llguidance-1.8.0-cp39-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 020b4ec2254a20555e69095c7d488907f17d7e65ab5a840b267530f2cb369f70
MD5 ca04a7974d7f19aee6ad5010381ee151
BLAKE2b-256 0c7bcfd52b0b836c07c6b7d83c227f4fb82e85677fbc1087c021ac8d8a69b05d

See more details on using hashes here.

Provenance

The following attestation bundles were made for llguidance-1.8.0-cp39-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on guidance-ai/llguidance

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page