speculators 0.1.0a6

A unified library for creating, representing, and storing speculative decoding algorithms for LLM serving such as in vLLM.

A Unified Library for Speculative Decoding Algorithms for LLMs

Overview

Speculators is a unified library for creating, representing, and storing speculative decoding algorithms for large language model (LLM) serving, such as in vLLM. It provides a standard format and tools to facilitate the productization of decoding algorithms for inference servers.

Key Features

• Speculative Decoding: Simplify the creation and representation of speculative decoding algorithms for LLMs.
• Standardized Format: Ensure compatibility and ease of use with a standardized format for decoding algorithms.
• Integration Ready: Designed to integrate seamlessly with LLM inference servers like vLLM.
• Productization Tools: Streamline the process of deploying decoding algorithms in production environments.

Getting Started

Installation

Before installing, ensure you have the following prerequisites:

• OS: Linux or MacOS
• Python: 3.9 or higher

The latest Speculators release can be installed using pip:

pip install speculators

Or from source code using pip:

pip install git+https://github.com/neuralmagic/speculators.git

Quick Start

Coming soon

Resources

Documentation

Coming soon

Releases

Visit our GitHub Releases page and review the release notes to stay updated with the latest releases.

License

Speculators is licensed under the Apache License 2.0.

Cite

If you find Speculators helpful in your research or projects, please consider citing it:

@misc{speculators2025,
  title={Speculators: A Unified Library for Speculative Decoding Algorithms in LLM Serving},
  author={Red Hat},
  year={2025},
  howpublished={\url{https://github.com/neuralmagic/speculators}},
}