No project description provided
Project description
💥 Speedster
Automatically apply SOTA optimization techniques to achieve the maximum inference speed-up on your hardware.
📖 What is this?
Speedster
is an open-source module designed to speed up AI inference in just a few lines of code. The library boosts your model to achieve the maximum acceleration that is physically possible on your hardware.
We are building a new AI inference acceleration product leveraging state-of-the-art open-source optimization tools enabling the optimization of the whole software to hardware stack. If you like the idea, give us a star to support the project ⭐
The core Speedster
workflow consists of 3 steps:
- Select: input your model in your preferred DL framework and express your preferences regarding:
- Accuracy loss: do you want to trade off a little accuracy for much higher performance?
- Optimization time: stellar accelerations can be time-consuming. Can you wait, or do you need an instant answer?
- Search: the library automatically tests every combination of optimization techniques across the software-to-hardware stack (sparsity, quantization, compilers, etc.) that is compatible with your needs and local hardware.
- Serve: finally,
Speedster
chooses the best configuration of optimization techniques and returns an accelerated version of your model in the DL framework of your choice (just on steroids 🚀).
Installation
Install Speedster and its base requirements:
pip install speedster
Then make sure to install the deep learning compilers to leverage during the optimization:
python -m nebullvm.installers.auto_installer --backends all --compilers all
:warning: For MacOS with ARM processors, please use a conda environment. Moreover, if you want to optimize a PyTorch model, PyTorch must be pre-installed on your environment before proceeding to the next step, please install it from this link.
For more details on the installation step, please visit Installation.
API quick view
Only a single line of code is needed to get your accelerated model:
from speedster import optimize_model
optimized_model = optimize_model(model, input_data=input_data)
Checkout how to define the model
and input_data
parameters depending on which framework you want to use and how to use the optimized model:
PyTorch,
HuggingFace,
TensorFlow,
ONNX.
For more details, please visit also the documentation sections Getting Started and How-to guides.
Documentation
Key concepts
Speedster
's design reflects the mission to automatically master all the available AI acceleration techniques and deliver the fastest AI ever. As a result, Speedster
leverages available enterprise-grade open-source optimization tools. If these tools and communities already exist, and are distributed under a permissive license (Apache, MIT, etc), we integrate them and happily contribute to their communities. However, many tools do not exist yet, in which case we implement them and open-source the code so that the community can benefit from it.
Speedster
is shaped around 4 building blocks and leverages a modular design to foster scalability and integration of new acceleration components across the stack.
- Converter: converts the input model from its original framework to the framework backends supported by
Speedster
, namely PyTorch, TensorFlow, and ONNX. This allows the Compressor and Optimizer modules to apply any optimization technique to the model. - Compressor: applies various compression techniques to the model, such as pruning, knowledge distillation, or quantization-aware training.
- Optimizer: converts the compressed models to the intermediate representation (IR) of the supported deep learning compilers. The compilers apply both post-training quantization techniques and graph optimizations, to produce compiled binary files.
- Inference Learner: takes the best performing compiled model and converts it to the same interface as the original input model.
The compressor stage leverages the following open-source projects:
- Intel/neural-compressor: targeting to provide unified APIs for network compression technologies, such as low precision quantization, sparsity, pruning, knowledge distillation, across different deep learning frameworks to pursue optimal inference performance.
- SparseML: libraries for applying sparsification recipes to neural networks with a few lines of code, enabling faster and smaller models.
The optimizer stage leverages the following open-source projects:
- Apache TVM: open deep learning compiler stack for cpu, gpu and specialized accelerators.
- BladeDISC: end-to-end Dynamic Shape Compiler project for machine learning workloads.
- DeepSparse: neural network inference engine that delivers GPU-class performance for sparsified models on CPUs.
- OpenVINO: open-source toolkit for optimizing and deploying AI inference.
- ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
- TensorRT: C++ library for high performance inference on NVIDIA GPUs and deep learning accelerators.
- TFlite and XLA: open-source libraries to accelerate TensorFlow models.
Community
- Discord: best for sharing your projects, hanging out with the community and learning about AI acceleration.
- GitHub issues: ideal for suggesting new acceleration components, requesting new features, and reporting bugs and improvements.
We’re developing Speedster
together with our community so the best way to get started is to pick a good-first issue
. Please read our contribution guidelines for a deep dive on how to best contribute to our project!
Don't forget to leave a star ⭐ to support the project and happy acceleration 🚀
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for speedster-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76256a7fd99563b1f42bcf8b7eddf371fca35331c7ed909beb1a4653cc448d48 |
|
MD5 | 26f3f12ef86c2695e26f87b63299a27e |
|
BLAKE2b-256 | 748b9e3728c288cd70cc2e3c96816d5aee0c4c9d5a9fcd052b38d077e8145205 |