Skip to main content

docs badge PyPI license issue resolution open issues

English | 简体中文

👋 join us on Twitter, Discord and WeChat


Latest News 🎉

2024
  • [2024/01] OpenAOE seamless integration with LMDeploy Serving Service.
  • [2024/01] Support for multi-model, multi-machine, multi-card inference services. For usage instructions, please refer to here
  • [2024/01] Support PyTorch inference engine, developed entirely in Python, helping to lower the barriers for developers and enable rapid experimentation with new features and technologies.
2023
  • [2023/12] Turbomind supports multimodal input. Gradio Demo
  • [2023/11] Turbomind supports loading hf model directly. Click here for details.
  • [2023/11] TurboMind major upgrades, including: Paged Attention, faster attention kernels without sequence length limitation, 2x faster KV8 kernels, Split-K decoding (Flash Decoding), and W4A16 inference for sm_75
  • [2023/09] TurboMind supports Qwen-14B
  • [2023/09] TurboMind supports InternLM-20B
  • [2023/09] TurboMind supports all features of Code Llama: code completion, infilling, chat / instruct, and python specialist. Click here for deployment guide
  • [2023/09] TurboMind supports Baichuan2-7B
  • [2023/08] TurboMind supports flash-attention2.
  • [2023/08] TurboMind supports Qwen-7B, dynamic NTK-RoPE scaling and dynamic logN scaling
  • [2023/08] TurboMind supports Windows (tp=1)
  • [2023/08] TurboMind supports 4-bit inference, 2.4x faster than FP16, the fastest open-source implementation. Check this guide for detailed info
  • [2023/08] LMDeploy has launched on the HuggingFace Hub, providing ready-to-use 4-bit models.
  • [2023/08] LMDeploy supports 4-bit quantization using the AWQ algorithm.
  • [2023/07] TurboMind supports Llama-2 70B with GQA.
  • [2023/07] TurboMind supports Llama-2 7B/13B.
  • [2023/07] TurboMind supports tensor-parallel inference of InternLM.

Introduction

LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by the MMRazor and MMDeploy teams. It has the following core features:

  • Efficient Inference: LMDeploy delivers up to 1.8x higher request throughput than vLLM, by introducing key features like persistent batch(a.k.a. continuous batching), blocked KV cache, dynamic split&fuse, tensor parallelism, high-performance CUDA kernels and so on.

  • Effective Quantization: LMDeploy supports weight-only and k/v quantization, and the 4-bit inference performance is 2.4x higher than FP16. The quantization quality has been confirmed via OpenCompass evaluation.

  • Effortless Distribution Server: Leveraging the request distribution service, LMDeploy facilitates an easy and efficient deployment of multi-model services across multiple machines and cards.

  • Interactive Inference Mode: By caching the k/v of attention during multi-round dialogue processes, the engine remembers dialogue history, thus avoiding repetitive processing of historical sessions.

Performance

v0 1 0-benchmark

For detailed inference benchmarks in more devices and more settings, please refer to the following link:

  • A100
  • V100
  • 4090
  • 3090
  • 2080

Supported Models

Model Size
Llama 7B - 65B
Llama2 7B - 70B
InternLM 7B - 20B
InternLM2 7B - 20B
InternLM-XComposer 7B
QWen 7B - 72B
QWen-VL 7B
Baichuan 7B - 13B
Baichuan2 7B - 13B
Code Llama 7B - 34B
ChatGLM2 6B
Falcon 7B - 180B
YI 6B - 34B

LMDeploy has developed two inference engines - TurboMind and PyTorch, each with a different focus. The former strives for ultimate optimization of inference performance, while the latter, developed purely in Python, aims to decrease the barriers for developers.

They differ in the types of supported models and the inference data type. Please refer to this table for each engine's capability and choose the proper one that best fits your actual needs.

Quick Start

Installation

Install lmdeploy with pip ( python 3.8+) or from source

pip install lmdeploy

The default prebuilt package is compiled on CUDA 11.8. However, if CUDA 12+ is required, you can install lmdeploy by:

export LMDEPLOY_VERSION=0.2.0
export PYTHON_VERSION=38
pip install https://github.com/InternLM/lmdeploy/releases/download/v${LMDEPLOY_VERSION}/lmdeploy-${LMDEPLOY_VERSION}-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux2014_x86_64.whl

Offline Batch Inference

import lmdeploy
pipe = lmdeploy.pipeline("internlm/internlm-chat-7b")
response = pipe(["Hi, pls intro yourself", "Shanghai is"])
print(response)

For more information about inference pipeline, please refer to here.

Tutorials

Please overview getting_started section for the basic usage of LMDeploy.

For detailed user guides and advanced guides, please refer to our tutorials:

Contributing

We appreciate all contributions to LMDeploy. Please refer to CONTRIBUTING.md for the contributing guideline.

Acknowledgement

License

This project is released under the Apache 2.0 license.

Metadata

Release files for lmdeploy 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for lmdeploy 0.2.2
File Actions
lmdeploy-0.2.2-cp311-cp311-win_amd64.whl CPython 3.11 CPython 3.11 Windows x86-64 Details for lmdeploy-0.2.2-cp311-cp311-win_amd64.whl
lmdeploy-0.2.2-cp311-cp311-manylinux2014_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.17+ x86-64 Details for lmdeploy-0.2.2-cp311-cp311-manylinux2014_x86_64.whl
lmdeploy-0.2.2-cp310-cp310-win_amd64.whl CPython 3.10 CPython 3.10 Windows x86-64 Details for lmdeploy-0.2.2-cp310-cp310-win_amd64.whl
lmdeploy-0.2.2-cp310-cp310-manylinux2014_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ x86-64 Details for lmdeploy-0.2.2-cp310-cp310-manylinux2014_x86_64.whl
lmdeploy-0.2.2-cp39-cp39-win_amd64.whl CPython 3.9 CPython 3.9 Windows x86-64 Details for lmdeploy-0.2.2-cp39-cp39-win_amd64.whl
lmdeploy-0.2.2-cp39-cp39-manylinux2014_x86_64.whl CPython 3.9 CPython 3.9 Linux glibc 2.17+ x86-64 Details for lmdeploy-0.2.2-cp39-cp39-manylinux2014_x86_64.whl
lmdeploy-0.2.2-cp38-cp38-win_amd64.whl CPython 3.8 CPython 3.8 Windows x86-64 Details for lmdeploy-0.2.2-cp38-cp38-win_amd64.whl
lmdeploy-0.2.2-cp38-cp38-manylinux2014_x86_64.whl CPython 3.8 CPython 3.8 Linux glibc 2.17+ x86-64 Details for lmdeploy-0.2.2-cp38-cp38-manylinux2014_x86_64.whl

Total release size: 635.7 MB

Release files / lmdeploy-0.2.2-cp311-cp311-win_amd64.whl

Download URL lmdeploy-0.2.2-cp311-cp311-win_amd64.whl
Size 64.0 MB
Tags CPython 3.11 Windows x86-64
SHA-256 checksum
How to use checksums
a6d5019b77a2a443633318e12e9ab7e575413ec102a9911b6040e6a80c87703a
BLAKE2b-256 checksum
How to use checksums
d4a8717a3c4f0acfb4fc57d63632ac27296e3416ff85689c68cf6314937f6a1b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.8.18

Release files / lmdeploy-0.2.2-cp311-cp311-manylinux2014_x86_64.whl

Download URL lmdeploy-0.2.2-cp311-cp311-manylinux2014_x86_64.whl
Size 94.9 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
b25bd98cddabb84493629b81f2e4cd375d2c3015fdb90d484761988872f0bfe2
BLAKE2b-256 checksum
How to use checksums
71f5a6eb801fe3ffd4ed65545f8d1e9f2a5931d80d1e91cd26c125e0bb88c66b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.8.18

Release files / lmdeploy-0.2.2-cp310-cp310-win_amd64.whl

Download URL lmdeploy-0.2.2-cp310-cp310-win_amd64.whl
Size 64.0 MB
Tags CPython 3.10 Windows x86-64
SHA-256 checksum
How to use checksums
b20b9e93ccd2f19350164d9c24658a2efe347585b430a32c6935042924e664c3
BLAKE2b-256 checksum
How to use checksums
7b268a88c49ce2de9edc538053f93d41d86207b866d07ce2b8dd33676dc20ac3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.8.18

Release files / lmdeploy-0.2.2-cp310-cp310-manylinux2014_x86_64.whl

Download URL lmdeploy-0.2.2-cp310-cp310-manylinux2014_x86_64.whl
Size 94.9 MB
Tags CPython 3.10 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
f335634c8bebb1ac8589df0f1e43422d15040fe9eb8c2501c20e7ace903a7287
BLAKE2b-256 checksum
How to use checksums
7a02d7f3d6145387a3858e32653dbe5d27e546c5cf944a2c586771f7edc932a5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.8.18

Release files / lmdeploy-0.2.2-cp39-cp39-win_amd64.whl

Download URL lmdeploy-0.2.2-cp39-cp39-win_amd64.whl
Size 64.0 MB
Tags CPython 3.9 Windows x86-64
SHA-256 checksum
How to use checksums
c6c8ebe1291a41690af83eb2b0646890a39084e020f96b1372f9034c2a43a1c2
BLAKE2b-256 checksum
How to use checksums
db85c27c194406f60631a3bdcece2c17522cb0d477596c1bbc7216038b891fad
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.8.18

Release files / lmdeploy-0.2.2-cp39-cp39-manylinux2014_x86_64.whl

Download URL lmdeploy-0.2.2-cp39-cp39-manylinux2014_x86_64.whl
Size 94.9 MB
Tags CPython 3.9 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
a4cb4091fd5a38f881cb169e8eb128e04185c9cb7b9d32b040f1b6028677f08c
BLAKE2b-256 checksum
How to use checksums
a66f1c84951a91f7b02163e6ca89e9f97e4881b9baa48b9e7ae65f153e1f6078
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.8.18

Release files / lmdeploy-0.2.2-cp38-cp38-win_amd64.whl

Download URL lmdeploy-0.2.2-cp38-cp38-win_amd64.whl
Size 64.0 MB
Tags CPython 3.8 Windows x86-64
SHA-256 checksum
How to use checksums
fe5920fb36c4e05beb6cddfbd8b9aab7aa443abd4a2807fbdb9d459ea9566e41
BLAKE2b-256 checksum
How to use checksums
9de68e901bd244ec03b6b26738bc24304a10d0b97f1d3c5d2caf340aedb1d59d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.8.18

Release files / lmdeploy-0.2.2-cp38-cp38-manylinux2014_x86_64.whl

Download URL lmdeploy-0.2.2-cp38-cp38-manylinux2014_x86_64.whl
Size 94.9 MB
Tags CPython 3.8 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
73c9de95264fe18f75f2af4c4356ae65be5735026188b3f255d2ff716c26b06f
BLAKE2b-256 checksum
How to use checksums
9121f00b155d21cb48911823060d6bb4ef3f46a38cc86ef13da0aef269b2ec08
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.8.18

Security for lmdeploy 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.2 This release

8 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page