Skip to main content

Latest News 🎉

2025
2024
  • [2024/11] Support Mono-InternVL with PyTorch engine
  • [2024/10] PyTorchEngine supports graph mode on ascend platform, doubling the inference speed
  • [2024/09] LMDeploy PyTorchEngine adds support for Huawei Ascend. See supported models here
  • [2024/09] LMDeploy PyTorchEngine achieves 1.3x faster on Llama3-8B inference by introducing CUDA graph
  • [2024/08] LMDeploy is integrated into modelscope/swift as the default accelerator for VLMs inference
  • [2024/07] Support Llama3.1 8B, 70B and its TOOLS CALLING
  • [2024/07] Support InternVL2 full-series models, InternLM-XComposer2.5 and function call of InternLM2.5
  • [2024/06] PyTorch engine support DeepSeek-V2 and several VLMs, such as CogVLM2, Mini-InternVL, LlaVA-Next
  • [2024/05] Balance vision model when deploying VLMs with multiple GPUs
  • [2024/05] Support 4-bits weight-only quantization and inference on VLMs, such as InternVL v1.5, LLaVa, InternLMXComposer2
  • [2024/04] Support Llama3 and more VLMs, such as InternVL v1.1, v1.2, MiniGemini, InternLMXComposer2.
  • [2024/04] TurboMind adds online int8/int4 KV cache quantization and inference for all supported devices. Refer here for detailed guide
  • [2024/04] TurboMind latest upgrade boosts GQA, rocketing the internlm2-20b model inference to 16+ RPS, about 1.8x faster than vLLM.
  • [2024/04] Support Qwen1.5-MOE and dbrx.
  • [2024/03] Support DeepSeek-VL offline inference pipeline and serving.
  • [2024/03] Support VLM offline inference pipeline and serving.
  • [2024/02] Support Qwen 1.5, Gemma, Mistral, Mixtral, Deepseek-MOE and so on.
  • [2024/01] OpenAOE seamless integration with LMDeploy Serving Service.
  • [2024/01] Support for multi-model, multi-machine, multi-card inference services. For usage instructions, please refer to here
  • [2024/01] Support PyTorch inference engine, developed entirely in Python, helping to lower the barriers for developers and enable rapid experimentation with new features and technologies.
2023
  • [2023/12] Turbomind supports multimodal input.
  • [2023/11] Turbomind supports loading hf model directly. Click here for details.
  • [2023/11] TurboMind major upgrades, including: Paged Attention, faster attention kernels without sequence length limitation, 2x faster KV8 kernels, Split-K decoding (Flash Decoding), and W4A16 inference for sm_75
  • [2023/09] TurboMind supports Qwen-14B
  • [2023/09] TurboMind supports InternLM-20B
  • [2023/09] TurboMind supports all features of Code Llama: code completion, infilling, chat / instruct, and python specialist. Click here for deployment guide
  • [2023/09] TurboMind supports Baichuan2-7B
  • [2023/08] TurboMind supports flash-attention2.
  • [2023/08] TurboMind supports Qwen-7B, dynamic NTK-RoPE scaling and dynamic logN scaling
  • [2023/08] TurboMind supports Windows (tp=1)
  • [2023/08] TurboMind supports 4-bit inference, 2.4x faster than FP16, the fastest open-source implementation. Check this guide for detailed info
  • [2023/08] LMDeploy has launched on the HuggingFace Hub, providing ready-to-use 4-bit models.
  • [2023/08] LMDeploy supports 4-bit quantization using the AWQ algorithm.
  • [2023/07] TurboMind supports Llama-2 70B with GQA.
  • [2023/07] TurboMind supports Llama-2 7B/13B.
  • [2023/07] TurboMind supports tensor-parallel inference of InternLM.

Introduction

LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by the MMRazor and MMDeploy teams. It has the following core features:

  • Efficient Inference: LMDeploy delivers up to 1.8x higher request throughput than vLLM, by introducing key features like persistent batch(a.k.a. continuous batching), blocked KV cache, dynamic split&fuse, tensor parallelism, high-performance CUDA kernels and so on.

  • Effective Quantization: LMDeploy supports weight-only and k/v quantization, and the 4-bit inference performance is 2.4x higher than FP16. The quantization quality has been confirmed via OpenCompass evaluation.

  • Effortless Distribution Server: Leveraging the request distribution service, LMDeploy facilitates an easy and efficient deployment of multi-model services across multiple machines and cards.

  • Interactive Inference Mode: By caching the k/v of attention during multi-round dialogue processes, the engine remembers dialogue history, thus avoiding repetitive processing of historical sessions.

  • Excellent Compatibility: LMDeploy supports KV Cache Quant, AWQ and Automatic Prefix Caching to be used simultaneously.

Performance

v0 1 0-benchmark

Supported Models

LLMs VLMs
  • Llama (7B - 65B)
  • Llama2 (7B - 70B)
  • Llama3 (8B, 70B)
  • Llama3.1 (8B, 70B)
  • Llama3.2 (1B, 3B)
  • InternLM (7B - 20B)
  • InternLM2 (7B - 20B)
  • InternLM3 (8B)
  • InternLM2.5 (7B)
  • Qwen (1.8B - 72B)
  • Qwen1.5 (0.5B - 110B)
  • Qwen1.5 - MoE (0.5B - 72B)
  • Qwen2 (0.5B - 72B)
  • Qwen2-MoE (57BA14B)
  • Qwen2.5 (0.5B - 32B)
  • Baichuan (7B)
  • Baichuan2 (7B-13B)
  • Code Llama (7B - 34B)
  • ChatGLM2 (6B)
  • GLM4 (9B)
  • CodeGeeX4 (9B)
  • Falcon (7B - 180B)
  • YI (6B-34B)
  • Mistral (7B)
  • DeepSeek-MoE (16B)
  • DeepSeek-V2 (16B, 236B)
  • DeepSeek-V2.5 (236B)
  • Mixtral (8x7B, 8x22B)
  • Gemma (2B - 7B)
  • Dbrx (132B)
  • StarCoder2 (3B - 15B)
  • Phi-3-mini (3.8B)
  • Phi-3.5-mini (3.8B)
  • Phi-3.5-MoE (16x3.8B)
  • MiniCPM3 (4B)
  • LLaVA(1.5,1.6) (7B-34B)
  • InternLM-XComposer2 (7B, 4khd-7B)
  • InternLM-XComposer2.5 (7B)
  • Qwen-VL (7B)
  • Qwen2-VL (2B, 7B, 72B)
  • Qwen2.5-VL (3B, 7B, 72B)
  • DeepSeek-VL (7B)
  • DeepSeek-VL2 (3B, 16B, 27B)
  • InternVL-Chat (v1.1-v1.5)
  • InternVL2 (1B-76B)
  • InternVL2.5(MPO) (1B-78B)
  • Mono-InternVL (2B)
  • ChemVLM (8B-26B)
  • MiniGeminiLlama (7B)
  • CogVLM-Chat (17B)
  • CogVLM2-Chat (19B)
  • MiniCPM-Llama3-V-2_5
  • MiniCPM-V-2_6
  • Phi-3-vision (4.2B)
  • Phi-3.5-vision (4.2B)
  • GLM-4V (9B)
  • Llama3.2-vision (11B, 90B)
  • Molmo (7B-D,72B)
  • Gemma3 (1B - 27B)

LMDeploy has developed two inference engines - TurboMind and PyTorch, each with a different focus. The former strives for ultimate optimization of inference performance, while the latter, developed purely in Python, aims to decrease the barriers for developers.

They differ in the types of supported models and the inference data type. Please refer to this table for each engine's capability and choose the proper one that best fits your actual needs.

Quick Start Open In Colab

Installation

It is recommended installing lmdeploy using pip in a conda environment (python 3.8 - 3.12):

conda create -n lmdeploy python=3.8 -y
conda activate lmdeploy
pip install lmdeploy

The default prebuilt package is compiled on CUDA 12 since v0.3.0. For more information on installing on CUDA 11+ platform, or for instructions on building from source, please refer to the installation guide.

Offline Batch Inference

import lmdeploy
with lmdeploy.pipeline("internlm/internlm3-8b-instruct") as pipe:
    response = pipe(["Hi, pls intro yourself", "Shanghai is"])
    print(response)

[!NOTE] By default, LMDeploy downloads model from HuggingFace. If you would like to use models from ModelScope, please install ModelScope by pip install modelscope and set the environment variable:

export LMDEPLOY_USE_MODELSCOPE=True

If you would like to use models from openMind Hub, please install openMind Hub by pip install openmind_hub and set the environment variable:

export LMDEPLOY_USE_OPENMIND_HUB=True

For more information about inference pipeline, please refer to here.

Tutorials

Please review getting_started section for the basic usage of LMDeploy.

For detailed user guides and advanced guides, please refer to our tutorials:

Third-party projects

  • Deploying LLMs offline on the NVIDIA Jetson platform by LMDeploy: LMDeploy-Jetson

  • Example project for deploying LLMs using LMDeploy and BentoML: BentoLMDeploy

Contributing

We appreciate all contributions to LMDeploy. Please refer to CONTRIBUTING.md for the contributing guideline.

Acknowledgement

Citation

@misc{2023lmdeploy,
    title={LMDeploy: A Toolkit for Compressing, Deploying, and Serving LLM},
    author={LMDeploy Contributors},
    howpublished = {\url{https://github.com/InternLM/lmdeploy}},
    year={2023}
}

License

This project is released under the Apache 2.0 license.

Metadata

Release files for lmdeploy 0.7.2.post1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for lmdeploy 0.7.2.post1
File Actions
lmdeploy-0.7.2.post1-cp312-cp312-win_amd64.whl CPython 3.12 CPython 3.12 Windows x86-64 Details for lmdeploy-0.7.2.post1-cp312-cp312-win_amd64.whl
lmdeploy-0.7.2.post1-cp312-cp312-manylinux2014_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.17+ x86-64 Details for lmdeploy-0.7.2.post1-cp312-cp312-manylinux2014_x86_64.whl
lmdeploy-0.7.2.post1-cp311-cp311-win_amd64.whl CPython 3.11 CPython 3.11 Windows x86-64 Details for lmdeploy-0.7.2.post1-cp311-cp311-win_amd64.whl
lmdeploy-0.7.2.post1-cp311-cp311-manylinux2014_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.17+ x86-64 Details for lmdeploy-0.7.2.post1-cp311-cp311-manylinux2014_x86_64.whl
lmdeploy-0.7.2.post1-cp310-cp310-win_amd64.whl CPython 3.10 CPython 3.10 Windows x86-64 Details for lmdeploy-0.7.2.post1-cp310-cp310-win_amd64.whl
lmdeploy-0.7.2.post1-cp310-cp310-manylinux2014_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ x86-64 Details for lmdeploy-0.7.2.post1-cp310-cp310-manylinux2014_x86_64.whl
lmdeploy-0.7.2.post1-cp39-cp39-win_amd64.whl CPython 3.9 CPython 3.9 Windows x86-64 Details for lmdeploy-0.7.2.post1-cp39-cp39-win_amd64.whl
lmdeploy-0.7.2.post1-cp39-cp39-manylinux2014_x86_64.whl CPython 3.9 CPython 3.9 Linux glibc 2.17+ x86-64 Details for lmdeploy-0.7.2.post1-cp39-cp39-manylinux2014_x86_64.whl
lmdeploy-0.7.2.post1-cp38-cp38-win_amd64.whl CPython 3.8 CPython 3.8 Windows x86-64 Details for lmdeploy-0.7.2.post1-cp38-cp38-win_amd64.whl
lmdeploy-0.7.2.post1-cp38-cp38-manylinux2014_x86_64.whl CPython 3.8 CPython 3.8 Linux glibc 2.17+ x86-64 Details for lmdeploy-0.7.2.post1-cp38-cp38-manylinux2014_x86_64.whl

Total release size: 793.7 MB

Release files / lmdeploy-0.7.2.post1-cp312-cp312-win_amd64.whl

Download URL lmdeploy-0.7.2.post1-cp312-cp312-win_amd64.whl
Size 48.7 MB
Tags CPython 3.12 Windows x86-64
SHA-256 checksum
How to use checksums
d3f995c29dafc4d5d5c3134374ca368be82e77d4473ffc844cc8a473b7d54013
BLAKE2b-256 checksum
How to use checksums
2c4f5e0c2d57d2423a99b0a5f12028c5c861cbb72a86050447a39b9cc9c91360
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.8.18

Release files / lmdeploy-0.7.2.post1-cp312-cp312-manylinux2014_x86_64.whl

Download URL lmdeploy-0.7.2.post1-cp312-cp312-manylinux2014_x86_64.whl
Size 110.1 MB
Tags CPython 3.12 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
342dbfbc6d79e2901befb82bc65fcdc5b47f3f521fe0a7ab1642c169205599a3
BLAKE2b-256 checksum
How to use checksums
0bf2070790012fe476348492808d55a4cab2a8190175aa4e7250bc2cae529c67
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.8.18

Release files / lmdeploy-0.7.2.post1-cp311-cp311-win_amd64.whl

Download URL lmdeploy-0.7.2.post1-cp311-cp311-win_amd64.whl
Size 48.7 MB
Tags CPython 3.11 Windows x86-64
SHA-256 checksum
How to use checksums
a8ad1cf08d45f59581636d53512fca76ae3988efd8395a9863e8ceb07ca4a075
BLAKE2b-256 checksum
How to use checksums
d0737e70cb280fe4992b92e1f73dd13835d2398ea665b4d8f5e4bf00ef53af1d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.8.18

Release files / lmdeploy-0.7.2.post1-cp311-cp311-manylinux2014_x86_64.whl

Download URL lmdeploy-0.7.2.post1-cp311-cp311-manylinux2014_x86_64.whl
Size 110.0 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
439ea952f2b8613d488aaca0f03aff4a8d0e766217871058fb6224033217b5be
BLAKE2b-256 checksum
How to use checksums
9f803324c3def1f542bd8cb3588a4c8b0f7115dd2fd0dd6ac5bd91649dbe5614
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.8.18

Release files / lmdeploy-0.7.2.post1-cp310-cp310-win_amd64.whl

Download URL lmdeploy-0.7.2.post1-cp310-cp310-win_amd64.whl
Size 48.7 MB
Tags CPython 3.10 Windows x86-64
SHA-256 checksum
How to use checksums
72380a0e6175038627b9bfe526dd3920f38a42a19f859d4d6240ec35027475aa
BLAKE2b-256 checksum
How to use checksums
7b48840c24bc88cbaef46d0962b2cd598e4e523c3137c6b9690307de8ddc9414
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.8.18

Release files / lmdeploy-0.7.2.post1-cp310-cp310-manylinux2014_x86_64.whl

Download URL lmdeploy-0.7.2.post1-cp310-cp310-manylinux2014_x86_64.whl
Size 110.0 MB
Tags CPython 3.10 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
ceeb6cafc1c72a4bfc50ac586c4e868b8045bf8e9fc4fcbde9904f9e19507c77
BLAKE2b-256 checksum
How to use checksums
f7e03e3da9dcbc626ed3bf363ee41a9c266819e79c7882c44509fbf20d2354f1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.8.18

Release files / lmdeploy-0.7.2.post1-cp39-cp39-win_amd64.whl

Download URL lmdeploy-0.7.2.post1-cp39-cp39-win_amd64.whl
Size 48.7 MB
Tags CPython 3.9 Windows x86-64
SHA-256 checksum
How to use checksums
dc47ce7c9312029a6c0bace821e71a34820206883eec6ff0018c67ae92403d1d
BLAKE2b-256 checksum
How to use checksums
4b2ba96d376bcdaf800a0fea44dc67590b71f2a7f430d7b66acb1f4367bb931c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.8.18

Release files / lmdeploy-0.7.2.post1-cp39-cp39-manylinux2014_x86_64.whl

Download URL lmdeploy-0.7.2.post1-cp39-cp39-manylinux2014_x86_64.whl
Size 110.0 MB
Tags CPython 3.9 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
d95b9c00df1a33d7e82cbd781d540ef0779a562817b7e4253d953c729fcf05b7
BLAKE2b-256 checksum
How to use checksums
7e97e58d098d52f9890288fede486ec648a166bf4909937724cdc83ae5b93c32
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.8.18

Release files / lmdeploy-0.7.2.post1-cp38-cp38-win_amd64.whl

Download URL lmdeploy-0.7.2.post1-cp38-cp38-win_amd64.whl
Size 48.7 MB
Tags CPython 3.8 Windows x86-64
SHA-256 checksum
How to use checksums
5094e3b8d70a9bc0a40e0ab5260f7a1e216bea5e1eaed4ba73db3b0ee657f91a
BLAKE2b-256 checksum
How to use checksums
cb989bda2131f74f53ff3a6d11a5060bd5b5eb6d9b1635536cfc285dc944e624
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.8.18

Release files / lmdeploy-0.7.2.post1-cp38-cp38-manylinux2014_x86_64.whl

Download URL lmdeploy-0.7.2.post1-cp38-cp38-manylinux2014_x86_64.whl
Size 110.0 MB
Tags CPython 3.8 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
dafece951e2a56f038b31f3e72daa6d7c57230f41b64456e098d0a852788f2fa
BLAKE2b-256 checksum
How to use checksums
80f053c1df898323e3853bf582f8aafac2d78741e1f2935aa9a7f9067a12ac5b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.8.18

Security for lmdeploy 0.7.2.post1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.7.2.post1 This release

10 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page