BM25-Jieba 中文文本搜索
基于 Rust + PyO3 的高性能 BM25 中文文本搜索库，使用 jieba-rs 进行中文分词。
特性
- 🚀 高性能: Rust 实现，采用 倒排索引 + Block-Max WAND 算法加速，比纯 Python 快数倍
- 💾 持久化: 支持存取索引到磁盘 (MessagePack 格式)，无需重复训练
- 🔤 中文分词: 内置 jieba-rs 分词器
- 🎯 精确搜索: 经典 BM25 算法
- 🆔 自定义 ID: 支持绑定外部文档 ID (如数据库主键
u64) - 🔠 大小写混合: 支持大小写不敏感搜索
- 🐍 Python 3.11 ~ 3.14: 支持最新 Python 版本
安装
# 开发模式安装
uv run maturin develop
# 或构建 wheel
maturin build --release
pip install target/wheels/*.whl
快速开始
from bm25_jieba import BM25
# 准备文档
documents = [
"Python是一种广泛使用的高级编程语言",
"机器学习是人工智能的一个分支",
"深度学习是机器学习的子领域",
]
# 创建并训练模型 (可选传入自定义 ID)
bm25 = BM25(k1=1.5, b=0.75)
bm25.fit(documents, ids=[101, 102, 103])
# 搜索 (返回自定义 ID 和分数)
results = bm25.search("机器学习", top_k=3)
for doc_id, score in results:
print(f"ID: {doc_id}, Score: {score:.4f}")
# 保存模型 (无需重复训练)
bm25.save("bm25_model.bin")
# 加载模型
loaded_bm25 = BM25.load("bm25_model.bin")
API 参考
BM25(k1=1.5, b=0.75, lowercase=False)
创建 BM25 实例。
| 参数 | 类型 | 默认值 | 说明 |
|---|---|---|---|
k1 |
float | 1.5 | 词频饱和参数 |
b |
float | 0.75 | 文档长度归一化参数 |
lowercase |
bool | False | 是否将文本转换为小写（大小写不敏感） |
fit(documents: list[str], ids: list[int] = None)
使用文档语料库训练模型。
ids: 可选，与documents长度一致的整数列表 (u64)。- 如果不提供
ids，默认使用0..N作为 ID。
search(query: str, top_k: int = None) -> list[tuple[int, float]]
搜索最相关的文档，返回 (文档 ID, 分数) 列表。
save(path: str)
保存当前索引和配置到文件 (MessagePack 格式)。
load(path: str) -> BM25
从文件加载 BM25 模型。
get_scores(query: str) -> list[float]
获取所有文档的 BM25 分数。
开发
# 安装依赖
uv sync
# 编译并安装
uv run maturin develop
# 运行测试
uv run pytest
# 运行示例
uv run python examples/demo.py
技术栈
| 组件 | 版本 | 用途 |
|---|---|---|
| PyO3 | 0.27.2 | Rust-Python 绑定 |
| maturin | 1.11.5 | 构建工具 |
| jieba-rs | 0.8.1 | 中文分词 |
性能测试
在 Apple M1 上测试 (10,000 文档，每文档约 100 字)：
| 测试项 | 结果 |
|---|---|
| 索引速度 | ~37,000 docs/s |
| 搜索 QPS | ~1,000,000 QPS |
| 搜索延迟 | ~0.001ms |
注：得益于 Block-Max WAND 算法的剪枝优化，搜索性能有数量级提升。
# 运行性能测试
uv run python tests/benchmark.py
算法验证
测试语料库：19 个文档，6 个查询
| 验证项 | 结果 | 说明 |
|---|---|---|
| 公式正确性 | ✅ | 手动计算与实现一致 |
| 排序一致性 | ✅ | 与 rank-bm25 排序完全一致 |
| 绝对分数 | ⚠️ | 因 IDF +1 修正略有差异（符合预期） |
# 运行验证
uv sync --group validation
uv run python tests/validate.py
算法实现说明
本实现采用带 +1 修正的 IDF 公式：
IDF(t) = ln((N - df + 0.5) / (df + 0.5) + 1)
与标准 BM25Okapi 的区别：
| 公式 | 特点 |
|---|---|
标准: ln((N-df+0.5)/(df+0.5)) |
可能产生负 IDF |
本实现: ln(...+1) |
保证 IDF ≥ 0 |
影响：
- ✅ 排序一致 - 与 rank-bm25 等标准实现排序结果相同
- ⚠️ 绝对分数不同 - 因
+1修正，分数值略有差异 - ✅ 数值稳定 - 无负值，无需额外处理
这种变体在只关心相对排序（而非绝对分数）的场景下完全适用。
License
MIT
1 maintainer
Metadata
1 maintainer
Release files for bm25-jieba 0.2.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| bm25_jieba-0.2.2.tar.gz | 33.6 kB | Details |
Built distributions (wheels)
Total release size: 42.2 MB
Release files / bm25_jieba-0.2.2.tar.gz
| Download URL | bm25_jieba-0.2.2.tar.gz |
|---|---|
| Size | 33.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
68297428746c237ad125ec80206c32bdcbd0deaa3e182ebb15c6b8b54182039d
|
|
BLAKE2b-256 checksum How to use checksums |
2da1af6ad6728a023174f4a2676e0fb5e0fff9bc1daa47b370e2d5853fb41e49
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp314-cp314-win_amd64.whl
| Download URL | bm25_jieba-0.2.2-cp314-cp314-win_amd64.whl |
|---|---|
| Size | 3.4 MB |
| Tags | CPython 3.14 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
861fe8f416a6a9e5d01313f924d0706fd374eee49cad22c9924886c0d23b5d21
|
|
BLAKE2b-256 checksum How to use checksums |
dbe8e68ff9cf59fcd64536be5a72eaab18d1f4b8b592e7545674d266910589a3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | bm25_jieba-0.2.2-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 3.7 MB |
| Tags | CPython 3.14 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
1b9f0a2c40a0eea7c9ba263541cffc83ffc4c078691593aebedfe365be3e3662
|
|
BLAKE2b-256 checksum How to use checksums |
8d4c1334c67670edded8c6b20fd3b786effde0d87f2cd76371d52114388d3862
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp314-cp314-macosx_11_0_arm64.whl
| Download URL | bm25_jieba-0.2.2-cp314-cp314-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.4 MB |
| Tags | CPython 3.14 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
dd3de4f99f4bc285d65cd80b960a1b2e1f70431c755538ce22b62a511576aeeb
|
|
BLAKE2b-256 checksum How to use checksums |
34436299ab46edd1677308fc7d6d06d720b86451492ea584bf29d742a51d7e43
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp313-cp313-win_amd64.whl
| Download URL | bm25_jieba-0.2.2-cp313-cp313-win_amd64.whl |
|---|---|
| Size | 3.4 MB |
| Tags | CPython 3.13 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
9e70be2deee723481c612797841e13f62475f7b648346c2373bad03d6fde1c4f
|
|
BLAKE2b-256 checksum How to use checksums |
e49f6da95aced7ce3c845584ccc1d1d45e00c2058cf886dc1fcb93e94e503847
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | bm25_jieba-0.2.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 3.7 MB |
| Tags | CPython 3.13 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
c09d2872934632382f47b9d4f3bac7496516c9530a9071d4e21fa69fff396048
|
|
BLAKE2b-256 checksum How to use checksums |
7cfb048bb51e3c5dda7f7e0c6cb0ec1205167e788b69f2f3b819c8896ff84562
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp313-cp313-macosx_11_0_arm64.whl
| Download URL | bm25_jieba-0.2.2-cp313-cp313-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.4 MB |
| Tags | CPython 3.13 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
f3b6b6dae30284e675edf571fe7f100fc8414226cc63356f4c613e95ca8b569a
|
|
BLAKE2b-256 checksum How to use checksums |
5638f55293b2e0ae9ef3a96811121fa542969a7ffa470e1626258773f0358841
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp312-cp312-win_amd64.whl
| Download URL | bm25_jieba-0.2.2-cp312-cp312-win_amd64.whl |
|---|---|
| Size | 3.4 MB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
064c823a23d5e02315bc770ca224e43078905265e47fa3b46d470ba5f35575ac
|
|
BLAKE2b-256 checksum How to use checksums |
e4d2ce3a4025085411c3b9209d996d390e1311d518b0529b31e8fd0109f0df1c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | bm25_jieba-0.2.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 3.7 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
d95f2da38087fc7f022c4a52c02184eacc9d7ae168e7640cb4bb56b3bbddca40
|
|
BLAKE2b-256 checksum How to use checksums |
16520baa57caac93ff8a7bd241af3ccd30e66a333063f1d999fd21a1742b0ec3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp312-cp312-macosx_11_0_arm64.whl
| Download URL | bm25_jieba-0.2.2-cp312-cp312-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.4 MB |
| Tags | CPython 3.12 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
bc8bde8029a95428630f943757d6d3534174408a5f87cc9f51f79ce07c7559d8
|
|
BLAKE2b-256 checksum How to use checksums |
3d8d5f4f4e770dc6b0a27c5b97683e7e79ab317ab89bab0629e5f0a948cb2569
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp311-cp311-win_amd64.whl
| Download URL | bm25_jieba-0.2.2-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 3.4 MB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
283b9537e4d1dd4ce3e501113e41e44b6109773d11def21f8480bc303385f42a
|
|
BLAKE2b-256 checksum How to use checksums |
b43e9669f8efc574791d2c8e802ea5e03871946f7abf335b945149d11025cc98
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | bm25_jieba-0.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 3.7 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
a1e5996965febd11c119aa285ea8d35d99b46f8495ed7a3b6da0f51b4684323a
|
|
BLAKE2b-256 checksum How to use checksums |
ea395e927ed1824481996dff4d6179d68c544a6c3836d9b2e5874763ab8f00f0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency logRelease files / bm25_jieba-0.2.2-cp311-cp311-macosx_11_0_arm64.whl
| Download URL | bm25_jieba-0.2.2-cp311-cp311-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.5 MB |
| Tags | CPython 3.11 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
68b3f06f4f3424d3ec768314acee92297d6a675d962c85531a28b5fea904a973
|
|
BLAKE2b-256 checksum How to use checksums |
dee025b4d8901da0304d1a02eb1756f7f19546ff02b03a08c63fa09810419c67
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 29, 2026.
Transparency log
