agent-data-tools
项目路径:https://github.com/NairongZheng/agent_data_tools
一个面向 Agent 训练数据的 Python 包,提供三类核心能力:
- 校验
openai messages中间格式 - 转换为
conversations训练格式 - 转换为
swift messages训练格式
这个仓库已经重构为标准的 src 布局,可以直接发布到 PyPI,也可以本地通过 pip install -e . 使用。
项目结构
agent_data_tools/
├── pyproject.toml
├── README.md
├── docs/
│ ├── architecture.md
│ ├── formats.md
│ └── templates.md
├── src/
│ └── agent_data_tools/
│ ├── __init__.py
│ ├── builtin_templates/
│ ├── cli.py
│ ├── converters.py
│ ├── models.py
│ ├── schemas.py
│ ├── template_loader.py
│ ├── utils.py
│ ├── validators.py
│ └── py.typed
├── tests/
│ └── test_api.py
└── data_examples/
安装
直接安装:
pip install -e .
安装 pypi 包:
pip install agent-data-tools
快速开始
使用 cli
agent-data-tools validate-openai --data-list datasets/meta.json --log-dir logs
agent-data-tools convert-conversations --data-list datasets/meta.json --output-dir outputs/convs
agent-data-tools convert-swift-messages --data-list datasets/meta.json --output-dir outputs/messages
agent-data-tools convert-conversations --data-list datasets/meta.json --output-dir outputs/convs --template-config ./templates.json
agent-data-tools list-templates
传入 --template-config 时,仅使用配置文件中的模板;未传时默认使用内置模板,其中默认实际生效的是 qwen3.5_35ba3b_think_xml.jinja。
使用代码
from agent_data_tools import batch_convert_from_meta
meta_path = batch_convert_from_meta(
"datasets/meta.json",
"outputs/conversations",
target="conversations",
max_workers=4,
template_config_path="./templates.json",
)
print(meta_path)
公开 API
常用函数:
validate_function_definitionvalidate_openai_messages_recordvalidate_openai_messages_filevalidate_meta_fileconvert_record_to_conversationsconvert_record_to_swift_messagesconvert_jsonl_filebatch_convert_from_metalist_builtin_template_namesresolve_template_specs
结果对象:
ValidationResultConversionStats
设计原则
- 以
openai messages作为统一中间格式 - 将“校验”“模板加载”“转换”“批处理”拆分为独立模块
- 默认内置模板开箱即用,同时允许用户传入模板配置 JSON 覆盖默认模板选择
- conversations 与 swift messages 共用同一套模板渲染入口
文档
- 结构设计:见 docs/architecture.md
- 数据格式说明:见 docs/formats.md
- 模板使用说明:见 docs/templates.md
- 发布与验收:见 docs/release.md
验证
PYTHONPATH=src python -m unittest tests/test_api.py
PYTHONPATH=src python -m agent_data_tools.cli list-templates
PYTHONPATH=src python -m agent_data_tools.cli --help
发布
构建分发包:
python -m pip install build
python -m build
上传前检查:
python -m pip install twine
python -m twine check dist/*
完整发布说明见 docs/release.md。
Metadata
Release files for agent-data-tools 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agent_data_tools-0.1.5.tar.gz | 19.6 kB | Without provenance | Details for agent_data_tools-0.1.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agent_data_tools-0.1.5-py3-none-any.whl | Python 3 | none | any | Details for agent_data_tools-0.1.5-py3-none-any.whl |
Total release size: 44.1 kB
Release files / agent_data_tools-0.1.5.tar.gz
| Download URL | agent_data_tools-0.1.5.tar.gz |
|---|---|
| Size | 19.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
22be45c3fe9569bbb400a924a2093f9a383cf15dd080bd306ee03fc973aad7c3
|
|
BLAKE2b-256 checksum How to use checksums |
311efe573dc1cb4975dcf56b4b2f2b565c7f8a4e5b449609a774a31265ccd3f7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.11
|
Release files / agent_data_tools-0.1.5-py3-none-any.whl
| Download URL | agent_data_tools-0.1.5-py3-none-any.whl |
|---|---|
| Size | 24.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5c997b728a19a663e79168cfae7d0f7725bf1c54cee53a223651bbee02eb3777
|
|
BLAKE2b-256 checksum How to use checksums |
022e68f77de5c32018e386982f4a727c1e1a61851ec1542cdc45c5d1753382e7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.11
|
Security for agent-data-tools 0.1.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.