DataMax

中文 | English

Documentation Portal: https://hi-dolphin.github.io/datamax

A powerful multi-format file parsing, data cleaning, and AI annotation toolkit built for modern Python applications.

✨ Key Features

🔄 Multi-format Support: PDF, DOCX/DOC, PPT/PPTX, XLS/XLSX, HTML, EPUB, TXT, images, and more
🧹 Intelligent Cleaning: Advanced data cleaning with anomaly detection, privacy protection, and text filtering
🤖 AI Annotation: LLM-powered automatic annotation and QA generation
⚡ High Performance: Efficient batch processing with caching and parallel execution
🎯 Developer Friendly: Modern SDK design with type hints, configuration management, and comprehensive error handling
☁️ Cloud Ready: Built-in support for OSS, MinIO, and other cloud storage providers

🚀 Quick Start

Install

pip install pydatamax

Examples

from datamax import DataMax

# prepare info
FILE_PATHS = ["/your/file/path/1.md", "/your/file/path/2.doc", "/your/file/path/3.xlsx"]
LABEL_LLM_API_KEY = "YOUR_API_KEY"
LABEL_LLM_BASE_URL = "YOUR_BASE_URL"
LABEL_LLM_MODEL_NAME = "YOUR_MODEL_NAME"
LLM_TRAIN_OUTPUT_FILE_NAME = "train"

# init client
client = DataMax(file_path=FILE_PATHS)

# get data
data = dm.get_data()

# get content
content = data.get("content")

# get pre label. return trainable qa list
qa = dm.get_pre_label(
    content=content,
    api_key=api_key,
    base_url=base_url,
    model_name=model,
    question_number=50,  # question_number_per_chunk
    max_qps=100.0,
    debug=False,
    structured_data=True,  # enable structured output
    auto_self_review_mode=True,  # auto review qa, pass with 4 and 5 score, drop with 1, 2 and 3 score.
    review_max_qps=100.0,
)


# save label data
client.save_label_data(qa_list, LLM_TRAIN_OUTPUT_FILE_NAME)

📚 Documentation

See docs: docs/index.md
Sections: Getting Started, Parsing, Cleaning, Labeling, Crawling, Evaluation, CLI, API, Extending, FAQ
For the complete text-modal QA generation pipeline, see examples/scripts/generate_qa.py

🤝 Contributing

Issues and Pull Requests are welcome!

📄 License

This project is licensed under the MIT License.

📞 Contact Us

📧 Email: cy.kron@foxmail.com, wang.xiangyuxy@outlook.com
🐛 Issues: GitHub Issues
📚 Documentation: Project Homepage
💬 Wechat Group:

⭐ If this project helps you, please give us a star!

Metadata

Release files for pydatamax 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pydatamax 0.2.1
File	Size	Uploaded
pydatamax-0.2.1.tar.gz	238.2 kB	Nov 26, 2025	Details

Built distribution (wheel)

Table of built distributions (wheels) for pydatamax 0.2.1
File	Interpreter	ABI	Platform	Reset
pydatamax-0.2.1-py3-none-any.whl 17.7 kB Nov 26, 2025	Python 3	none	any	Details

Total release size: 255.9 kB

Release files / pydatamax-0.2.1.tar.gz

Download URL	pydatamax-0.2.1.tar.gz
Size	238.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`9a12d130cb400a8d27abe042ddb5d6540fdb53d47bc50d5bf2929ef1c850caa1`
BLAKE2b-256 checksum How to use checksums	`eb140483d5fae59648fcb5610cb6045a0e9172216af9dfd9155df046a557fdb4`
Upload date	Nov 26, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.7`

Release files / pydatamax-0.2.1-py3-none-any.whl

Download URL	pydatamax-0.2.1-py3-none-any.whl
Size	17.7 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`a94ea8b2bc6cbbdf2d00b1345c8f5f851743f07e2ebb146645b6e0821b390636`
BLAKE2b-256 checksum How to use checksums	`9a706c44c02a2ff39b7754c49811c226de9c6886d37a3747218e7ee5684c2601`
Upload date	Nov 26, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.7`