tdi-rust-python-tools 0.9.3

Rust-based Python tools for faster processing of data.

TDI Rust Python Tools

A collection of high-performance Rust-based tools compiled as a Python extension module to speed up common data processing tasks.

🚀 What is This?

This project uses Rust (a fast, compiled language) to implement performance-critical functions, then makes them available to Python through PyO3 and Maturin. Think of it as writing the slow parts of your code in a faster language, while keeping your familiar Python interface.

Key Features:

• Fast text processing utilities (regex-based transformations)
• CSV to Excel conversion (convert_to_xlsx)
• High-performance DictWriter class (faster drop-in replacement for Python's csv.DictWriter)
• Seamless Python integration with full type hints

📋 Prerequisites

For Development

You'll need to install:

1. Python 3.13+ - Download here
2. Rust - Install via rustup:

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   

3. uv - Modern Python package manager (faster than pip):

   curl -LsSf https://astral.sh/uv/install.sh | sh
   

For Users

Just Python 3.13+ is needed to install the pre-built package:

pip install tdi-rust-python-tools

🛠️ Development Setup

1. Clone and Setup Environment

# Clone the repository
git clone https://github.com/TDI-Data/tdi_rust_python_tools.git
cd tdi_rust_python_tools

# Create virtual environment and install dependencies
uv sync

This creates a .venv/ directory with all dependencies installed.

2. Build the Extension Module

# Build and install in development mode (fast, includes debug symbols)
uv run maturin develop

# OR build optimized version (slower to compile, but faster to run)
uv run maturin develop --release

Note: With uv, you don't need to activate the virtual environment! The uv run command automatically uses the project's .venv/ environment.

After running uv run maturin develop, the Rust code is compiled and installed into your Python environment. You can now import it:

import tdi_rust_python_tools

# Use the fast DictWriter
writer = tdi_rust_python_tools.DictWriter(file, fieldnames=['name', 'age'])

3. Test Your Changes

uv run pytest tests/

🔄 Updating Dependencies

Updating Rust Dependencies

Rust dependencies are defined in Cargo.toml. To update them:

# Update a specific dependency (e.g., regex)
cargo update -p regex

# Update all dependencies to their latest compatible versions
cargo update

# Update to potentially breaking versions (edit Cargo.toml first)
# For example, change: regex = "1.11.1"
# To:                  regex = "1.12.0"
# Then run:
cargo update

Common Rust dependencies in this project:

• pyo3 - Python bindings (⚠️ keep in sync with Maturin version)
• regex - Regular expressions
• csv - CSV reading/writing
• rust_xlsxwriter - Excel file generation
• lazy_static - Compile-time regex optimization

After updating, rebuild with uv run maturin develop to test your changes.

Updating Python Dependencies

Python dependencies (managed by uv) are in pyproject.toml:

# Add a new Python dependency
uv add pytest

# Add a development-only dependency
uv add --group dev ruff

# Update all Python dependencies
uv sync -U

# Update a specific package
uv add pytest@latest

📦 Publishing a New Version

Publishing to PyPI (Automated via GitHub Actions)

The easiest way to publish is through GitHub Actions:

1. Update the version number in both files:

   # Cargo.toml
   [package]
   version = "0.8.3"  # Change this
   
   # pyproject.toml uses dynamic versioning from Cargo.toml
   # No changes needed there
   

2. Commit your changes:

   git add Cargo.toml
   git commit -m "Bump version to 0.8.3"
   

3. Create and push a git tag:

   git tag v0.8.3
   git push origin main --tags
   

4. GitHub Actions takes over:

   • Automatically builds wheels for Linux, macOS, and Windows
   • Publishes to PyPI when the workflow completes
   • Check progress at: https://github.com/TDI-Data/tdi_rust_python_tools/actions

Manual Publishing (Advanced)

If you need to publish manually:

# Build release wheels for your current platform
maturin build --release

# Wheels are created in target/wheels/
# Upload to PyPI (requires API token)
maturin publish --username __token__ --password $PYPI_TOKEN

Note: Manual publishing only creates wheels for your current OS/architecture. The GitHub Actions workflow is recommended as it builds for all platforms.

📁 Project Structure

tdi_rust_python_tools/
├── src/
│   └── lib.rs                      # All Rust code (functions & classes)
├── shared/
│   └── shared.py                   # Pure Python utilities
├── tests/                          # Python tests (pytest)
├── tdi_rust_python_tools.pyi       # Type stubs for IDE support
├── Cargo.toml                      # Rust dependencies & metadata
├── pyproject.toml                  # Python packaging & dependencies
├── uv.lock                         # Locked dependency versions
└── .github/workflows/CI.yml        # Automated testing & publishing

🐛 Common Issues

"Failed to build extension module"

• Make sure Rust is installed: rustc --version
• Try a clean rebuild: cargo clean && uv run maturin develop

"Module not found" after building

• Ensure you ran uv run maturin develop (not just cargo build)
• Check the build completed successfully

Version mismatch errors

• Keep Cargo.toml version in sync with git tags
• Ensure PyO3 version matches Maturin requirements

📚 Learn More

• PyO3 Documentation: https://pyo3.rs/
• Maturin Guide: https://www.maturin.rs/
• Rust Book: https://doc.rust-lang.org/book/
• uv Documentation: https://docs.astral.sh/uv/

📄 License

[Add your license here]

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes and test (uv run maturin develop && uv run pytest)
4. Commit your changes (git commit -m 'Add amazing feature')
5. Push to the branch (git push origin feature/amazing-feature)
6. Open a Pull Request