tursi 0.3.0a3

A simple framework to deploy AI models locally with one command, no containers needed

tursi-ai

Deploy AI models with unmatched simplicity - zero container overhead. Features efficient model quantization for reduced memory usage and faster inference, perfect for resource-constrained environments like IoT and Edge devices.

Table of Contents

• Features
• Installation
• Quick Start
• Common Use Cases
• Command Reference
• API Reference
• Configuration
• Best Practices
• Development
• Troubleshooting

Features

• Intuitive command-line interface for model deployment
• Zero container overhead - lightweight and efficient
• Optimized for resource-constrained environments
• Simple API interface
• Built-in security features
• Edge-efficient model quantization
  • Dynamic and static quantization support
  • 4-bit and 8-bit quantization options
  • Optimized for CPU inference
• Rate limiting and monitoring
• Automated versioning and releases

Installation

pip install tursi

Quick Start

# Start a model server
tursi up distilbert-base-uncased

# Start with custom settings
tursi up distilbert-base-uncased --port 8000 --quantization dynamic --bits 4

# List running models
tursi ps

# View server logs
tursi logs

# Check resource usage
tursi stats

# Stop the server
tursi down

Making Predictions

# Test with positive sentiment
curl -X POST -H "Content-Type: application/json" \
     -d '{"text": "This is a great product! I love it!"}' \
     http://localhost:5000/predict

# Example response:
# {"label": "POSITIVE", "score": 0.9998828172683716}

# Test with negative sentiment
curl -X POST -H "Content-Type: application/json" \
     -d '{"text": "This product is terrible, I regret buying it."}' \
     http://localhost:5000/predict

# Example response:
# {"label": "NEGATIVE", "score": 0.9995611310005188}

Command Reference

tursi up

Start a model server.

tursi up [OPTIONS] MODEL_NAME

Options:
  --port, -p INTEGER          Port to run the API server on (default: 5000)
  --host TEXT                 Host to bind the API server to (default: 127.0.0.1)
  --quantization, -q TEXT     Quantization mode: 'dynamic' or 'static' (default: dynamic)
  --bits, -b INTEGER         Number of bits for quantization (4 or 8) (default: 8)
  --rate-limit, -r TEXT      API rate limit (default: "100/minute")
  --cache-dir, -c PATH       Directory to cache models (default: ~/.tursi/models)
  -h, --help                 Show this message and exit

tursi down

Stop a running model server.

tursi down

tursi ps

List running models.

tursi ps

tursi logs

View server logs.

tursi logs [OPTIONS]

Options:
  --follow, -f    Follow log output
  --tail INTEGER  Number of lines to show (default: all)

tursi stats

Show resource usage statistics.

tursi stats

API Reference

POST /predict

Endpoint for making predictions using the loaded model.

Request Body:

{
    "text": "Your text here"
}

Response:

{
    "label": "POSITIVE",
    "score": 0.9
}

Configuration

The following environment variables can be set:

• RATE_LIMIT: API rate limit (default: "100 per minute")
• RATE_LIMIT_STORAGE_URI: Storage backend for rate limiting (default: "memory://")
• QUANTIZATION_MODE: Quantization mode (default: "dynamic")
• QUANTIZATION_BITS: Number of bits for quantization (default: 8)

Quantization Options

• Mode:

  • dynamic: Quantization is performed at runtime (default)
    • Best for general use cases
    • Maintains good accuracy while reducing model size
  • static: Quantization is performed during model loading
    • Better performance for specific use cases
    • Requires calibration data

• Bits:

  • 8: 8-bit quantization (default)
    • Good balance between compression and accuracy
    • Recommended for most use cases
  • 4: 4-bit quantization
    • More aggressive compression
    • May impact accuracy
    • Best for resource-constrained environments

Development

Setting Up Development Environment

# Clone the repository
git clone https://github.com/BlueTursi/tursi-ai.git
cd tursi-ai

# Install Poetry (if not installed)
curl -sSL https://install.python-poetry.org | python3 -

# Install dependencies
poetry install

# Run tests with coverage
poetry run pytest tests/ -v --cov=tursi --cov-report=xml

# Build package
poetry build

Release Process

For detailed information about the release process, please refer to RELEASE.md.

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Run tests to ensure everything works
5. Commit your changes (git commit -m 'Add amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

License

MIT License

Acknowledgments

Built with 💙 using:

• Transformers
• Flask
• PyTorch
• Poetry
• ONNX Runtime
• Optimum

Built by BlueTursi.

Common Use Cases

1. Edge Device Deployment

# Deploy a lightweight model optimized for edge devices
tursi up microsoft/squeezebert-mnli --bits 4 --quantization dynamic \
  --rate-limit "50/minute" --port 8080

# Monitor resource usage to ensure it fits device constraints
tursi stats

2. Production API Server

# Deploy a robust model with appropriate rate limiting
tursi up bert-base-uncased \
  --host 0.0.0.0 \
  --port 443 \
  --rate-limit "1000/minute" \
  --cache-dir /opt/tursi/models

# Monitor the deployment
tursi logs --follow

3. Local Development

# Quick deployment for testing
tursi up distilbert-base-uncased

# In another terminal, test the API
curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"text": "Testing the model"}' \
  http://localhost:5000/predict

4. Multi-Model Setup

# Deploy sentiment analysis model
tursi up distilbert-base-uncased --port 5000

# Deploy named entity recognition model
tursi up dslim/bert-base-NER --port 5001

# List all running models
tursi ps

5. Resource-Constrained Environments

# Deploy with maximum compression
tursi up distilbert-base-uncased \
  --bits 4 \
  --quantization static \
  --rate-limit "20/minute"

# Monitor memory usage
tursi stats

Each use case demonstrates different aspects of Tursi's capabilities:

• Edge deployment with 4-bit quantization for minimal resource usage
• Production setup with appropriate host/port configuration
• Local development workflow
• Running multiple models simultaneously
• Maximum compression for resource-constrained environments

Best Practices

Resource Management

• Start with 8-bit quantization and monitor performance
• Switch to 4-bit if memory constraints are tight
• Use tursi stats to monitor resource usage
• Set appropriate rate limits based on hardware capacity

Production Deployment

• Always set explicit --host and --port values
• Configure appropriate rate limits
• Use a dedicated cache directory
• Monitor logs with tursi logs --follow
• Set up health monitoring using the /health endpoint

Development Workflow

• Use default settings for quick iterations
• Test different quantization options
• Monitor memory usage with tursi stats
• Use curl or Postman for API testing