Comp Manager

Comp Manager is a Flask-based REST API framework for managing and tracking long-running mathematical computations with MongoDB persistence. It provides caching, computation lifecycle management, and comprehensive JSON serialization support for complex mathematical objects including SageMath types.

Features

Core Functionality

• Computation Management: Track computation lifecycle (started, paused, finished, failed)
• MongoDB Caching: Persistent caching for expensive computations
• RESTful API: OpenAPI/Swagger-compatible REST API via Connexion
• Flexible Serialization: JSON serialization for complex Python and SageMath objects

Mathematical Computing

• SageMath Integration: Support for Sage mathematical objects, including:
  • Integer, Real, Complex, Rational, and Number Fields
  • Matrices and Vectors
• Serialization: JSON encoding/decoding for mathematical structures
• Type Safety: TypedDict-based type system for mathematical objects

Developer Experience

• Decorator-Based Caching: Simple @mongo_cache decorator for function results
• Computation Tracking: @register_computation decorator for lifecycle management
• Type Hints: Full type hint coverage with mypy support
• Comprehensive Tests: >90% test coverage with pytest

Table of Contents

• Installation
• Quick Start
• Architecture
• Usage
  • Basic API Usage
  • Caching Computations
  • Tracking Computations
  • SageMath Integration
• API Reference
• Configuration
• Development
• Testing
• Contributing
• License

Installation

Prerequisites

• Python 3.11 or higher
• MongoDB 4.0 or higher (running locally or accessible remotely)
• (Optional) SageMath 9.0+ for mathematical computing features

Standard Installation

pip install comp-manager

Development Installation

# Clone the repository
git clone https://github.com/fredstro/comp-manager.git
cd comp-manager

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install with development dependencies
pip install -e ".[dev]"

Optional Dependencies

# For Jupyter notebook support
pip install comp-manager[notebook]

# For SageMath integration
pip install comp-manager[sage]

# For all optional dependencies
pip install comp-manager[dev,notebook,sage]

Quick Start

1. Start MongoDB

# Using Docker
docker run -d -p 27017:27017 --name mongodb mongo:latest

# Or use your local MongoDB installation
mongod --dbpath /path/to/data

2. Configure Connection

Set the MongoDB connection URI via environment variable:

export MONGO_URI="mongodb://localhost:27017/comp_manager"

3. Run the Application

Option A: Using the CLI (Recommended)

The simplest way to run the application is using the built-in command-line interface:

# Run with default settings (localhost:5000)
comp-manager

# Run with custom host and port
comp-manager --host 0.0.0.0 --port 8080

# Run in debug mode
comp-manager --debug

# Run with custom OpenAPI spec
comp-manager --spec-path /path/to/api.yaml

# View all options
comp-manager --help

Alternatively, you can use Python's module syntax:

python -m comp_manager --debug --port 8080

Option B: Programmatically

Create and run the Flask app directly in Python:

from comp_manager import create_app

# Create Flask app with OpenAPI spec
app = create_app()

if __name__ == "__main__":
    app.run(debug=True, port=5000)

4. Test the API

# Check health
curl http://localhost:5000/api/v1/health

# Access Swagger UI
open http://localhost:5000/api/v1/ui

Architecture

Project Structure

comp-manager/
 src/comp_manager/
    __init__.py              # App factory and connection management
    config.py                # Configuration management
    extensions.py            # MongoEngine extensions

    api/                     # REST API resources
       __init__.py
       resource.py          # Base MongoDB resource classes

    common/                  # Shared models and utilities
       __init__.py
       models.py            # Hashable document base classes

    core/                    # Core computation management
       __init__.py
       caching.py           # Cache storage implementations
       decorators.py        # Caching and tracking decorators
       models.py            # Computation and cache models
       queryset.py          # Custom querysets
       storage.py           # Storage abstractions

    utils/                   # Utility modules
        __init__.py
        api.py               # API helper functions
        db_helpers.py        # Database utilities
        json_encoder.py      # Base JSON encoder/decoder
        json_encoder_sage.py # SageMath JSON support
        mongoengine.py       # MongoEngine extensions
        serialization.py     # Serialization utilities
        types_sage.py        # SageMath type definitions

 tests/                      # Test suite
 CODE_OF_CONDUCT.md          # Code of conduct
 CONTRIBUTING.md             # For contributors
 Dockerfile                  # Dockerfile
 LICENSE                     # Project license
 Makefile                    # Convenience commands
 MANIFEST.in                 # Specify files to include / exclude
 pyproject.toml              # Project metadata and dependencies
 pytest.ini                  # Pytest configuration
 README.md                   # This file
 tox.ini                     # Tox configuration

Key Components

1. API Layer (src/comp_manager/api/)

• RESTful resources with MongoDB backend
• OpenAPI/Swagger specification support
• Automatic pagination and filtering

2. Core Layer (src/comp_manager/core/)

• Computation lifecycle management
• Caching infrastructure
• Decorator-based interfaces

3. Utilities (src/comp_manager/utils/)

• JSON serialization for complex types
• Database helpers
• SageMath type support

4. Data Models

• BaseDocument (common/models.py): Base class with timestamps and serialization
• HashableDocument (common/models.py): Adds content-based hashing
• Computation (core/models.py): Tracks computation lifecycle and status
• MongoCacheDB (core/models.py): Persistent cache storage

Usage

Basic API Usage

Defining a Resource

from comp_manager.api.resource import MongoDBResource
from comp_manager.common.models import BaseDocument
from comp_manager.extensions import me

class MyDocument(BaseDocument):
    """Custom document with name and value fields."""

    value = me.IntField()

    def to_dict(self):
        result = super().to_dict()
        result['value'] = self.value
        return result

# Create API resource
my_resource = MongoDBResource(MyDocument)

Note: BaseDocument provides name, created_at, updated_at fields and a base to_dict() method automatically.

Using the Resource

# GET /api/v1/mydocuments
response = my_resource.get()  # Returns paginated list

# GET /api/v1/mydocuments/{id}
response = my_resource.get_by_id(id='507f1f77bcf86cd799439011')

# POST /api/v1/mydocuments
response = my_resource.post(body={'name': 'example', 'value': 42})

# PUT /api/v1/mydocuments/{id}
response = my_resource.put(id='507f1f77bcf86cd799439011', body={'value': 100})

# DELETE /api/v1/mydocuments/{id}
response = my_resource.delete(id='507f1f77bcf86cd799439011')

Caching Computations

Use the @mongo_cache decorator to cache expensive function results:

from comp_manager.core.decorators import mongo_cache

@mongo_cache
def expensive_computation(n: int) -> int:
    """
    This function's results will be cached in MongoDB.
    """
    result = 0
    for i in range(n):
        result += i ** 2
    return result

# First call: computes and caches
result1 = expensive_computation(1000000)  # Takes time

# Second call: retrieved from cache
result2 = expensive_computation(1000000)  # Instant!

Cache Features:

• Automatic cache key generation from function arguments
• Support for both JSON-serializable and custom objects
• Configurable cache expiration (future feature)
• Cache statistics and monitoring

Tracking Computations

Track long-running computations with lifecycle management:

from comp_manager.core.decorators import register_computation

@register_computation
def long_computation(data: list[int]) -> dict:
    """
    This computation will be tracked in the database.
    Status updates: started -> finished/failed
    """
    # Perform computation
    result = {'sum': sum(data), 'count': len(data)}
    return result

# The computation status can be monitored via API
result = long_computation([1, 2, 3, 4, 5])

Computation States:

• started: Computation is running
• paused: Computation was paused (can be resumed)
• finished: Computation completed successfully
• failed: Computation encountered an error

Querying Computations:

from comp_manager.core.models import Computation

# Find all running computations
running = Computation.objects(status='started')

# Find computations by function name
my_comps = Computation.objects(function_name='long_computation')

# Get computation statistics
for comp in running:
    print(f"{comp.name}: {comp.running_time}")

SageMath Integration

Serializing Sage Objects

from sage.all import ZZ, QQ, matrix, NumberField, x
from comp_manager.utils.json_encoder_sage import SageJSONEncoder, SageJSONDecoder
import json

# Serialize a matrix
m = matrix(ZZ, [[1, 2], [3, 4]])
json_str = json.dumps(m, cls=SageJSONEncoder)

# Deserialize back to Sage object
m_restored = json.loads(json_str, cls=SageJSONDecoder)
assert m == m_restored

# Works with number fields too
F = NumberField(x^2 + 1, 'i')
i = F.gen()
json_str = json.dumps(i, cls=SageJSONEncoder)
i_restored = json.loads(json_str, cls=SageJSONDecoder)
assert i == i_restored

Supported Sage Types

• Rings: ZZ, QQ, RealField, ComplexField, NumberField
• Elements: Integers, Rationals, Reals, Complex numbers, Number field elements
• Structures: Matrices, Vectors
• Collections: Dictionaries with Sage keys, Tuples of Sage objects

Storing Sage Objects in MongoDB

from comp_manager.utils.mongoengine import JSONField
from comp_manager.extensions import me

class SageComputation(me.Document):
    name = me.StringField()
    matrix_data = JSONField()  # Can store Sage matrices
    result = JSONField()       # Can store any Sage object

    def to_dict(self):
        return {
            'name': self.name,
            'matrix_data': self.matrix_data,
            'result': self.result
        }

# Save Sage object
comp = SageComputation(
    name='my_computation',
    matrix_data=matrix(ZZ, [[1, 2], [3, 4]]),
    result={'eigenvalues': [1, 4]}
)
comp.save()

# Retrieve and use
comp_loaded = SageComputation.objects(name='my_computation').first()
m = comp_loaded.matrix_data  # Automatically deserialized as Sage matrix

API Reference

Endpoints

Health Check

GET /api/v1/health

Returns server health status.

Resource Endpoints

For each MongoDB resource, the following endpoints are available:

GET    /api/v1/{resource}           # List all (paginated)
GET    /api/v1/{resource}/{id}      # Get by ID
POST   /api/v1/{resource}           # Create new
PUT    /api/v1/{resource}/{id}      # Update existing
DELETE /api/v1/{resource}/{id}      # Delete

Query Parameters

Pagination:

• page: Page number (default: 1)
• per_page: Items per page (default: 20, max: 100)

Filtering:

• Field-based filters: ?name=value
• Date filters: ?created_at__gte=2024-01-01
• Array filters: ?tags__in=tag1,tag2

Example:

GET /api/v1/computations?status=finished&per_page=50&page=2

Configuration

Environment Variables

MongoDB Configuration

from comp_manager.config import Config

config = Config()
# Access MongoDB settings
settings = config.MONGODB_SETTINGS

Custom Configuration

from comp_manager import create_app

# Create app with custom config
app = create_app(spec_path='/path/to/openapi.yaml')

# Or override config
app.config['CUSTOM_SETTING'] = 'value'

Development

Setting Up Development Environment

# Clone and enter directory
git clone https://github.com/fredstro/comp-manager.git
cd comp-manager

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install with dev dependencies
pip install -e ".[dev]"

# Install pre-commit hooks (optional)
pip install pre-commit
pre-commit install

Code Quality Tools

The project uses Ruff for linting and formatting:

# Format code
ruff format .

# Lint code
ruff check .

# Fix linting issues
ruff check --fix .

Project Settings

• Line length: 100 characters
• Target Python: 3.12
• Complexity limit: 11
• Type checking: mypy (recommended)

Testing

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=comp_manager --cov-report=html

# Run specific test file
pytest tests/test_models.py

# Run with specific marker
pytest -m "not slow"

Test Structure

tests/
   fixtures/                   # Test fixtures and models
   test_api.py                 # API endpoint tests
   test_cache.py               # Caching tests
   test_computation.py         # Computation tracking tests
   test_fields.py              # Custom field tests
   test_json_encoder.py        # Base JSON encoder tests
   test_json_encoder_sage.py   # Sage serialization tests
   test_main.py                # CLI entry point tests
   test_models.py              # Database model tests
   test_serialize.py           # Serialization tests
   test_version.py             # Version tests

Writing Tests

import pytest
from comp_manager import create_app

@pytest.fixture
def app():
    """Create application for testing."""
    app = create_app()
    app.config['TESTING'] = True
    return app

@pytest.fixture
def client(app):
    """Create test client."""
    return app.test_client()

def test_api_endpoint(client):
    """Test API endpoint."""
    response = client.get('/api/v1/health')
    assert response.status_code == 200

Test Coverage Goals

• Unit tests: >90% coverage
• Integration tests: Critical user flows
• Sage tests: All serialization round-trips

Contributing

We welcome contributions! This project follows the Git Flow branching model. Please see CONTRIBUTING.md for complete guidelines.

Quick Contribution Guide

1. Fork the repository
2. Create a feature branch from develop: git checkout -b feature/my-feature develop
3. Make your changes with tests and documentation
4. Run tests: pytest
5. Format code: ruff format .
6. Commit: git commit -m "feat: add my feature"
7. Push: git push origin feature/my-feature
8. Open a Pull Request targeting the develop branch

Git Flow Branches

• main - Production-ready releases only
• develop - Integration branch (create PRs here!)
• feature/* - New features (your work here)
• release/* - Release preparation (maintainers)
• hotfix/* - Urgent fixes (maintainers)

Commit Message Convention

We follow Conventional Commits. Commit messages are validated by a pre-commit hook.

Format: <type>(<scope>): <subject>

Valid types:

Examples:

feat(api): add user authentication endpoint
fix(cache): resolve key collision for similar arguments
docs(readme): update installation instructions
refactor(models): extract validation logic to separate module

Roadmap

See IMPROVEMENT_PLAN.md for detailed future improvements including:

• Enhanced API documentation
• Performance optimizations
• Horizontal scaling support
• � CLI tool for running the application
• Python client library for remote management
• Advanced monitoring and observability

FAQ

Q: How do I run the Flask application?

A: Use the built-in CLI command comp-manager after installation:

comp-manager --debug --port 8080

All Flask configuration options are available as command-line arguments. See comp-manager --help for details.

Q: Can I use this without SageMath?

A: Yes! SageMath support is optional. The core computation management and caching features work without Sage.

Q: What's the maximum cache size?

A: Individual cached objects must be <16MB (MongoDB document limit). Total cache size can be configured in MongoCacheDB.meta.

Q: How do I clear the cache?

A: Use MongoDB directly or create an API endpoint:

from comp_manager.core.models import MongoCacheDB
MongoCacheDB.objects.delete()  # Clear all cache

Q: Can I use PostgreSQL instead of MongoDB?

A: Not currently. The project is built specifically for MongoDB's document model. PostgreSQL support would require significant additional work.

Q: Is this production-ready?

A: The project is in beta (Development Status 4). It's suitable for research and internal use. Production deployment should include additional security hardening and monitoring.

Q: Is this used somewhere?

A: The comp-manager project is currently used in two private projects related to research in number theory and in particular to computation of Maass wave forms in different settings: https://github.com/fredstro/hilbert-maass and https://github.com/fredstro/knot-maass (Both of these projects will go public soon.)

License

This project is licensed under the GNU General Public License v3.0 or later (GPLv3+). See LICENSE for details.

Credits

Author: Fredrik Stromberg (fredrik314@gmail.com)

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: Wiki

Acknowledgments

Built with:

• Flask - Web framework
• MongoEngine - MongoDB ODM
• Connexion - OpenAPI framework
• SageMath - Mathematical computing system
• pytest - Testing framework

AI tools:

• Claude Code (Claude Sonnet 4.5 / Opus 4.5): Generating commit messages, documentation and tests.