comp-manager 0.0.3

Api for managing computations with a MongoDB backend

Comp Manager

Comp Manager is a Flask-based REST API framework for managing and tracking long-running mathematical computations with MongoDB persistence. It provides caching, computation lifecycle management, and comprehensive JSON serialization support for complex mathematical objects including SageMath types.

Features

Core Functionality

• Computation Management: Track computation lifecycle (started, paused, finished, failed)
• MongoDB Caching: Persistent caching for expensive computations
• RESTful API: OpenAPI/Swagger-compatible REST API via Connexion
• Flexible Serialization: JSON serialization for complex Python and SageMath objects

Mathematical Computing

• SageMath Integration: Support for Sage mathematical objects, including:
  • Integer, Real, Complex, Rational, and Number Fields
  • Matrices and Vectors
• Serialization: JSON encoding/decoding for mathematical structures
• Type Safety: TypedDict-based type system for mathematical objects

Developer Experience

• Decorator-Based Caching: Simple @mongo_cache decorator for function results
• Computation Tracking: @register_computation decorator for lifecycle management
• Type Hints: Full type hint coverage with mypy support
• Comprehensive Tests: >90% test coverage with pytest

---

Table of Contents

• Installation
• Quick Start
• Architecture
• Usage
  • Basic API Usage
  • Caching Computations
  • Tracking Computations
  • SageMath Integration
• API Reference
• Configuration
• Development
• Testing
• Contributing
• License

---

Installation

Prerequisites

• Python 3.11 or higher
• MongoDB 4.0 or higher (running locally or accessible remotely)
• (Optional) SageMath 9.0+ for mathematical computing features

Standard Installation

pip install comp-manager

Development Installation

# Clone the repository
git clone https://github.com/fredstro/comp-manager.git
cd comp-manager

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install with development dependencies
pip install -e ".[dev]"

Optional Dependencies

# For Jupyter notebook support
pip install comp-manager[notebook]

# For SageMath integration
pip install comp-manager[sage]

# For all optional dependencies
pip install comp-manager[dev,notebook,sage]

---

Quick Start

1. Start MongoDB

# Using Docker
docker run -d -p 27017:27017 --name mongodb mongo:latest

# Or use your local MongoDB installation
mongod --dbpath /path/to/data

2. Configure Connection

Set the MongoDB connection URI via environment variable:

export MONGO_URI="mongodb://localhost:27017/comp_manager"

3. Run the Application

Option A: Using the CLI (Recommended)

The simplest way to run the application is using the built-in command-line interface:

# Run with default settings (localhost:5000)
comp-manager

# Run with custom host and port
comp-manager --host 0.0.0.0 --port 8080

# Run in debug mode
comp-manager --debug

# Run with custom OpenAPI spec
comp-manager --spec-path /path/to/api.yaml

# View all options
comp-manager --help

Alternatively, you can use Python's module syntax:

python -m comp_manager --debug --port 8080

Option B: Programmatically

Create and run the Flask app directly in Python:

from comp_manager import create_app

# Create Flask app with OpenAPI spec
app = create_app()

if __name__ == "__main__":
    app.run(debug=True, port=5000)

4. Test the API

# Check health
curl http://localhost:5000/api/v1/health

# Access Swagger UI
open http://localhost:5000/api/v1/ui

---

Architecture

Project Structure

comp-manager/
 src/comp_manager/
    __init__.py              # App factory and connection management
    config.py                # Configuration management
    extensions.py            # MongoEngine extensions

    api/                     # REST API resources
       __init__.py
       resource.py          # Base MongoDB resource classes

    common/                  # Shared models and utilities
       __init__.py
       models.py            # Hashable document base classes

    core/                    # Core computation management
       __init__.py
       caching.py           # Cache storage implementations
       decorators.py        # Caching and tracking decorators
       models.py            # Computation and cache models
       queryset.py          # Custom querysets
       storage.py           # Storage abstractions

    utils/                   # Utility modules
        __init__.py
        api.py               # API helper functions
        db_helpers.py        # Database utilities
        json_encoder.py      # Base JSON encoder/decoder
        json_encoder_sage.py # SageMath JSON support
        mongoengine.py       # MongoEngine extensions
        serialization.py     # Serialization utilities
        types_sage.py        # SageMath type definitions

 tests/                      # Test suite
 CODE_OF_CONDUCT.md          # Code of conduct
 CONTRIBUTING.md             # For contributors
 Dockerfile                  # Dockerfile
 LICENSE                     # Project license
 Makefile                    # Convenience commands
 MANIFEST.in                 # Specify files to include / exclude
 pyproject.toml              # Project metadata and dependencies
 pytest.ini                  # Pytest configuration
 README.md                   # This file
 tox.ini                     # Tox configuration

Key Components

1. API Layer (src/comp_manager/api/)

• RESTful resources with MongoDB backend
• OpenAPI/Swagger specification support
• Automatic pagination and filtering

2. Core Layer (src/comp_manager/core/)

• Computation lifecycle management
• Caching infrastructure
• Decorator-based interfaces

3. Utilities (src/comp_manager/utils/)

• JSON serialization for complex types
• Database helpers
• SageMath type support

4. Data Models

• BaseDocument (common/models.py): Base class with timestamps and serialization
• HashableDocument (common/models.py): Adds content-based hashing
• Computation (core/models.py): Tracks computation lifecycle and status
• MongoCacheDB (core/models.py): Persistent cache storage

---

Usage

Basic API Usage

Defining a Resource

from comp_manager.api.resource import MongoDBResource
from comp_manager.common.models import BaseDocument
from comp_manager.extensions import me

class MyDocument(BaseDocument):
    """Custom document with name and value fields."""

    value = me.IntField()

    def to_dict(self):
        result = super().to_dict()
        result['value'] = self.value
        return result

# Create API resource
my_resource = MongoDBResource(MyDocument)

Note: BaseDocument provides name, created_at, updated_at fields and a base to_dict() method automatically.

Using the Resource

# GET /api/v1/mydocuments
response = my_resource.get()  # Returns paginated list

# GET /api/v1/mydocuments/{id}
response = my_resource.get_by_id(id='507f1f77bcf86cd799439011')

# POST /api/v1/mydocuments
response = my_resource.post(body={'name': 'example', 'value': 42})

# PUT /api/v1/mydocuments/{id}
response = my_resource.put(id='507f1f77bcf86cd799439011', body={'value': 100})

# DELETE /api/v1/mydocuments/{id}
response = my_resource.delete(id='507f1f77bcf86cd799439011')

Caching Computations

Use the @mongo_cache decorator to cache expensive function results:

from comp_manager.core.decorators import mongo_cache

@mongo_cache
def expensive_computation(n: int) -> int:
    """
    This function's results will be cached in MongoDB.
    """
    result = 0
    for i in range(n):
        result += i ** 2
    return result

# First call: computes and caches
result1 = expensive_computation(1000000)  # Takes time

# Second call: retrieved from cache
result2 = expensive_computation(1000000)  # Instant!

Cache Features:

• Automatic cache key generation from function arguments
• Support for both JSON-serializable and custom objects
• Configurable cache expiration (future feature)
• Cache statistics and monitoring

Tracking Computations

Track long-running computations with lifecycle management:

from comp_manager.core.decorators import register_computation

@register_computation
def long_computation(data: list[int]) -> dict:
    """
    This computation will be tracked in the database.
    Status updates: started -> finished/failed
    """
    # Perform computation
    result = {'sum': sum(data), 'count': len(data)}
    return result

# The computation status can be monitored via API
result = long_computation([1, 2, 3, 4, 5])

Computation States:

• started: Computation is running
• paused: Computation was paused (can be resumed)
• finished: Computation completed successfully
• failed: Computation encountered an error

Querying Computations:

from comp_manager.core.models import Computation

# Find all running computations
running = Computation.objects(status='started')

# Find computations by function name
my_comps = Computation.objects(function_name='long_computation')

# Get computation statistics
for comp in running:
    print(f"{comp.name}: {comp.running_time}")

SageMath Integration

Serializing Sage Objects

from sage.all import ZZ, QQ, matrix, NumberField, x
from comp_manager.utils.json_encoder_sage import SageJSONEncoder, SageJSONDecoder
import json

# Serialize a matrix
m = matrix(ZZ, [[1, 2], [3, 4]])
json_str = json.dumps(m, cls=SageJSONEncoder)

# Deserialize back to Sage object
m_restored = json.loads(json_str, cls=SageJSONDecoder)
assert m == m_restored

# Works with number fields too
F = NumberField(x^2 + 1, 'i')
i = F.gen()
json_str = json.dumps(i, cls=SageJSONEncoder)
i_restored = json.loads(json_str, cls=SageJSONDecoder)
assert i == i_restored

Supported Sage Types

• Rings: ZZ, QQ, RealField, ComplexField, NumberField
• Elements: Integers, Rationals, Reals, Complex numbers, Number field elements
• Structures: Matrices, Vectors
• Collections: Dictionaries with Sage keys, Tuples of Sage objects

Storing Sage Objects in MongoDB

from comp_manager.utils.mongoengine import JSONField
from comp_manager.extensions import me

class SageComputation(me.Document):
    name = me.StringField()
    matrix_data = JSONField()  # Can store Sage matrices
    result = JSONField()       # Can store any Sage object

    def to_dict(self):
        return {
            'name': self.name,
            'matrix_data': self.matrix_data,
            'result': self.result
        }

# Save Sage object
comp = SageComputation(
    name='my_computation',
    matrix_data=matrix(ZZ, [[1, 2], [3, 4]]),
    result={'eigenvalues': [1, 4]}
)
comp.save()

# Retrieve and use
comp_loaded = SageComputation.objects(name='my_computation').first()
m = comp_loaded.matrix_data  # Automatically deserialized as Sage matrix

---

API Reference

Endpoints

Health Check

GET /api/v1/health

Returns server health status.

Resource Endpoints

For each MongoDB resource, the following endpoints are available:

GET    /api/v1/{resource}           # List all (paginated)
GET    /api/v1/{resource}/{id}      # Get by ID
POST   /api/v1/{resource}           # Create new
PUT    /api/v1/{resource}/{id}      # Update existing
DELETE /api/v1/{resource}/{id}      # Delete

Query Parameters

Pagination:

• page: Page number (default: 1)
• per_page: Items per page (default: 20, max: 100)

Filtering:

• Field-based filters: ?name=value
• Date filters: ?created_at__gte=2024-01-01
• Array filters: ?tags__in=tag1,tag2

Example:

GET /api/v1/computations?status=finished&per_page=50&page=2

---

Configuration

Environment Variables

Variable	Description	Default
MONGO_URI	MongoDB connection URI	mongodb://localhost:27017/comp_manager
SECRET_KEY	Flask secret key (for sessions)	None (generated)
FLASK_ENV	Flask environment	production
LOG_LEVEL	Logging level	INFO

MongoDB Configuration

from comp_manager.config import Config

config = Config()
# Access MongoDB settings
settings = config.MONGODB_SETTINGS

Custom Configuration

from comp_manager import create_app

# Create app with custom config
app = create_app(spec_path='/path/to/openapi.yaml')

# Or override config
app.config['CUSTOM_SETTING'] = 'value'

---

Development

Setting Up Development Environment

# Clone and enter directory
git clone https://github.com/fredstro/comp-manager.git
cd comp-manager

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install with dev dependencies
pip install -e ".[dev]"

# Install pre-commit hooks (optional)
pip install pre-commit
pre-commit install

Code Quality Tools

The project uses Ruff for linting and formatting:

# Format code
ruff format .

# Lint code
ruff check .

# Fix linting issues
ruff check --fix .

Project Settings

• Line length: 100 characters
• Target Python: 3.12
• Complexity limit: 11
• Type checking: mypy (recommended)

---

Testing

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=comp_manager --cov-report=html

# Run specific test file
pytest tests/test_models.py

# Run with specific marker
pytest -m "not slow"

Test Structure

tests/
   fixtures/                   # Test fixtures and models
   test_api.py                 # API endpoint tests
   test_cache.py               # Caching tests
   test_computation.py         # Computation tracking tests
   test_fields.py              # Custom field tests
   test_json_encoder.py        # Base JSON encoder tests
   test_json_encoder_sage.py   # Sage serialization tests
   test_main.py                # CLI entry point tests
   test_models.py              # Database model tests
   test_serialize.py           # Serialization tests
   test_version.py             # Version tests

Writing Tests

import pytest
from comp_manager import create_app

@pytest.fixture
def app():
    """Create application for testing."""
    app = create_app()
    app.config['TESTING'] = True
    return app

@pytest.fixture
def client(app):
    """Create test client."""
    return app.test_client()

def test_api_endpoint(client):
    """Test API endpoint."""
    response = client.get('/api/v1/health')
    assert response.status_code == 200

Test Coverage Goals

• Unit tests: >90% coverage
• Integration tests: Critical user flows
• Sage tests: All serialization round-trips

---

Contributing

We welcome contributions! This project follows the Git Flow branching model. Please see CONTRIBUTING.md for complete guidelines.

Quick Contribution Guide

1. Fork the repository
2. Create a feature branch from develop: git checkout -b feature/my-feature develop
3. Make your changes with tests and documentation
4. Run tests: pytest
5. Format code: ruff format .
6. Commit: git commit -m "feat: add my feature"
7. Push: git push origin feature/my-feature
8. Open a Pull Request targeting the develop branch

Git Flow Branches

• main - Production-ready releases only
• develop - Integration branch (create PRs here!)
• feature/* - New features (your work here)
• release/* - Release preparation (maintainers)
• hotfix/* - Urgent fixes (maintainers)

Commit Message Convention

We follow Conventional Commits. Commit messages are validated by a pre-commit hook.

Format: <type>(<scope>): <subject>

Valid types:

Type	Description
feat	New feature
fix	Bug fix
docs	Documentation changes
style	Code style (formatting, missing semicolons, etc.)
refactor	Code refactoring (no functional change)
perf	Performance improvement
test	Test additions or changes
chore	Maintenance tasks
ci	CI/CD configuration changes
revert	Reverting a previous commit
build	Build system or dependencies

Examples:

feat(api): add user authentication endpoint
fix(cache): resolve key collision for similar arguments
docs(readme): update installation instructions
refactor(models): extract validation logic to separate module

---

Roadmap

See IMPROVEMENT_PLAN.md for detailed future improvements including:

• Enhanced API documentation
• Performance optimizations
• Horizontal scaling support
• � CLI tool for running the application
• Python client library for remote management
• Advanced monitoring and observability

---

FAQ

Q: How do I run the Flask application?

A: Use the built-in CLI command comp-manager after installation:

comp-manager --debug --port 8080

All Flask configuration options are available as command-line arguments. See comp-manager --help for details.

Q: Can I use this without SageMath?

A: Yes! SageMath support is optional. The core computation management and caching features work without Sage.

Q: What's the maximum cache size?

A: Individual cached objects must be <16MB (MongoDB document limit). Total cache size can be configured in MongoCacheDB.meta.

Q: How do I clear the cache?

A: Use MongoDB directly or create an API endpoint:

from comp_manager.core.models import MongoCacheDB
MongoCacheDB.objects.delete()  # Clear all cache

Q: Can I use PostgreSQL instead of MongoDB?

A: Not currently. The project is built specifically for MongoDB's document model. PostgreSQL support would require significant additional work.

Q: Is this production-ready?

A: The project is in beta (Development Status 4). It's suitable for research and internal use. Production deployment should include additional security hardening and monitoring.

Q: Is this used somewhere?

A: The comp-manager project is currently used in two private projects related to research in number theory and in particular to computation of Maass wave forms in different settings: https://github.com/fredstro/hilbert-maass and https://github.com/fredstro/knot-maass (Both of these projects will go public soon.)

License

This project is licensed under the GNU General Public License v3.0 or later (GPLv3+). See LICENSE for details.

---

Credits

Author: Fredrik Stromberg ([e-mail])

---

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: Wiki

---

Acknowledgments

Built with:

• Flask - Web framework
• MongoEngine - MongoDB ODM
• Connexion - OpenAPI framework
• SageMath - Mathematical computing system
• pytest - Testing framework

AI tools:

• Claude Code (Claude Sonnet 4.5 / Opus 4.5): Generating commit messages, documentation and tests.

---