corebrum 0.1.1

Execute Python code transparently on Corebrum distributed compute infrastructure

Corebrum Python Library

Execute Python code transparently on Corebrum's distributed compute infrastructure with minimal code changes.

Installation

Install Corebrum using pip:

pip install corebrum

Quick Start

Using the Decorator Pattern

Decorate your functions to execute them on Corebrum:

import corebrum

# Configure Corebrum connection (optional, defaults to http://localhost:6502)
corebrum.configure(
    base_url="http://localhost:6502",
    identity_id="your-identity-id"  # Optional
)

# Decorate function to run on Corebrum
@corebrum.run()
def process_data(data):
    import pandas as pd
    import numpy as np
    
    df = pd.DataFrame(data)
    result = df.describe().to_dict()
    return result

# Call normally - executes on Corebrum
result = process_data([
    {"x": 1, "y": 2},
    {"x": 3, "y": 4},
    {"x": 5, "y": 6}
])

print(result)

Using the Execute Method

Execute raw Python code directly:

import corebrum

# Execute code with inputs
result = corebrum.execute("""
import math

def calculate():
    return math.sqrt(144)

# Must assign to a variable for execute() to capture it
result = calculate()
""", input_data={}, name="calculate_task")

print(result)  # 12.0

With Input Data

@corebrum.run()
def train_model(dataset_url, epochs=10):
    import torch
    # Your training code here
    return {"accuracy": 0.95, "loss": 0.05}

result = train_model("https://example.com/data.csv", epochs=20)

Features

• Transparent Execution: Code runs as if it were local, but executes on Corebrum's distributed infrastructure
• Automatic Dependency Detection: Automatically detects and includes Python package dependencies
• Input/Output Serialization: Handles JSON-serializable inputs and outputs automatically
• Error Handling: Corebrum errors surface naturally as Python exceptions
• Identity Support: Works with Corebrum's identity and memory system
• Timeout Control: Configurable task timeouts
• Progress Tracking: Real-time status updates via Server-Sent Events (SSE)

API Reference

Corebrum Class

Main client class for interacting with Corebrum.

client = Corebrum(
    base_url="http://localhost:6502",  # Corebrum web server URL
    identity_id=None,                   # Optional identity ID
    timeout=300,                        # Task timeout in seconds
    poll_interval=2.0,                  # Polling interval for status checks
    max_poll_attempts=300,              # Maximum polling attempts
)

@run() Decorator

Execute a function on Corebrum.

@client.run()
def my_function(x, y):
    return x + y

result = my_function(1, 2)  # Executes on Corebrum

Options:

• input: Additional input data dictionary
• identity_id: Override identity ID for this task
• timeout: Override timeout for this task

execute() Method

Execute raw Python code on Corebrum.

result = client.execute(
    code="""
def add(x, y):
    return x + y

# Must assign result to a variable for execute() to capture it
result = add(x, y)
""",
    input_data={"x": 1, "y": 2},  # Required: explicit input data
    name="my_task",                # Optional task name
    dependencies=["pandas"],        # Optional dependencies
)

run() vs execute(): When to Use Which?

Both run() and execute() submit tasks to Corebrum and wait for results, but they differ in how they handle code and inputs:

corebrum.run() - Decorator for Functions

Purpose: Decorator to run an existing function on Corebrum

Best for:

• Existing functions you want to run remotely
• When you want automatic argument extraction
• Minimal code changes (just add a decorator)

How it works:

• Extracts the function's source code using inspect.getsource()
• Automatically extracts function arguments from the function call
• Wraps the function definition and calls it with the extracted inputs
• Uses the function's return value as the result

Example:

@corebrum.run()
def factorial(number):
    if number < 0:
        raise ValueError("Factorial is not defined for negative numbers")
    if number == 0 or number == 1:
        return 1
    result = 1
    for i in range(2, number + 1):
        result *= i
    return result

# Call it normally - arguments are automatically extracted
result = factorial(8)  # Executes on Corebrum

corebrum.execute() - Execute Raw Code

Purpose: Execute raw Python code strings on Corebrum

Best for:

• Raw code strings (not in a function)
• Dynamic code generation
• When you need more control over input/output structure
• Working with code that isn't in a function

How it works:

• Executes the code directly (no function extraction needed)
• Requires explicit input_data dictionary
• Captures results by looking for common variable names (result, output, data, value, answer, res)
• Executes code in module namespace

Example:

code = """
def factorial(number):
    if number < 0:
        raise ValueError("Factorial is not defined for negative numbers")
    if number == 0 or number == 1:
        return 1
    result = 1
    for i in range(2, number + 1):
        result *= i
    return result

# Must assign to a variable for execute() to capture it
result = factorial(number)
"""

result = corebrum.execute(
    code,
    input_data={"number": 8},
    name="factorial_task"
)

Comparison Table

Feature	run()	execute()
Input	Function object	Code string
Arguments	Auto-extracted from function call	Explicit input_data dict
Code extraction	Uses inspect.getsource()	Uses provided string
Result capture	Function return value	Looks for variables (result, output, etc.)
Use case	Existing functions you want to run remotely	Ad-hoc code, dynamic code generation
Convenience	Higher (just add decorator)	Lower (must structure code manually)
Flexibility	Lower (must be a function)	Higher (any code structure)

Quick Decision Guide

Use run() when:

• ✅ You have an existing function
• ✅ You want automatic argument extraction
• ✅ You want minimal code changes
• ✅ The code is already in a function

Use execute() when:

• ✅ You have raw code strings
• ✅ You're generating code dynamically
• ✅ You need more control over input/output structure
• ✅ You're working with code that isn't in a function
• ✅ You want to execute scripts or multi-statement code blocks

Global Functions

For convenience, you can use global functions:

import corebrum

# Configure global instance
corebrum.configure(base_url="http://localhost:6502")

# Use global decorator
@corebrum.run()
def my_function():
    return "Hello"

# Use global execute
result = corebrum.execute("print(42)")

Examples

Data Processing

@corebrum.run()
def analyze_dataset(url):
    import pandas as pd
    import numpy as np
    
    df = pd.read_csv(url)
    return {
        "mean": df.mean().to_dict(),
        "std": df.std().to_dict(),
        "count": len(df)
    }

result = analyze_dataset("https://example.com/data.csv")

Machine Learning

@corebrum.run()
def train_classifier(features, labels):
    from sklearn.ensemble import RandomForestClassifier
    from sklearn.model_selection import cross_val_score
    
    clf = RandomForestClassifier()
    scores = cross_val_score(clf, features, labels, cv=5)
    
    return {
        "mean_accuracy": scores.mean(),
        "std_accuracy": scores.std()
    }

result = train_classifier(X_train, y_train)

Parallel Processing

import corebrum
from concurrent.futures import ThreadPoolExecutor

@corebrum.run()
def process_chunk(chunk):
    # Process data chunk
    return sum(chunk)

chunks = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

# Submit multiple tasks in parallel
with ThreadPoolExecutor(max_workers=3) as executor:
    futures = [executor.submit(process_chunk, chunk) for chunk in chunks]
    results = [f.result() for f in futures]

print(results)  # [6, 15, 24]

Example Scripts

The repository includes several example scripts in the examples/ directory that demonstrate different use cases:

basic_usage.py

Basic examples covering fundamental Corebrum usage:

• Example 1: Simple function execution with @run() decorator
• Example 2: Data processing with pandas (demonstrates automatic dependency installation)
• Example 3: Mathematical computations using standard library
• Example 4: Using execute() method for raw code execution

Run it:

python examples/basic_usage.py

advanced_usage.py

Advanced features and patterns:

• Example 1: Functions with default arguments
• Example 2: Error handling and exception catching
• Example 3: Custom timeout configuration
• Example 4: Using identity context for memory access
• Example 5: execute() with input data
• Example 6: Comprehensive error handling patterns

Run it:

python examples/advanced_usage.py

factorial_demo.py

Comprehensive demonstration comparing run() vs execute():

• Method 1: Using @run() decorator - best for existing functions
• Method 2: Using execute() method - best for raw code strings
• Method 3: Parallel execution of multiple factorial calculations
• Includes detailed comments explaining when to use each approach

Run it:

python examples/factorial_demo.py

Error Handling

Corebrum provides specific exception types:

from corebrum.exceptions import (
    CorebrumError,
    TaskSubmissionError,
    TaskExecutionError,
    TaskTimeoutError,
)

try:
    result = my_function()
except TaskSubmissionError as e:
    print(f"Failed to submit task: {e}")
except TaskExecutionError as e:
    print(f"Task execution failed: {e}")
except TaskTimeoutError as e:
    print(f"Task timed out: {e}")

Limitations

1. Serialization: Only JSON-serializable inputs and outputs are supported
2. Dependencies: Python packages must be available on Corebrum workers
3. File I/O: Local file access won't work (use URLs or Corebrum storage)
4. Interactive Code: Functions must be defined in files, not interactively
5. State: Functions should be stateless (no global state persistence)

Development

Setup

# Clone repository
git clone https://github.com/Corebrum/corebrum-pip.git
cd corebrum-pip

# Install in development mode
pip install -e .

# Install development dependencies
pip install -e ".[dev]"

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=corebrum --cov-report=html

# Run specific test file
pytest tests/test_corebrum.py

Code Formatting

# Format code
black corebrum tests

# Check linting
flake8 corebrum tests

# Type checking
mypy corebrum

Requirements

• Python 3.7+
• Corebrum server running and accessible
• Network access to Corebrum web server

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Support

• Documentation: GitHub README
• Issues: GitHub Issues
• Email: [e-mail]

Changelog

0.1.0 (2025-01-02)

• Initial release
• Decorator pattern support (@run())
• execute() method for raw code execution
• Automatic dependency detection
• Input/output serialization
• Error handling and exceptions
• Identity support
• Timeout configuration