MultiEnvEmployer

Execute Python code from different virtual environments as regular functions.

Run functions from isolated environments with different Python versions and conflicting dependencies without import conflicts or version issues.

---

Features

• ✅ Isolated Execution - Run code in separate virtual environments
• ✅ Transparent API - Call remote functions as if they were local
• ✅ Multiple Python Versions - Support for Python 3.5+ modules
• ✅ Stateful & Stateless - Choose between persistent or one-shot execution
• ✅ Print Interception - Capture stdout from remote processes
• ✅ Generators - Full support for generator functions
• ✅ Async Functions - Automatic handling of async/await
• ✅ Result Caching - Optional caching for expensive operations
• ✅ Large Data Streaming - Automatic chunking for big results
• ✅ Timeout Control - Flexible timeout modes (none, absolute, progress)
• ✅ Error Handling - Remote exceptions as local exceptions
• ✅ Process Management - Fine-grained control over worker processes

---

Installation

pip install multi-env-employer

---

Quick Start

from pathlib import Path
from MultiEnvEmployer import Employer, RemoteModule

# Initialize employer with target environment
emp = Employer(
    project_dir=Path("path/to/modules"),
    venv_path=Path("path/to/venv")
)

# Connect to remote module
module = RemoteModule(emp, "my_module")

# Call functions as if they were local
result = module.add(2, 3)
print(result)  # 5

# Use context manager for automatic cleanup
with Employer("path/to/modules", "path/to/venv") as emp:
    module = RemoteModule(emp, "my_module")
    result = module.process_data([1, 2, 3])

---

Basic Usage

Stateless Execution (default)

New process per function call:

module = RemoteModule(emp, "my_module", stateful=False)
module.func1()  # Process A
module.func2()  # Process B

Stateful Execution

Single process for all calls:

module = RemoteModule(emp, "my_module", stateful=True)
module.set_value(10)  # Process A
module.get_value()    # Process A (same process)

Print Interception

module = RemoteModule(emp, "my_module", print_output="terminal")
module.my_function()  # Prints are captured and displayed

Generators

for value in module.stream_numbers(5):
    print(value)  # 0, 1, 2, 3, 4

Async Functions

result = module.async_operation(10)  # Called synchronously

Result Caching

module = RemoteModule(emp, "my_module", caching=True)
result1 = module.expensive_function(x=10)  # Executes
result2 = module.expensive_function(x=10)  # From cache

Timeout Control

from MultiEnvEmployer import TimeoutPolicy

# No timeout
module = RemoteModule(emp, "my_module", 
    timeout=TimeoutPolicy(seconds=60, mode="none"))

# Absolute timeout (hard limit)
module = RemoteModule(emp, "my_module",
    timeout=TimeoutPolicy(seconds=30, mode="absolute"))

# Progress timeout (resets on activity)
module = RemoteModule(emp, "my_module",
    timeout=TimeoutPolicy(seconds=10, mode="progress"))

Error Handling

from MultiEnvEmployer import errors

try:
    result = module.failing_function()
except errors.RemoteExecutionError as e:
    print(f"Remote error: {e.error_type}")
    print(f"Message: {e.error_message}")
    print(f"Traceback:\n{e.remote_traceback}")
except errors.RemoteTimeoutError as e:
    print(f"Timeout after {e.timeout_seconds}s")
except errors.WrongArgumentsError as e:
    print(f"Invalid arguments: {e.details}")

---

Advanced Features

Large Data Streaming

Large return values are automatically streamed in chunks:

result = module.get_large_list()  # Automatically chunked if > 1 MB

Process Management

# Close specific module
emp.close(module)

# Close specific function (stateless)
emp.close("module_name.function_name")

# Close all processes
emp.close()

Custom Configuration

emp = Employer(
    project_dir=Path("modules"),
    venv_path=Path("venv"),
    cache_path=Path("cache"),
    pickle_protocol=4,
    stream_threshold=1024 * 1024,  # 1 MB
    chunk_size=512 * 1024          # 512 KB
)

---

Requirements

• Python 3.8+ (for the library itself)
• Python 3.5+ (for remote modules)
• Virtual environment with target Python version

---

Security Warning

⚠️ CRITICAL: This library uses pickle for inter-process communication. Never use with untrusted data sources.

• Pickle can execute arbitrary code during deserialization
• Only use MultiEnvEmployer with code and data you control
• Not suitable for processing user-supplied data or external inputs

---

Use Cases

• Legacy Code Integration - Run old Python 2.7 code from Python 3.x
• Dependency Isolation - Use conflicting library versions in one project
• Version Testing - Test code across multiple Python versions
• Resource Isolation - Isolate memory-intensive operations
• Sandboxing - Run untrusted code in separate processes

---

Documentation

Full documentation: GitHub Repository

---

License

MIT License - see LICENSE

---

Contact

• GitHub Issues: https://github.com/REYIL/MultiEnvEmployer/issues
• Telegram: @REYIL_DEV