multi-env-employer 1.1.0

Execute Python code from different virtual environments as regular functions. Supports multiple Python versions, async/await, generators, and more.

MultiEnvEmployer

Execute Python code from different virtual environments as regular functions.

Run functions from isolated environments with different Python versions and conflicting dependencies without import conflicts or version issues.

---

Features

• ✅ Isolated Execution - Run code in separate virtual environments
• ✅ Transparent API - Call remote functions as if they were local
• ✅ Multiple Python Versions - Support for Python 3.5+ modules
• ✅ Stateful & Stateless - Choose between persistent or one-shot execution
• ✅ Print Interception - Capture stdout from remote processes
• ✅ Generators - Full support for generator functions
• ✅ Async Functions - Automatic handling of async/await
• ✅ Result Caching - Optional caching for expensive operations
• ✅ Large Data Streaming - Automatic chunking for big results
• ✅ Timeout Control - Flexible timeout modes (none, absolute, progress)
• ✅ Error Handling - Remote exceptions as local exceptions
• ✅ Process Management - Fine-grained control over worker processes

---

Installation

pip install multi-env-employer

---

Quick Start

from pathlib import Path
from MultiEnvEmployer import Employer, RemoteModule

# Initialize employer with target environment
emp = Employer(
    project_dir=Path("path/to/modules"),
    venv_path=Path("path/to/venv")
)

# Connect to remote module
module = RemoteModule(emp, "my_module")

# Call functions as if they were local
result = module.add(2, 3)
print(result)  # 5

# Use context manager for automatic cleanup
with Employer("path/to/modules", "path/to/venv") as emp:
    module = RemoteModule(emp, "my_module")
    result = module.process_data([1, 2, 3])

---

Basic Usage

Stateless Execution (default)

New process per function call:

module = RemoteModule(emp, "my_module", stateful=False)
module.func1()  # Process A
module.func2()  # Process B

Stateful Execution

Single process for all calls:

module = RemoteModule(emp, "my_module", stateful=True)
module.set_value(10)  # Process A
module.get_value()    # Process A (same process)

Print Interception

module = RemoteModule(emp, "my_module", print_output="terminal")
module.my_function()  # Prints are captured and displayed

Generators

for value in module.stream_numbers(5):
    print(value)  # 0, 1, 2, 3, 4

Async Functions

result = module.async_operation(10)  # Called synchronously

Result Caching

module = RemoteModule(emp, "my_module", caching=True)
result1 = module.expensive_function(x=10)  # Executes
result2 = module.expensive_function(x=10)  # From cache

Timeout Control

from MultiEnvEmployer import TimeoutPolicy

# No timeout
module = RemoteModule(emp, "my_module", 
    timeout=TimeoutPolicy(seconds=60, mode="none"))

# Absolute timeout (hard limit)
module = RemoteModule(emp, "my_module",
    timeout=TimeoutPolicy(seconds=30, mode="absolute"))

# Progress timeout (resets on activity)
module = RemoteModule(emp, "my_module",
    timeout=TimeoutPolicy(seconds=10, mode="progress"))

Error Handling

from MultiEnvEmployer import errors

try:
    result = module.failing_function()
except errors.RemoteExecutionError as e:
    print(f"Remote error: {e.error_type}")
    print(f"Message: {e.error_message}")
    print(f"Traceback:\n{e.remote_traceback}")
except errors.RemoteTimeoutError as e:
    print(f"Timeout after {e.timeout_seconds}s")
except errors.WrongArgumentsError as e:
    print(f"Invalid arguments: {e.details}")

---

Advanced Features

Large Data Streaming

Large return values are automatically streamed in chunks:

result = module.get_large_list()  # Automatically chunked if > 1 MB

Process Management

# Close specific module
emp.close(module)

# Close specific function (stateless)
emp.close("module_name.function_name")

# Close all processes
emp.close()

Custom Configuration

emp = Employer(
    project_dir=Path("modules"),
    venv_path=Path("venv"),
    cache_path=Path("cache"),
    pickle_protocol=4,
    stream_threshold=1024 * 1024,  # 1 MB
    chunk_size=512 * 1024          # 512 KB
)

---

Requirements

• Python 3.8+ (for the library itself)
• Python 3.5+ (for remote modules)
• Virtual environment with target Python version

---

Security Warning

⚠️ CRITICAL: This library uses pickle for inter-process communication. Never use with untrusted data sources.

• Pickle can execute arbitrary code during deserialization
• Only use MultiEnvEmployer with code and data you control
• Not suitable for processing user-supplied data or external inputs

---

Use Cases

• Legacy Code Integration - Run old Python 2.7 code from Python 3.x
• Dependency Isolation - Use conflicting library versions in one project
• Version Testing - Test code across multiple Python versions
• Resource Isolation - Isolate memory-intensive operations
• Sandboxing - Run untrusted code in separate processes

---

Documentation

Full documentation: GitHub Repository

---

License

MIT License - see LICENSE

---

Contact

• GitHub Issues: https://github.com/REYIL/MultiEnvEmployer/issues
• Telegram: @REYIL_DEV