A package to run multiple instance of the same script.
Project description
Spawn Parallel Instances
Spawn Parallel Instances is a Python package that allows you to run parrallel instances of the same script. It supports loading environment variables from a YAML file and allows specifying the Python interpreter to use.
Features
- Run multiple instances of a script concurrently.
- Load environment variables from a YAML file.
- Specify the Python interpreter to use.
- Log output for each process with unique identifiers.
Installation
You can install the package using pip:
pip install spawn_parallel_instances
Alternate Installation
You can install it from source
git clone https://github.com/stenzr/spawn_parallel_instances
cd spawn_parallel_instances
python -m pip install .
Usage
To use the concurrent executor, run the following command:
spawn_parallel_instances <script_name> --processes <num_processes> --env_file <env_file> --python_interpreter <interpreter>
Arguments
- <script_name>: The Python script to run (positional argument).
- --processes: Number of processes to spawn (default: 1).
- --env_file: YAML file containing environment variables (default: env_variables.yml).
- --python_interpreter: Python interpreter to use (default: python3.11).
Example
Assuming you have a script called tests/sample/test_script.py
and an environment file tests/sample/config.yml
, you can run:
spawn_parallel_instances tests/sample/test_script.py --processes 3 --env_file tests/sample/config.yml --python_interpreter python3.9
Environment Variables
Environment variables should be defined in a YAML file. For example:
FOO: bar
BAZ: qux
The above YAML file can be used with the --env_file argument to set environment variables for the script.
Logging
Each process will log its output to a file with a unique identifier. The log files are named based on the script name, process number, and a unique UUID.
Development
To set up a development environment:
Clone the repository:
git clone https://github.com/stenzr/spawn_parallel_instances
cd spawn_parallel_instances
Create a virtual environment and activate it:
Copy code
python3.9 -m venv venv
source venv/bin/activate
Install the package in editable mode with development dependencies:
Copy code
pip install -e .[test]
Contributing
Contributions are welcome! Please follow these steps:
- Fork the repository.
- Create a new branch for your feature or bug fix.
- Make your changes.
- Write tests for your changes.
- Ensure all tests pass.
- Submit a pull request.
License
This project is licensed under the MIT License - see the LICENSE file for details.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for spawn_parallel_instances-1.0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e57d1e310d4d45b547a2e4b45d7d3b99a494438d4c23604eb76d430a1967c09 |
|
MD5 | cd191b5eb32377aef7cbc9f6eb0d7e3c |
|
BLAKE2b-256 | 3ae769547b4bca1b86b6b1851acf2c098fc6069563157380f109082664ebdb09 |
Hashes for spawn_parallel_instances-1.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4050e2954bb1b4581ca4fa3f155a37e5cb99f880ca2aa81fec2af2fcde28d264 |
|
MD5 | da0db031b7cfa61238f3a2f65f75a958 |
|
BLAKE2b-256 | 1f4e653b16ae86a4469476410680e0610ba74630b6022c7c81ac8eed890ac999 |