spawn-parallel-instances 1.1.0

A package to run multiple instances of the same script.

Spawn Parallel Instances

Spawn Parallel Instances is a versatile tool designed to facilitate running multiple instances of various types of scripts with customizable environment variables and arguments. By leveraging its command-line options, you can efficiently manage and execute scripts in parallel, making it an essential tool for developers and system administrators.

Features

• Run multiple instances of a script concurrently.
• Load environment variables from a YAML file.
• Pass arguments to each process via a json file
• Specify the interpreter to use.
• Log output for each process with unique identifiers.

Installation

You can install the package using pip:

pip install spawn_parallel_instances

Usage

To use the concurrent executor, run the following command:

spawn_parallel_instances <script_name> --processes <num_processes> --env_file <env_file> --interpreter <interpreter> --script-type <script_type>

Arguments

Command-Line Arguments

The ScriptRunner accepts the following command-line arguments:

1. script (positional, required):

   • Description: The script to run.
   • Type: str
   • Example: my_script.py or my_script.js

2. --processes (optional):

   • Description: Number of processes to spawn.
   • Type: int
   • Default: 1
   • Example: --processes 4

3. --env-file (optional):

   • Description: YAML file containing environment variables.
   • Type: str
   • Default: env_variables.yml
   • Example: --env-file config.yml

4. --script-type (optional):

   • Description: Script type (either python, nodejs, go).
   • Type: str
   • Choices: ["python", "nodejs", "go"]
   • Default: python
   • Example: --script-type nodejs

5. --interpreter (optional):

   • Description: Interpreter to use (Python interpreter or node).
   • Type: str
   • Default: If not specified, defaults to python3.11 for Python scripts or node for Node.js scripts or go run for go.
   • Example: --interpreter python3.8 or --interpreter node

6. --script-args-file (optional):

   • Description: JSON file containing script arguments for each process.
   • Type: str
   • Default: ""
   • Example: --script-args-file args.json

7. --script-args (optional):

   • Description: Arguments to pass to each script instance, comma-separated for each instance.
   • Type: str
   • Default: ""
   • Example: --script-args "arg1,arg2,arg3"

Environment Variables

Environment variables should be defined in a YAML file. For example:

FOO: bar
BAZ: qux

The above YAML file can be used with the --env_file argument to set environment variables for the script.

Arguments

The JSON file specified with --script-args-file should contain a list of argument sets, one for each process instance. The number of argument sets in the JSON file must match the number of processes specified.

Json file format

[
    ["arg1", "arg2", "arg3"],
    ["arg4", "arg5", "arg6"],
    ["arg7", "arg8", "arg9"]
]

Logging

Each process will log its output to a file with a unique identifier. The log files are named based on the script name, process number, and a unique UUID.

Example

Assuming you have a script called tests/sample/test_script.py and an environment file tests/sample/config.yml, you can run:

spawn_parallel_instances tests/sample/test_script.py --processes 3 --env_file tests/sample/config.yml --script-type python --interpreter python3.9

Use Cases

Running a Python Script with Default Settings:

spawn_parallel_instances my_script.py

Running a Node.js Script with 4 Processes:

spawn_parallel_instances my_script.js --script-type nodejs --processes 4

Running a Go Script with Environment Variables:

spawn_parallel_instances my_script.go --script-type go --env-file my_env.yml

Running a Python Script with Specific Interpreter and Arguments:

spawn_parallel_instances my_script.py --interpreter python3.9 --script-args-file args.json

Running a Python Script with Direct Arguments:

spawn_parallel_instances my_script.py --script-args "arg1,arg2,arg3,arg4"

Troubleshooting

If you encounter issues while running ScriptRunner, consider the following tips:

Interpreter Not Found:

Ensure that the interpreter specified in --interpreter or derived from --script-type is correctly installed and available in your PATH.

Incorrect JSON Format:

Verify that the JSON file used for --script-args-file is correctly formatted as a list of lists. Each inner list should correspond to the arguments for one process instance.

Environment Variables Not Loaded:

Check if the YAML file specified with --env-file exists and is correctly formatted. Ensure it contains valid key-value pairs.

Script Execution Errors:

Review the log files generated for each process to identify any specific errors related to script execution.

Contributing

Contributions are welcome! Please follow these steps:

• Fork the repository.
• Create a new branch for your feature or bug fix.
• Make your changes.
• Write tests for your changes.
• Ensure all tests pass.
• Submit a pull request.

Development

To set up a development environment:

Clone the repository:

git clone https://github.com/stenzr/spawn_parallel_instances
cd spawn_parallel_instances

Create a virtual environment and activate it:

Copy code
python3.9 -m venv venv
source venv/bin/activate

Install the package in editable mode with development dependencies:

Copy code
pip install -e .[test]

License

This project is licensed under the MIT License - see the LICENSE file for details.