Skip to main content

Python ProcessRunner

Pypi Version Documentation Status

Designed to make reading from external processes easier. While targeted for use cases like processing log output, it also allows multiple writers to send text to the stdin of external processes.

Output can be sent to multiple locations. E.g. the stdout and stderr of an external process can be written to one or multiple files, AND also to the pipes of the local process.

Several convenience functions simplify common use cases. All the classes and functions take the command to execute in the subprocess.Popen/.call format, a list of strings starting with the command name, followed by any arguments for that command.

See the full documentation for complete details.

Chain commands together

Connect the stdout of one command to the stdin of another. Just use the “or” function, similar to how this is done in a shell.

This is not all purpose. Commands often do not end when you expect them to, and require use of watchers to stop them independently. (collectLines, for instance, will hang.)

See the tests directory “processrunner_chaining_test.py” for examples.

Provided classes

ProcessRunner

The ProcessRunner class uses subprocess.Popen. It does not use the shell=True flag. All instances of the class are saved in PROCESSRUNNER_PROCESSES. A list of currently active processes started by the class can be retrieved by calling getActiveProcesses(), which IS NOT a class member.

Properties
  • Non-blocking: Returns immediately; the external process is managed in a new thread

  • Output can be read by iterating over ProcessRunner.[stdout|stderr|output]

  • Supports with context manager syntax

Convenience functions

A number of convenience functions are available to quickly run a command with common options, run a command via SSH, and other activities. These are referenced on the Convenience Functions documentation page: Convenience Functions.

Examples

A larger set of examples is available on the Examples page in the documentation.

Simple

Use SCP to copy a local file to a remote host, using SSH key-based authentication.

# Run a command, wait for it to complete, and gather its return code
command = ["scp", "-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=no", "/path/to/local/file", clientAddress+":/tmp/"]
result = ProcessRunner(command).wait().poll()

Complex

While an external command runs, write the external process’ stdout and stderr to the corresponding local pipes, as well as corresponding files. Further, prefix the local pipe output with dedicated notes, and prefix the file output with timestamps.

# Imports
import os
import sys
from datetime import datetime
from processrunner import ProcessRunner, WriteOut

if __name__ == "__main__":
    # Logging files
    working_dir = os.path.dirname(os.path.realpath(__file__))
    stdoutFile = open(working_dir+'/stdout.txt', 'a')
    stderrFile = open(working_dir+'/stderr.txt', 'a')

    # Date/time notation for output lines in files
    class DateNote:
        def init(self):
            pass
        def __repr__(self):
            return datetime.now().isoformat() + " "

    # Prep the process
    # Script available in the ProcessRunner source:
    # https://github.com/arobb/python-processrunner/blob/main/tests/test-output-script.py
    command = ["tests/test-output-script.py",
               "--lines", "5",
               "--out-pipe", "both"]
    proc = ProcessRunner(command, autostart=False)

    # Attach output mechanisms to the process's output pipes. These are
    # handled asynchronously, so you can see the output while it is happening
    # Write to the console's stdout and stderr, with custom prefixes for each
    proc.mapLines(WriteOut(pipe=sys.stdout,
                           outputPrefix="validation-stdout> "),
                  procPipeName="stdout")
    proc.mapLines(WriteOut(pipe=sys.stderr,
                           outputPrefix="validation-stderr> "),
                  procPipeName="stderr")

    # Write to the log files, prepending each line with a date/time stamp
    proc.mapLines(WriteOut(pipe=stdoutFile, outputPrefix=DateNote()),
                  procPipeName="stdout")
    proc.mapLines(WriteOut(pipe=stderrFile, outputPrefix=DateNote()),
                  procPipeName="stderr")

    # Start the process, then block regular execution until the
    # process finishes
    proc.start().wait()

    stdoutFile.close()
    stderrFile.close()

Metadata

Release files for processrunner 2.6.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for processrunner 2.6.0
File Size Uploaded Provenance Actions
processrunner-2.6.0.tar.gz 50.3 kB Without provenance Details for processrunner-2.6.0.tar.gz

Release files / processrunner-2.6.0.tar.gz

Download URL processrunner-2.6.0.tar.gz
Size 50.3 kB
Tags Source
SHA-256 checksum
How to use checksums
4d598012ccd9c60e0e652b12de56f43154b838f01a97d4c81233fe6c06ba86f8
BLAKE2b-256 checksum
How to use checksums
105bbef4b0fd200161038c9b1ec395db2444ee05a0f7b2c34d225053c866c0dc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/50.3.1 requests-toolbelt/0.9.1 tqdm/4.55.0 CPython/3.7.9

Security for processrunner 2.6.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page