watiba 0.2.13

Python syntactical sugar for embedded shell commands

Watiba

Watiba, pronounced wah-TEE-bah, is a lightweight Python pre-compiler for embedding Linux shell commands within Python applications. It is similar to other languages' syntactical enhancements where XML or HTML is integrated into a language such as JavaScript. That is the concept applied here but integrating BASH shell commands with Python.

As you browse this document, you'll Watiba is rich with features for shell command integration with Python.

Features:

• Shell command integration with Python code
• Current directory context maintained across commands throughout your Python code
• Async/promise support for integrated shell commands
• Remote shell command execution
• Remote shell command chaining and piping

Usage

Watiba files, suffixed with ".wt", are Python programs containing embedded shell commands. Shell commands are expressed within backtick characters emulating BASH's original capture syntax. They can be placed in any Python statement or expression. Watiba keeps track of the current working directory after the execution of any shell command so that all subsequent shell commands keep context. For example:

#!/usr/bin/python3

if __name__ == "__main__":
    `cd /tmp`
    for l in `ls -lrt`.stdout:
        print(l)

This loop will display the file list from /tmp. The ls -lrt is run in the context of previous cd /tmp.

Commands Expressed as Variables

Commands within backticks can be a variable, but cannot contain snippets of Python code or Python variables. The statement within the backticks must be either a pure shell command or a Python variable containing a pure shell command. To execute commands in a Python variable, prefix the variable name between backticks with a dollar sign.

A command variable is denoted by prepending a dollar sign on the variable name within backticks:

# Set the Python variable to the command
touch_cmd = "touch /tmp/blah.txt"

# Execute it
`$touch_cmd`  # Execute the command within Python variable touch_cmd

This example demonstrates keeping dir context and executing a command by variable:

#!/usr/bin/python3

if __name__ == "__main__":
    # Change CWD to /tmp
    `cd /tmp`
    
    # Set a command string
    my_cmd = "tar -zxvf blah.tar.gz"
    
    # Execute that command and save the result object in variable "w"
    w = `$my_cmd`
    if w.exit_code == 0:
        for l in w.stderr:
            print(l)

An example of building a command from other variables and then executing it within a print() statement:

in_file = "some_file.txt"
my_cmd = "cat {}".format(in_file)
print(`$my_cmd`.stdout)

These constructs are not supported:

file_name = "blah.txt"

# Python variable within backticks
`touch file_name`  # NOT SUPPORTED!

# Attempting to access Python variable with dollar sign
`touch $file_name` # NOT SUPPORTED!

# Python within backticks is NOT SUPPORTED!
`if x not in l: ls -lrt x`

Directory Context

An important Watiba usage point is directory context is kept for dispersed shell commands. Any command that changes the shell's CWD is discovered and kept by Watiba. Watiba achieves this by tagging a && echo pwd to the user's command, locating the result in the command's STDOUT, and finally setting the Python environment to that CWD with os.chdir(dir). This is automatic and opaque to the user. The user will not see the results of the generated suffix. If the echo suffix presents a problem for the user, it can be eliminated by prefixing the leading backtick with a dash. The dash turns off the context track, by not suffixing the command, and so causes Watiba to lose its context. However, the context is maintained within the set of commands in the backticks just not when it returns. For example, out = -`cd /tmp && ls -lrt` honors the cd within the scope of that execution line, but not for any backticked commands that follow later in your code.

Warning! The dash will cause Watiba to lose its directory context should the command cause a CWD change either explicitly or implicitly.

Example:

`cd /tmp`  # Context will be kept

# This will print from /home/user, but context is NOT kept  
for line in -`cd /home/user && ls -lrt`.stdout:
    print(line) 

# This will print from /tmp, not /home/user
for line in `ls -lrt`.stdout:
    print(line)

Command Results

The results of the command issued in backticks are available in the properties of the object returned by Watiba. Following are those properties:

• stdout - array of output lines from the command normalized for display
• stderr - array of standard error output lines from the command normalized for display
• exit_code - integer exit code value from command
• cwd - string current working directory after command was executed

Asynchronous Spawning and Promises

Shell commands can be executed asynchronously with a defined resolver callback block. Each spawn expression creates and runs a new OS thread. The resolver is a callback block that follows the Watiba spawn expression. The spawn feature is executed when a spawn `cmd` args: resolver block code block is encountered. The resolver is passed the results in the promise object. (The promise structure contains the properties defined in "Results from Spawned Command" of this README.) The spawn expression also returns a promise object to the caller of spawn. The promise object is passed to the resolver block in argument promise. The outer code can check its state with a call to resolved() on the returned promise object. Output from the command is found in promise.output. The examples throughout this README and in the examples.wt file make this clear.

Spawn Controller

All spawned threads are managed by Watiba's Spawn Controller. The controller watches for too many threads and incrementally slows down each thread start when that threshold is exceeded until either all the promises in the tree resolve, or an expiration count is reached, at which time an exception is thrown on the last spawned command.
This exception is raised by the default error method. This method as well as other spawn controlling parameters can be overridden. The controller's purpose is to not allow run away threads and provide signaling of possible hung threads.

Example of file that overrides spawn controller parameters:

#!/usr/bin/python3.8
def spawn_expired(promise, count):
    print("I do nothing just to demonstrate the error callback.")
    print("This command failed {} at this threshold {}".format(promise.command, count)
    
    raise Exception("Too many threads.")
    
if __name__ == "__main__":
    # Example showing default values
    parms = {"max": 10, # Max number of threads allowed before slowdown mode
         "sleep-floor": .125,  # Starting sleep value
         "sleep-ceiling": 3,  # Maximum sleep value
         "sleep-increment": .125,  # Incremental sleep value
         "expire": -1,  # Default: no expiration
         "error": spawn_expired  # Method called upon slowdown expiration
    }
     
    # Set spawn controller parameter values
    spawn-ctl parms

Spawn control parameters:

• max - Integer The maximum number of spawned commands allowed before the controller enters slowdown mode
• sleep-floor - Seconds The starting sleep value when the controller enters slowdown mode
• sleep-increment - Seconds The amount of seconds sleep will increase every third cycle when in slowdown mode
• sleep-ceiling - Seconds The highest length sleep value allowed when in slowdown mode. (As slow as it will get.)
• expire - Integer Total number of slowdown cycles allowed before the error method is called
• error - Method Callback method invoked when slowdown mode expires. By default, this will throw an exception. This method is passed 2 arguments:
  • promise - The promise attempting execution at the time of expiration
  • count - The thread count (unresolved promises) at the time of expiration

spawn-ctl only overrides the values it sets and does not affect values not specified. spawn-ctl statements can set whichever values it wants, can be dispersed throughout your code (i.e. multiple spawn-ctl statements) and only affects spawns subsequent to its setting at execution time.

Notes:

1. Arguments can be passed to the resolver by specifying a trailing variable after the command. If the arguments variable is omitted, an empty dictionary, i.e. {}, is passed to the resolver in args. Warning! Python threading does not deep copy objects passed as arguments to threads. What you place in args of the spawn expression will only be shallow copied so if there are references to other objects, it's not likely to survive the copy.
2. The resolver must return True to set the promise to resolved, or False to leave it unresolved.
3. A resolver can also set the promise to resolved by calling promise.set_resolved(). This is handy in cases where a resolver has spawned another command and doesn't want the outer promise resolved until the inner resolvers are done. To resolve an outer, i.e. parent, resolver issue promise.resolve_parent(). Then the parent resolver can return False at the end of its block so it leaves the resolved determination to the inner resolver block.
4. Each promise object holds its OS thread object in property thread and its thread id in property thread_id. This can be useful for controlling the thread directly. For example, to signal a kill.
5. spawn-ctl has no affect on join, wait, or watch. This is because spawn-ctl establishes an upper end throttle on the overall spawning process. When the number of spawns hits the max value, throttling (i.e. slowdown mode) takes affect and will expire if none of the promises resolve. Conversely, the arguments used by join, wait and watch control the sleep cycle and expiration of just those calls, not the spawned threads as a whole. When an expiration is set for, say, join, then that join will expire at that time. When an expiration is set in spawn-ctl, then if all the spawned threads as a whole don't resolve in time then an expiration function is called.

This is demonstrated in the examples. Spawn Syntax:

my_promise = spawn `cmd` [args]:
    resolver block (promise, args)
    args passed in args
    return resolved or unresolved (True or False)

For spawns within class definitions:

my_promise = self.spawn `cmd` [args]:
    resolver block (promise, args)
    args passed in args
    return resolved or unresolved (True or False)

Spawn with resolver arguments omitted:

my_promise = spawn `cmd`:
    resolver block (promise, args)
    return resolved or unresolved (True or False)

For spawns within class definitions resolver arguments omitted:

my_promise = self.spawn `cmd`:
    resolver block (promise, args)
    return resolved or unresolved (True or False)

Join, Wait, or Watch

Once commands are spawned, the caller can wait for all promises, including inner or child promises, to complete, or the caller can wait for just a specific promise to complete. To wait for all child promises including the promise on which you're calling this method, call join(). It will wait for that promise and all its children. To wait for just one specific promise, call wait() on the promise of interest. To wait for all promises in the promise tree, call join() on the root promise.

join and wait can be controlled through parameters. Each are iterators paused with a sleep method and will throw an expiration exception should you set a limit for iterations. If an expiration value is not set, no exception will be thrown and the cycle will run only until the promise(s) are resolved. join and wait are not affected by spawn-ctl.

watch is called to establish a separate asynchronous thread that will call back a function of your choosing should the command the promise is attached to times out. This is different than join and wait in that watch is not synchronous and does not pause. This is used to keep an eye on a spawned command and take action should it hang. Your watcher function is passed the promise on which the watcher was attached, and the arguments, if any, from the spawn expression. If your command does not time out (i.e. hangs and expires), the watcher thread will quietly go away when the promise is resolved. watch expiration is expressed in seconds, unlike join and wait which are expressed as total iterations paused at the sleep value. watch's polling cycle pause is .250 seconds, so the expiration value is multiplied by 4. The default expiration is 15 seconds.

Examples with controlling parameters:

# Spawn a thread running this command
p = spawn `ls -lrt`:
    resolver block
    return True
    
# Wait for promises, pause for 1/4 second each iteration, and throw an exception after 4 iterations (1 second)
try:
    p.join({"sleep": .250, "expire": 4})
except Exception as ex:
    print(ex.args)

# Wait for this promise, pause for 1 second each iteration, and throw an exception after 5 iterations (5 seconds)
try:
    p.wait({"sleep": 1, "expire": 5})
except Exception as ex:
    print(ex.args)
 
# My watcher function (called if spawned command never resolves by its experation period)
def watcher(promise, args):
    print("This promise is likely hung: {}".format(promise.command))
    print("and I still have the spawn expression's args: {}".format(args))

p = spawn `echo "hello" && sleep 5` args:
    print("Args passed to me: {}".format(args))
    return True

# Attach a watcher to this thread.  It will be called upon experation.
p.watch(watcher)
print("watch() does not pause like join or wait")

# Attach a watcher that will expire in 5 seconds
p.watch(watcher, {"expire": 5})

Promise Tree

Each spawn issued inserts its promise object into the promise tree. The outermost spawn will generate the root promise and each inner spawn will be among its children. There's no limit to how far it can nest. wait only applies to the promise on which it is called and is how it is different than join. wait does not consider any other promise state but the one it's called for, whereas join considers the one it's called for and anything below it in the tree.

The promise tree can be printed:

p = spawn `date`:
    return True
    
p.tree_dump()  # Dump tree from root
# or
p.tree_dump(subtree_node)  # Dump tree from node in argument

Parent and child joins shown in these two examples:

root_promise = spawn `ls -lr`:
    for file in promise.stdout:
        t = "touch {}".format(file)
        spawn `$t` {"file" file}:  # This promise is a child of root
            print("{} updated".format(promise.args["file"]))
            spawn `echo "done" > /tmp/done"`:  # Another child promise (root's grandchild)
                print("Complete")
                promise.resolve_parent()
                return True
            promise.resolve_parent()
            return False
    return False

root_promise.join()  # Wait on the root promise and all its children.  Thus, waiting for everything.

root_promise = spawn `ls -lr`:
    for file in promise.stdout:
        t = "touch {}".format(file)
        spawn `$t` {"file" file}:  # This promise is a child of root
            print("{} updated".format(promise.args["file"]))
            promise.join() # Wait for this promise and its children but not its parent (root)
            spawn `echo "done" > /tmp/done"`:
                print("Complete")

join syntax:

promise.join(optional args)
Where args is a Python dictionary with the following options:
    "sleep" - seconds of sleep for each iteration (fractions such as .5 are honored)
        default: .5 seconds
    "expire" - number of sleep iterations until an excpetions is raised
        default: no expiration
Note: "args" is optional and can be omitted

Example of joining parent and children promises:

p = spawn `ls *.txt`:
    for f in promise.output.stdout:
        cmd = "tar -zcvf {}.tar.gz {}".format(f)
        spawn `$cmd` {"file":f}:
            print("{} completed".format(f)
            promise.resolve_parent()
            return True
    return False

# Wait for all commands to complete
try:
    p.join({"sleep":1, "expire":20})
except Exception as ex:
    print(ex.args)

wait syntax

promise.wait(optional args)
Where args is a Python dictionary with the following options:
    "sleep" - seconds of sleep for each iteration (fractions such as .5 are honored)
        default: .5 seconds
    "expire" - number of sleep iterations until an excpetions is raised
        default: no expiration
Note: "args" is optional and can be omitted

Example of waiting on just the parent promise:

p = spawn `ls *.txt`:
    for f in promise.output.stdout:
        cmd = "tar -zcvf {}.tar.gz {}".format(f)
        spawn `$cmd` {"file":f}:
            print("{} completed".format(f)
            promise.resolve_parent() # Wait completes here
            return True
    return False

# Wait for just the parent promise to complete
try:
    p.wait({"sleep":1, "expire":20})
except Exception as ex:
    print(ex.args)

Resolving a parent promise:

p = spawn `ls -lrt`:
    for f in promise.output.stdout:
        cmd = "touch {}".format(f)
        # Spawn command from this resolver and pass our promise
        spawn `$cmd`:
            print("Resolving all promises")
            promise.resolve_parent() # Resolve parent promise here
            return True # Resolve child promise
        return False # Do NOT resolve parent promise here
p.join()  # Wait for ALL promises to be resolved

Results from Spawned Commands

Spawned commands return their results in the promise.output property of the promise object passed to the resolver block, and in the spawn expression if there is an assignment in that spawn expression.
The result properties can then be accessed as followed:

• promise.output.stdout - array of output lines from the command normalized for display
• promise.output.stderr - array of standard error output lines from the command normalized for display
• promise.output.exit_code - integer exit code value from command
• promise.output.cwd - current working directory after command was executed

Notes:

1. Watiba backticked commands can exist within the resolver
2. Other spawn blocks can be embedded within a resolver (recursion allowed)
3. The command within the spawn definition can be a variable (The same rules apply as for all backticked shell commands. This means the variable must contain pure shell commands.)
4. The leading dash to ignore CWD cannot be used in the spawn expression
5. The promise.output object is not available until promise.resolved() returns True

Simple example with the shell command as a Python variable:

#!/usr/bin/python3

# run "date" command asynchronously 
d = 'date "+%Y/%m/%d"'
spawn `$d`:
    print(promise.output.stdout[0])
    return True

Example with shell commands executed within resolver block:

#!/usr/bin/python3

print("Running Watiba spawn with wait")
`rm /tmp/done`

# run "ls -lrt" command asynchronously 
p = spawn `ls -lrt`:
    print("Exit code: {}".format(promise.output.exit_code))
    print("CWD: {}".format(promise.output.cwd))
    print("STDERR: {}".format(promise.output.stderr))

    # Loop through STDOUT from command
    for l in promise.output.stdout:
        print(l)
    `echo "Done" > /tmp/done`

    # Resolve promise
    return True

# Pause until spawn command is complete
p.wait()
print("complete")

Thread Counting

To access to the number of threads your code has spawned.

num_of_spawns = promise.spawn_count()  # Returns number of nodes in the promise tree
num_of_resolved_promises = promise.resolved_count() # Returns the number of promises resolved in tree

Remote Execution

Shell commands can be executed remotely. This is achieved though the SSH command, issued by Watiba, and has the following requirements:

• OpenSSH is installed on the local and remote hosts
• The local SSH key is in the remote's authorized_keys file. The details of this process is beyond the scope of this README. For those instructions, consult www.ssh.com

To execute a command remotely, an @host parameter is suffixed to the backticked command. The host name can be a literal or a variable. To employ a variable, prepend a $ to the name following @.
Examples:

p = spawn `ls -lrt`@remoteserver {parms}

remotename = "serverB"
p = spawn `ls -lrt`@$remotename {parms}

out = `ls -lrt`@remoteserver

remotename = "serverB"
out = `ls -lrt`@$remotename

Command Chaining

Watiba extends its remote command execution to chaining commands across multiple remote hosts. This is achieved by the chain expression. This expression will execute the backticked command across a list of hosts, passed by the user, sequentially, synchronously until the hosts list is exhausted, or the command fails. chain returns a Python dictionary where the keys are the host names and the values the WTOutput from the command run on that host.

Examples:

out = chain `tar -zcvf backup/file.tar.gz dir/*` {"hosts", ["serverA", "serverB"]}
for host,output in out.items():
    print(f'{host} exit code: {output.exit_code}')
    for line in output.stderr:
        print(line)

Piping Output with Chain (Experimental)

The chain expression supports piping STDOUT and/or STDERR to other commands executed on remote servers. Complex arrangements can be constructed through the Python dictionary passed to the chain expression. The dictionary contents function as follows:

• "hosts": [server, server, ...] This entry instructions chain on which hosts the backticked command will run. This is a required entry.

• "stdout": {server:command, server:command, ...} This is an optional entry.

• "stderr": {server:command, server:command, ...} This is an optional entry.

Just like a chain expression that does not pipe output, the return object is a dictionary of WTOutput object keyed by the host name from the hosts list and not from the commands recieving the piped output.

If any command fails, a WTChainException is raised.

Note: The piping feature is experimental as of this release, and a better design will eventually supercede it.

Examples:

# This is a simple chain with no piping
try:
    args = {"hosts": ["serverA", "serverB", "serverC"]}
    out = chain `ls -lrt dir/` args
    for host, output in out.items():
        print(f'{host} exit code: {output.exit_code}')
except WTChainException as ex:
    print(f'ERROR: {ex.msg}, {ex.host}, {ex.command}, {ex.output.stderr})

# This is a more complex chain that runs the "ls -lrt" command on each server listed in "hosts"
# and pipes the STDOUT output from serverC to serverV and serverD, to those commands, and serverB's STDERR
# to serverX and its command
try:
    args = {"hosts": ["serverA", "serverB", "serverC"],
                "stdout": {"serverC":{"serverV": "grep something", "serverD":"grep somethingelse"}},
                "stderr": {"serverB":{"serverX": "cat >> /tmp/serverC.err"}}
           }
    out = chain `ls -lrt dir/` args
    for host, output in out.items():
        print(f'{host} exit code: {output.exit_code}')
except WTChainException as ex:
    print(f'ERROR: {ex.msg}, {ex.host}, {ex.command}, {ex.output.stderr})

####How does this work? Watiba will run the backticked command in the expression on each host listed in hosts, in sequence and synchronously. If there is a "stdout" found in the arguments, then it will name the source host as the key, i.e. the host from which STDOUT will be read, and fed to each host and command listed under that host. This is true for STDERR as well.

The method in which Watiba feeds the piped output is through a an echo command shell piped to the command to be run on that host. So, "stdout": {"serverC":{"serverV": "grep something"}} causes Watiba to read each line of STDOUT from serverC and issue echo "$line" | grep something on serverV. It is piping from serverC to serverV.

Installation

PIP

If you installed this as a Python package, e.g. pip, then the pre-compiler can be found where the package was installed. For example:

/home/{user}/.local/lib/python3.8/site-packages/watiba/watiba-c.py

This file can be copied to any directory in your PATH. It is stand-alone and can be copied anywhere you need.

GITHUB

If you cloned this from github, you'll still need to install the package with pip, first, for the watbia module. Follow these steps to install Watiba locally.

# Must install Watiba package
pip3 install watiba

# Now edit makefile and run make
cd {to where you git cloned watiba}/watiba

# 1. Edit makefile
# 2. Change the top two variables to your target destinations
# 3. venv = /home/rwalk/Projects/python3/venv/lib64/python3.8/site-packages/
#    bin = /home/rwalk/bin/watiba-c
#        -- CHANGE TO --
#    venv = {your Python venv environment}
#    bin = {your bin directory and file name}

# Execute command
make

Pre-compiling

Once you've installed watiba-c.py into your path, you can execute it to pre-compile your .wt (Watiba) code. Output will be written to STDOUT, so you'll need to redirect it to your final Python file. Example follows:

watiba-c.py my_file.wt > my_file.py
chmod +x my_file.py
./my_file.py

To show the Watiba version of your pre-compiler, enter:

watiba-c.py version

Examples

my_file.wt

#!/usr/bin/python3

# Stand alone commands.  One with directory context, one without

# This CWD will be active until a subsequent command changes it
`cd /tmp`

# Simple statement utilizing command and results in one statement
print(`cd /tmp`.cwd)

# This will not change the Watiba CWD context, because of the dash prefix, but within 
# the command itself the cd is honored.  file.txt is created in /home/user/blah but
# this does not impact the CWD of any subsequent commands.  They
# are still operating from the previous cd command to /tmp
-`cd /home/user/blah && touch file.txt`

# This will print "/tmp" _not_ /home because of the leading dash on the command
print("CWD is not /home: {}".format(-`cd /home`.cwd))

# This will find text files in /tmp/, not /home/user/blah  (CWD context!)
w=`find . -name '*.txt'`
for l in w.stdout:
    print("File: {}".format(l))


# Embedding commands in print expressions that will print the stderr output, which tar writes to
print(`echo "Some textual comment" > /tmp/blah.txt && tar -zcvf /tmp/blah.tar.gz /tmp`).stdout)

# This will print the first line of stdout from the echo
print(`echo "hello!"`.stdout[0])

# Example of more than one command in a statement line
if len(`ls -lrt`.stdout) > 0 or len(-`cd /tmp`.stdout) > 0:
    print("You have stdout or stderr messages")


# Example of a command as a Python varible and
#  receiving a Watiba object
cmd = "tar -zcvf /tmp/watiba_test.tar.gz /mnt/data/git/watiba/src"
cmd_results = `$cmd`
if cmd_results.exit_code == 0:
    for l in cmd_results.stderr:
        print(l)

# Simple reading of command output
#  Iterate on the stdout property
for l in `cat blah.txt`.stdout:
    print(l)

# Example of a failed command to see its exit code
xc = `lsvv -lrt`.exit_code
print("Return code: {}".format(xc))

# Example of running a command asynchronously and using the resolver callback code block
spawn `cd /tmp && tar -zxvf tarball.tar.gz`:
    for l in promise.output.stderr:
        print(l)
print("This prints before the tar output.")
`sleep 5`  # Pause for 5 seconds so spawn can complete

# List dirs from CWD, iterate through them, spawn a tar command
# then within the resolver, spawn a move command
# Demonstrates spawns within resolvers
for dir in `ls -d *`.stdout:
    tar = "tar -zcvf {}.tar.gz {}"
    prom = spawn `$tar` {"dir": dir}:
        print("{} tar complete".format(args["dir"]))
        mv = "mv -r {}/* /tmp/.".format(args["dir"])
        spawn `$mv`:
            print("Move done")
            # Resolve outer promise
            promise.resolve_parent()
            return True
        # Do not resolve this promise yet.  Let the inner resolver do it
        return False
    prom.join()