xontrib-pipeliner 0.2.1

Easily process the lines using pipes in xonsh.

Easily process the lines using pipes in xonsh shell. Multicore processing supported.

If you like the idea of pipeliner click ⭐ on the repo and stay tuned.

Install

xpip install -U xontrib-pipeliner
echo 'xontrib load pipeliner' >> ~/.xonshrc
# Reload xonsh

Usage

Let your pipe lines flow thru the Python code:

<cmd> | ... | pl "<lambda expression>" | <cmd> | ...

There are two variables available in lambda expression:

• line from pipe.
• num of the line starts with 0.

To run expression in multi core mode use parallel pl - ppl. Also take a look at plx below to execute the commands on the data from pipe lines.

Examples

Python way to line modification

ls -1 / | pl "line + ' is here'" | head -n 3

bin is here
boot is here
dev is here

Line number

ls -1 / | head -n 4 | pl "f'{num} {line}'"

0 bin
1 boot
2 cdrom
3 dev

Ignore line

$ ls -1 / | head -n 4 | pl "f'{num} {line}' if num%2 == 0 else None"

0 bin
2 cdrom

Splitting

cat /etc/passwd | head -n 3 | pl "line.split(':')[6]"

/bin/bash
/usr/sbin/nologin
/usr/sbin/nologin

Imports

import re
cat /etc/passwd | head -n 3 | pl "re.sub('/bin/bash', '/usr/bin/xonsh', line)"

root:x:0:0:root:/root:/usr/bin/xonsh
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin

Arrays

cat /etc/passwd | head -n 3 | pl "line.split(':')" | grep nologin | pl "':'.join(eval(line)[::-1])"

/usr/sbin/nologin:/usr/sbin:daemon:1:1:x:daemon
/usr/sbin/nologin:/bin:bin:2:2:x:bin

Variables and operations chaining

Expression is a lambda function so using variables and operations chaining since Python 3.8+ are available by trick with the walrus operator and the list:

ls -1 / | head -n3 | pl "[s:='b', line.replace(s, s.upper()+')')][-1]"

B)in
B)oot
dev

Execute command with the line

ls / | head -n 3 | pl "execx('du -sh /'+line) or 'Done command with /'+line"

0       /bin
Done command with /bin
840M    /boot
Done command with /boot
4,0K    /cdrom
Done command with /cdrom

Multicore pipelining

By default pipeliner works using one CPU core. To use them all in parallel try ppl command:

head /etc/passwd | ppl "str(num) + ' ' + line.split(':')[0]"

1 daemon
0 root
2 bin
4 sync
5 games
8 mail
9 news
6 man
7 lp
3 sys

Note! The order of result lines is unpredictable because lines will be processed in parallel. The num variable contains the real line number.

Escape from the string

To avoid writing Python inside the string and get the syntax highlighting there is a tricky way with using xonsh macro:

def py(code):
    return code

echo 123 | pl @(py!(line + '2'))

Experimental

Pipeliner exec

There are plx and pplx commands to run execx(f"{plx_command}") most shorter way.

For example when you want to rename files you can do it Pythonic way:

$ mkdir -p /tmp/plx-test && cd /tmp/plx-test
$ touch 111 222 333 && ls
111 222 333

$ ls | plx "mv {line} prefix-{line}"
mv 111 prefix-111
mv 222 prefix-222
mv 333 prefix-333

$ ls
prefix-111 prefix-222 prefix-333

Echo example:

$ ls | plx 'echo {line} # {num}'
echo prefix-111 # 0
prefix-111
echo prefix-222 # 1
prefix-222
echo prefix-333 # 2
prefix-333