blockpype 1.0.0a1

Breaks apart input stream into blocks and pipes each block into newly spawned processes

Blockpype

Breaks apart input stream into blocks and pipes each block into newly spawned processes

Installing

pip install git+https://github.com/kfzteile24/blockpype.git

Testing

Clone the repository and run visual tests that display each started PID and their output:

• test/by-byte.sh - should break apart in the middle of the 2-byte character
• test/by-char.sh - should break apart in the middle of some numbers
• test/by-line.sh - should break apart nicely line by line

Usage

cat content.txt | blockpype -l 10 other-program --arg-for-program --another-arg "argument value"

It will cut content.txt into blocks of 10 lines, and for each block, a process will be started using the command:

other-program --arg-for-program --another-arg "argument value"

and the contents of the block will be served to stdin

What's it for?

Whenever you want to break apart a large stream when the other-program consumer stores it into memory until EOF, and you don't have enough memory for it.

Similar to split --filter "other-program --arg-for-program --another-arg \"argument value\"" except less clunky, other-program gets launched from current pwd with the current user, and with the current environment variables.