Skip to main content

Breaks apart input stream into blocks and pipes each block into newly spawned processes

Project description

Blockpype

Breaks apart input stream into blocks and pipes each block into newly spawned processes

Installing

pip install git+https://github.com/kfzteile24/blockpype.git

Testing

Clone the repository and run visual tests that display each started PID and their output:

  • test/by-byte.sh - should break apart in the middle of the 2-byte character
  • test/by-char.sh - should break apart in the middle of some numbers
  • test/by-line.sh - should break apart nicely line by line

Usage

cat content.txt | blockpype -l 10 other-program --arg-for-program --another-arg "argument value"

It will cut content.txt into blocks of 10 lines, and for each block, a process will be started using the command:

other-program --arg-for-program --another-arg "argument value"

and the contents of the block will be served to stdin

What's it for?

Whenever you want to break apart a large stream when the other-program consumer stores it into memory until EOF, and you don't have enough memory for it.

Similar to split --filter "other-program --arg-for-program --another-arg \"argument value\"" except less clunky, other-program gets launched from current pwd with the current user, and with the current environment variables.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for blockpype, version 1.0.0a1
Filename, size File type Python version Upload date Hashes
Filename, size blockpype-1.0.0a1-py3.7.egg (4.7 kB) File type Egg Python version 3.7 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page