Skip to main content

Breaks apart input stream into blocks and pipes each block into newly spawned processes

Project description

Blockpype

Breaks apart input stream into blocks and pipes each block into newly spawned processes

Installing

pip install git+https://github.com/kfzteile24/blockpype.git

Testing

Clone the repository and run visual tests that display each started PID and their output:

  • test/by-byte.sh - should break apart in the middle of the 2-byte character
  • test/by-char.sh - should break apart in the middle of some numbers
  • test/by-line.sh - should break apart nicely line by line

Usage

cat content.txt | blockpype -l 10 other-program --arg-for-program --another-arg "argument value"

It will cut content.txt into blocks of 10 lines, and for each block, a process will be started using the command:

other-program --arg-for-program --another-arg "argument value"

and the contents of the block will be served to stdin

What's it for?

Whenever you want to break apart a large stream when the other-program consumer stores it into memory until EOF, and you don't have enough memory for it.

Similar to split --filter "other-program --arg-for-program --another-arg \"argument value\"" except less clunky, other-program gets launched from current pwd with the current user, and with the current environment variables.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

blockpype-1.0.0a1-py3.7.egg (4.7 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page