A pipeline framework for python
Documentation | ChangeLog | Examples | API
Features
- Easy to use
- Nearly zero-configuration
- Nice logging
- Highly extendable
Installation
pip install -U pipen
Quickstart
example.py
from pipen import Proc, Pipen
class P1(Proc):
"""Sort input file"""
input = "infile"
input_data = ["/tmp/data.txt"]
output = "outfile:file:intermediate.txt"
script = "cat {{in.infile}} | sort > {{out.outfile}}"
class P2(Proc):
"""Paste line number"""
requires = P1
input = "infile"
output = "outfile:file:result.txt"
script = "paste <(seq 1 3) {{in.infile}} > {{out.outfile}}"
Pipen().run(P1)
> echo -e "3\n2\n1" > /tmp/data.txt
> python example.py
[09/13/21 04:23:37] I main _____________________________________ __
[09/13/21 04:23:37] I main ___ __ \___ _/__ __ \__ ____/__ | / /
[09/13/21 04:23:37] I main __ /_/ /__ / __ /_/ /_ __/ __ |/ /
[09/13/21 04:23:37] I main _ ____/__/ / _ ____/_ /___ _ /| /
[09/13/21 04:23:37] I main /_/ /___/ /_/ /_____/ /_/ |_/
[09/13/21 04:23:37] I main
[09/13/21 04:23:37] I main version: 0.1.0
[09/13/21 04:23:37] I main
[09/13/21 04:23:37] I main ╭══════════════════════════════════ PIPEN-0 ═══════════════════════════════════╮
[09/13/21 04:23:37] I main ║ # procs = 2 ║
[09/13/21 04:23:37] I main ║ plugins = ['main', 'verbose-0.0.1'] ║
[09/13/21 04:23:37] I main ║ profile = default ║
[09/13/21 04:23:37] I main ║ outdir = pipen-0_results ║
[09/13/21 04:23:37] I main ║ cache = True ║
[09/13/21 04:23:37] I main ║ dirsig = 1 ║
[09/13/21 04:23:37] I main ║ error_strategy = ignore ║
[09/13/21 04:23:37] I main ║ forks = 1 ║
[09/13/21 04:23:37] I main ║ lang = bash ║
[09/13/21 04:23:37] I main ║ loglevel = info ║
[09/13/21 04:23:37] I main ║ num_retries = 3 ║
[09/13/21 04:23:37] I main ║ plugin_opts = {} ║
[09/13/21 04:23:37] I main ║ plugins = None ║
[09/13/21 04:23:37] I main ║ scheduler = local ║
[09/13/21 04:23:37] I main ║ scheduler_opts = {} ║
[09/13/21 04:23:37] I main ║ submission_batch = 8 ║
[09/13/21 04:23:37] I main ║ template = liquid ║
[09/13/21 04:23:37] I main ║ template_opts = {} ║
[09/13/21 04:23:37] I main ║ workdir = ./.pipen ║
[09/13/21 04:23:37] I main ╰══════════════════════════════════════════════════════════════════════════════╯
[09/13/21 04:23:37] I main
[09/13/21 04:23:37] I main ╭───────────────────────────────────── P1 ─────────────────────────────────────╮
[09/13/21 04:23:37] I main │ Sort input file │
[09/13/21 04:23:37] I main ╰──────────────────────────────────────────────────────────────────────────────╯
[09/13/21 04:23:37] I main P1: Workdir: '.pipen/pipen-0/p1'
[09/13/21 04:23:37] I main P1: <<< [START]
[09/13/21 04:23:37] I main P1: >>> ['P2']
[09/13/21 04:23:37] I verbose P1: size: 1
[09/13/21 04:23:37] I verbose P1: [0/0] in.infile: /tmp/data.txt
[09/13/21 04:23:37] I verbose P1: [0/0] out.outfile:
/home/pwwang/github/pipen/.pipen/pipen-0/p1/0/output/intermediate.txt
[09/13/21 04:23:38] I verbose P1: Time elapsed: 00:00:01.039s
[09/13/21 04:23:38] I main
[09/13/21 04:23:38] I main ╭═════════════════════════════════════ P2 ═════════════════════════════════════╮
[09/13/21 04:23:38] I main ║ Paste line number ║
[09/13/21 04:23:38] I main ╰══════════════════════════════════════════════════════════════════════════════╯
[09/13/21 04:23:38] I main P2: Workdir: '.pipen/pipen-0/p2'
[09/13/21 04:23:38] I main P2: <<< ['P1']
[09/13/21 04:23:38] I main P2: >>> [END]
[09/13/21 04:23:38] I verbose P2: size: 1
[09/13/21 04:23:38] I verbose P2: [0/0] in.infile:
/home/pwwang/github/pipen/.pipen/pipen-0/p1/0/output/intermediate.txt
[09/13/21 04:23:38] I verbose P2: [0/0] out.outfile: /home/pwwang/github/pipen/pipen-0_results/P2/result.txt
[09/13/21 04:23:40] I verbose P2: Time elapsed: 00:00:02.074s
[09/13/21 04:23:40] I main
PIPEN-0: 100%|█████████████████████████████████████████████████| 2/2 [00:04<00:00, 0.56 procs/s]
> cat ./pipen-0_results/P2/result.txt
1 1
2 2
3 3
Examples
See more examples at examples/ and a more realcase example at:
https://github.com/pwwang/pipen-report/tree/master/example
1 maintainer
Metadata
1 maintainer
Release files for pipen 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pipen-0.1.1.tar.gz | 33.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| pipen-0.1.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 71.3 kB
Release files / pipen-0.1.1.tar.gz
| Download URL | pipen-0.1.1.tar.gz |
|---|---|
| Size | 33.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a02e6399285930fda3ec3fcef621757dd7a8b024fd1053bdbd31995246562fcd
|
|
BLAKE2b-256 checksum How to use checksums |
0c3ca1d036d7e42d6d6ec6a89669af5adf0d0a7458672450172f300b438545c1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.10 CPython/3.9.5 Linux/4.19.128-microsoft-standard
|
Release files / pipen-0.1.1-py3-none-any.whl
| Download URL | pipen-0.1.1-py3-none-any.whl |
|---|---|
| Size | 37.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1ad76b3145711b839fdbfcc46ec2841e8ca56cab5b27840186e28063f562e02a
|
|
BLAKE2b-256 checksum How to use checksums |
1739b0d4734ead0a4230ffa764a2ef22b5f8a4354a2aa5c496ad8118f23db352
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.10 CPython/3.9.5 Linux/4.19.128-microsoft-standard
|
