Pyline is a grep-like, sed-like, awk-like command-line tool for line-based text processing in Python.
Project description
pyline
GitHub | PyPi | Warehouse | ReadTheDocs | Travis-CI
Pyline is a grep-like, sed-like, awk-like command-line tool for line-based text processing in Python.
Features
Compatibility with the original pyline recipe
Python str.split by an optional delimiter str (-F, --input-delim)
Python regex (-r, --regex, -R, --regex-options)
Output as txt, csv, tsv, json, html (-O csv, --output-filetype=csv)
Output as Markdown/ReStructuredText checkboxes (-O checkbox, --output-filetype=checkbox)
Lazy sorting (-s, --sort-asc; -S, --sort-desc)
Create path.py (or pathlib) objects from each line (-p, --path-tools)
Functional namedtuples, iterators yield -ing generators
optparse argument parsing (-h, --help)
cookiecutter-pypackage project templating
Why
Somewhat unsurprisingly, I found the original pyline recipe while searching for “python grep sed” (see AUTHORS.rst and LICENSE.psf).
I added an option for setting p = Path(line) in the eval/compile command context and added it to my dotfiles ; where it grew tests and an optparse.OptionParser; and is now promoted to a GitHub project with ReadTheDocs documentation, tests with tox and Travis-CI, and a setup.py for PyPi.
What
Pyline is an ordered MapReduce tool:
- Input Readers:
stdin (default)
file (codecs.open(file, 'r', encoding='utf-8'))
- Map Functions:
Python module imports (-m os)
Python regex pattern (-r '\(.*\)')
path library (p from --pathpy OR --pathlib)
Python codeobj eval output transform:
ls | pyline -m os 'line and os.path.abspath(line.strip())' ls | pyline -r '\(.*\)' 'rgx and (rgx.group(0), rgx.group(1)) or line' ls | pyline -p 'p and p.abspath() or ("# ".format(line))' # With an extra outer loop to bind variables in # (because (_p = p.abspath(); <codeobj>) does not work) find $PWD | pyline --pathpy -m os -m collections --input-delim='/' \ 'p and [collections.OrderedDict(( ("p", p), ("_p", _p), ("_p.split()", str(_p).split(os.path.sep)), ("line.rstrip().split()", line.rstrip().split(os.path.sep)), ("l.split()", l.split(os.path.sep)), ("words", words), ("w", w))) for _p in [p.abspath()]][0]' \ -O json
- Partition Function:
None
- Compare Function:
Result(collections.namedtuple).__cmp__
- Reduce Functions:
bool(), sorted()
- Output Writers:
ResultWriter classes
pyline -O csv pyline -O tsv pyline -O json
Installing
Install from PyPi:
pip install pyline
Install from GitHub as editable (add a pyline.pth in site-packages):
pip install -e git+https://github.com/westurner/pyline#egg=pyline
Usage
Print help:
pyline --help
Process:
# Print every line (null transform) cat ~/.bashrc | pyline line cat ~/.bashrc | pyline l # Number every line cat ~/.bashrc | pyline -n l # Print every word (str.split(input-delim=None)) cat ~/.bashrc | pyline words cat ~/.bashrc | pyline w # Split into words and print (default: tab separated) cat ~/.bashrc | pyline 'len(w) >= 2 and w[1] or "?"' # Select the last word, dropping lines with no words pyline -f ~/.bashrc 'w[-1:]' # Regex matching with groups cat ~/.bashrc | pyline -n -r '^#(.*)' 'rgx and rgx.group()' cat ~/.bashrc | pyline -n -r '^#(.*)' ## Original Examples # Print out the first 20 characters of every line tail access_log | pyline "line[:20]" # Print just the URLs in the access log (seventh "word" in the line) tail access_log | pyline "words[6]"
Work with paths and files:
# List current directory files larger than 1 Kb ls | pyline -m os \ "os.path.isfile(line) and os.stat(line).st_size > 1024 and line" # List current directory files larger than 1 Kb #pip install path.py ls | pyline -p 'p and p.size > 1024 and line'
Documentation
License
.
History
release/0.3.10 (2016-02-14 21:55:05 -0600)
git log --reverse --pretty=format:'* %s [%h]' v0.3.9..release/0.3.10
MRG: Merge tag ‘v0.3.9’ into develop [f7c8a16]
BUG,UBY: pyline.py: logging config (default INFO, -q/–quiet, -v/–verbose (DEBUG)) [8a060ab]
UBY,DOC: pyline.py: log.info((‘pyline.version’, __version__)) at startup [da1e883]
BUG,UBY: pyline.py: log.info((‘argv’, argv)) [ede1d5e]
BUG,REF: opts[‘cmd’], main->(int, results[]), log opts after all config [3cf9585]
UBY: pyline.py: log.info((‘_rgx’, _regexstr)) [02bd234]
RLS,DOC: setup.py,pyline.py: version 0.3.10 [ea6a1fd]
v0.3.9 (2016-02-14 17:58:36 -0600)
git log --reverse --pretty=format:'* %s [%h]' v0.3.8..v0.3.9
ENH: pyline.py: –version arg [a38bf5a]
MRG: Merge tag ‘v0.3.8’ into develop [85cd8e9]
BUG,REF: pyline.py: output-filetype/-> output-format [fbcd9e2]
BUG: pyline.py: only print version when opts.get(‘version’) [ef8ac20]
RLS,DOC: setup.py,pyline.py: version 0.3.9 [5f2c4a6]
DOC: HISTORY.rst: git-changelog.py –hdr=+ –rev ‘release/0.3.9’ | pbcopy [ce95bae]
MRG: Merge branch ‘release/0.3.9’ [38e0393]
v0.3.8 (2016-02-14 17:34:08 -0600)
git log --reverse --pretty=format:'* %s [%h]' v0.3.7..v0.3.8
MRG: Merge tag ‘v0.3.7’ into develop [0cd0e3c]
BUG,ENH: fix CSV header row; add -O jinja:template=path.jinja support (#1,) [d5fe67b]
ENH: pyline.py: –version arg [818fc1d]
RLS: setup.py, pyline.py: version 0.3.8 [245214d]
DOC: HISTORY.rst: git-changelog.py –hdr=+ –rev ‘release/0.3.8’ | pbcopy [983b535]
DOC: HISTORY.rst: git-changelog.py –hdr=+ –rev ‘release/0.3.8’ | pbcopy [7b65d8e]
MRG: Merge branch ‘release/0.3.8’ [2f5f249]
v0.3.7 (2016-02-12 20:04:39 -0600)
git log --reverse --pretty=format:'* %s [%h]' v0.3.6..v0.3.7
Merge tag ‘v0.3.5’ into develop [8c5de0a]
ENH: pyline.py: main(args=None, iterable=None, output=None) [dd490e1]
UBY: pyline.py: -O chk == -O checkbox [3aa96ce]
UBY: pyline.py: l = line = o = obj [3aa9a81]
DOC: pyline.py: -f/–in/–input-file, -o/–out/–output-file [bcc9eff]
TST: requirements-test.txt: nose, nose-parameterized, nose-progressive [213e0c0]
BUG: pyline: collections.OrderedDict, return 0 [5fd1114]
DOC: setup.py: install_requires=[] [a41bf30]
TST,BUG,CLN: test_pyline.py: chk, main(_args), docstrings, #opts._output.close() [0254f30]
Merge tag ‘v0.3.6’ into develop [f46f90c]
DOC,REF: pyline.py: type_func->typefunc, docstrings [08c8d9c]
UBY: pyline.py: [–input-delim-split-max|–max|–max-split] [b509726]
REF: pyline.py: ResultWriter.get_writer ValueError, expand [143c5f7]
DOC: pyline.py: usage docstring, main docstring [bc44747]
TST: tests/test_pylinepy: more tests of sorting [b60750a]
DOC: pyline.py: docstrings [89ea5c7]
BLD,TST,BUG: Makefile, setup.py, pyline.py, test_pyline.py: pyline.main does sorting, kwargs, opts obj [e80cde6]
TST,REF: split to SequenceTestCase, LoggingTestCase, Test* [62ff39b]
TST: tests/test_pyline.py: TestPylinePyline.test_30_pyline_codefunc [49928d5]
Merge branch ‘feature/split_tests’ into develop [ef63a18]
RLS,DOC: README.rst, setup.py, pyline.py 0.3.7 description [9fc262e]
Merge branch ‘release/0.3.7’ [07b00b2]
v0.3.6 (2015-12-21 04:17:23 -0600)
git log --reverse --pretty=format:'* %s [%h]' v0.3.5..v0.3.6
BUG: pyline.py: #!/usr/bin/env python2 [9729816]
RLS: HISTORY.rst, __init__.py, pyline.py, setup.py: __version__ = ‘0.3.6’ [a463d39]
Merge branch ‘hotfix/0.3.6’ [445c089]
v0.3.5 (2015-05-24 20:58:39 -0500)
git log --reverse --pretty=format:'* %s [%h]' v0.3.4..v0.3.5
Merge tag ‘v0.3.4’ into develop [3ec1391]
CLN: patchheader: rm [c9f6304]
ENH: pyline.py: add a codefunc() kwarg [be8dcc8]
BUG,DOC: pyline.py: set default regex_options to ‘’, optparse helpstrings [fa9e9cb]
DOC: pyline.py: docstrings (calling a function, stdlib/vendoring) [ee22e2c]
ENH,TST: pyline.py: add a codefunc() kwarg [91aa0a8]
RLS: setup.py, __init__, HISTORY: v0.3.5, git log –format=’* %s [%h]’ master..develop [78f3ad9]
Merge branch ‘release/0.3.5’ [065797d]
v0.3.4 (2015-04-25 06:48:47 -0500)
git log --reverse --pretty=format:'* %s [%h]' v0.3.3..v0.3.4
Merge tag ‘v0.3.3’ into develop [e630114]
RLS: HISTORY.rst, __init__.py, setup.py: v0.3.4 [e448183]
Merge branch ‘release/0.3.4’ [612228d]
v0.3.3 (2015-04-25 06:43:37 -0500)
git log --reverse --pretty=format:'* %s [%h]' v0.3.2..v0.3.3
Merge tag ‘v0.3.2’ into develop [061840b]
BUG: pyline.pyline.__main__ [db71796]
DOC,BLD,CLN: Makefile: sphinx-apidoc –no-toc [209bff8]
TST,CLN: pyline.py: remote -t/–test option [2629924]
DOC,CLN: modules.rst: remove generated modules.rst [abdc00d]
BUG, ENH, BUG, TST: [b5a21e7]
RLS: __init__.py, setup.py: v0.3.3 [eb81129]
BLD: Makefile: release (dist), twine [7e602c8]
Merge branch ‘release/0.3.3’ [c0df4ab]
v0.3.2 (2014-11-30 19:49:42 -0600)
git log --reverse --pretty=format:'* %s [%h]' v0.3.1..v0.3.2
Merge tag ‘v0.3.1’ into develop [a3f8c1c]
ENH: Add pyline.__main__ (pyline.pyline.main) for ‘python -m pyline’ [1bd5e10]
DOC: README.rst [a26d97a]
DOC: HISTORY.rst: link to Source: http://code.activestate.com/recipes/437932-pyline-a-grep-like-sed-like-command-line-tool/ [5871727]
DOC: usage.rst: add :shell: option to ‘pyline –help’ output [d1f32de]
BUG: pyline/__init__.py: Set pyline.pyline.__main__ correctly [49ae891]
DOC: pyline/pyline.py: docstrings, import path as pathpy [178af4e]
RLS: HISTORY.txt, pyline/__init__.py, setup.py: set version to v0.3.2 [6c547e4]
Merge branch ‘release/0.3.2’ [10b84f5]
v0.3.1 (2014-10-27 07:53:27 -0500)
git log --reverse --pretty=format:'* %s [%h]' v0.3.0..v0.3.1
Merge tag ‘v0.3.0’ into develop [35a380b]
DOC: README.rst [f803665]
Merge branch ‘hotfix/readme-travis-link’ [35f7b44]
Merge tag ‘vreadme-travis-link’ into develop [6849887]
DOC: setup.py version 0.3.1 [a7fae60]
Merge branch ‘release/0.3.1’ [276d16b]
v0.3.0 (2014-10-27 07:34:58 -0500)
git log --reverse --pretty=format:'* %s [%h]' v0.2.0..v0.3.0
Added tag v0.2.0 for changeset cddc5c513cd2 [c53a725]
DOC: Update README.rst: typo -output-filetype -> –output-filetype [6897954]
DOC: Update README.rst: update ‘Features’ [548c426]
DOC: Update README.rst: update ‘Features’ [273b475]
DOC: Update README.rst: update ‘Features’ [254ed95]
DOC: Update README.rst add additional link to docs [8415a7c]
BLD,DOC: Update requirements.txt: add ../ (from ./docs) as editable [d94ff0e]
Revert “BLD,DOC: Update requirements.txt: add ../ (from ./docs) as editable” [fa062b8]
DOC: program-output:: -> command-output:: [984b8a6]
ENH,BUG,CLN: #10, #12, #13 [a75d2f9]
CLN: remove _import_path_module [0cc9bb9]
RLS: pyline v0.3.0 [14941af]
Merge branch ‘release/0.3.0’ [53609dc]
v0.2.0 (2014-08-24 14:44:31 -0500)
git log --reverse --pretty=format:'* %s [%h]' v0.1.5..v0.2.0
Added tag v0.1.5 for changeset 8cd9c44a80ab [4bb3fc7]
BLD: Add docs for ‘make release’; remove bdist_wheel upload [e76b592]
BLD: Add docs for ‘make release’: HISTORY.rst [e5b3e9a]
ENH: Add checkbox output formatter (closes #5) [46b7177]
BUG: add NullHandler to logger (closes #6) [a9fac28]
RLS: Release v0.2.0 [9ef4a25]
Added tag v0.2.0 for changeset f510a75a37a8 [38c7eeb]
v0.1.5 (2014-05-12 20:59:34 -0500)
git log --reverse --pretty=format:'* %s [%h]' v0.1.4..v0.1.5
Added tag v0.1.4 for changeset c79a1068cb1c [0abdc5e]
DOC: setup.py keywords and classifiers [9079d03]
DOC: Update HISTORY.rst: 0.1.0 -> 0.1.5 [9bfe2a5]
BLD: bump version to v0.1.5 [0af9381]
v0.1.4 (2014-05-12 20:42:52 -0500)
git log --reverse --pretty=format:'* %s [%h]' v0.1.3..v0.1.4
Added tag v0.1.3 for changeset d49705961509 [4f8cfec]
DOC: correct license and download_url in setup.py [49ea953]
v0.1.3 (2014-05-12 20:30:47 -0500)
git log --reverse --pretty=format:'* %s [%h]' v0.1.2..v0.1.3
Added tag v0.1.2 for changeset 09cca8fa5555 [828d223]
DOC: missing newline in description [63a442c]
DOC: version bump, setup description [53ad0f4]
v0.1.2 (2014-05-12 20:24:26 -0500)
git log --reverse --pretty=format:'* %s [%h]' v0.1.1..v0.1.2
Added tag v0.1.1 for changeset 13ad121ea966 [5727951]
BLD: add pathlib and path.py to requirements.txt [aa6dda7]
DOC,BLD,BUG: setup.py build_long_description, file handles [f7a73c1]
DOC: README.rst: remove includes [2d2bd6f]
DOC: version bump, setup description [e920ff2]
v0.1.1 (2014-05-12 19:41:54 -0500)
git log --reverse --pretty=format:'* %s [%h]' v0.1.0..v0.1.1
DOC,BLD: Update AUTHORS.rst, HISTORY.rst, README.rst, docs/license.rst [7b087c8]
CLN: pyline rename arg[0] _input -> iterable [7040271]
BUG: default command in – ls | pyline -p “ p = path = Path(line.strip()) [30dce3a]
LOG: log.info(cmd) … after shell parsing, exception [c449765]
CLN: pep8 test command kwargs formatting [993c65a]
DOC: README.rst; ReST doesn’t seem to like path.py .. _path.py:, links [209ecb5]
TST: Update setup.py test command (runtests -v ./tests/test_*.py) [bc84652]
TST: tox.ini: make html rather than sphinx-build [c96b3b0]
CLN: factor out _import_pathmodule and get_path_module [d0aebfb]
TST: move tests from pyline.py to tests/test_pyline.py [477fbb4]
BUG: file handles (was causing tests to fail silently) [80e84b6]
CLN: move optparse things into get_option_parser() [723a8b7]
BLD: Release 0.1.1 [3f9f56f]
v0.1.0 (2014-05-12 04:03:15 -0500)
git log --reverse --pretty=format:'* %s [%h]' b1303ba..v0.1.0
CLN: Update .gitignore and .hgignore [0d07ad1]
DOC: Update README.rst: comment out unconfigured badges [b0e0fc1]
ENH: Add pyline script from https://github.com/westurner/dotfiles/blob/e7f766f3/src/dotfiles/pyline.py [ce2dba8]
BLD,TST: Add py.test runtests.py and setup.py:PyTestCommand [953edbe]
BUG: try/except import StringIO (Python 3 compatibility) [97d5781]
BLD: remove py33 section from tox.ini for now [b103587]
BLD: remove py33 section from tox.ini for now [2ff4a77]
BLD: Update tox.ini, .travis.yml, reqs, docs/conf [13b5487]
CLN: pyline cleanup [9724f8e]
CLN: update .hgignore [59196b7]
0.0.1 (Unreleased)
Updated 2012.11.17, Wes Turner
Updated 2005.07.21, thanks to Jacob Oscarson
Updated 2006.03.30, thanks to Mark Eichin
Credits
Graham Fawcett
Jacob Oscarson
Mark Eichin
Wes Turner – https://github.com/westurner
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pyline-0.3.10-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 855bad49b7cb964bf58155b205910c29ef4ade99a7a2f31d70cc681e1c4e9122 |
|
MD5 | c58e6ccb049d105f50bb0980601ff481 |
|
BLAKE2b-256 | bc527bceffe17ce47dfeb3a728c3e091cfde4aeeb0bc59efff4288d011c5df43 |