Skip to main content

Profile manager of text processing pipelines: Pandoc filters, any text CLI filters. Atom+Markdown+Pandoc+Jupyter workflow, export to ipynb.

Project description

Pandoctools

Pandoctools is a combination of tools that help write reproducible markdown reports. They rely on Pandoc and Jupyter kernels.

Introduction articles:

“Glueing” part of pandoctools is a profile manager of text processing pipelines. It stores short crossplatform bash scripts that define chain operations over text. They are mostly Pandoc filters but any CLI text filter is OK.

Update instructions

(Update instructions to v.1.3.7)

  • Switch to bash profiles as batch profiles are no longer supported (and install bash if needed),
  • results=pandoc was a misunderstanding. The right way to output Markdown is to use from IPython.display import Markdown; Markdown('hello').
  • Import pandas, matplotlib and feather helpers from separate modules: matplotlibhelper, featherhelper, tabulatehelper,
  • v1.3.7 is not backward compatible but profiles can be easily fixed. Uninstall Pandoctools before updating. Update your custom bash scripts as names and logic changed. References: Default_args, Default (profile), Default_pipe.

Notable parts of Pandoctools

  • Pandoc, Jupyter, pandoc-crossref (dependence) - classical tools.
  • Pandoctools CLI app: profile manager of text processing pipelines. It stores short bash scripts - called profiles - that define chain operations over text. They are mostly Pandoc filters but any CLI text filter is OK. Profiles can be used to convert any document of choise in the specified manner.
  • Knitty (dependence): Knitty is a Pandoc filter and another CLI for Stitch/Knotr: reproducible report generation tool via Jupyter, Pandoc and Markdown. Insert python code (or other Jupyter kernel code) to the Markdown document and have code’s results in the output document. Can even export to Jupyter ipynb notebooks. You can use vscode-ipynb-py-converter to convert .ipynb to .py to use with Knitty.
  • SugarTeX (dependence): SugarTeX is a more readable LaTeX language extension and transcompiler to LaTeX.
  • Pyppdf (dependence): Pyppeteer PDF. Prints html output to pdf via patched Pyppeteer.
  • (optional) Tabulate Helper converts tabular data like Pandas dataframe to GitHub Flavored Markdown pipe table.
  • (optional) Matplotlib Helper: custom helper to tune Matplotlib experience in Atom/Hydrogen and Pandoctools/Knitty.
  • (optional) Feather Helper: concise interface to cache numpy arrays and pandas dataframes.
Pandoctools is a tool for converting markdown document. But we also need tools for writing markdown and deploying python/Jupyter code blocks.
And the best one for it is:

Examples

Here are examples that demonstrate converting documents:

  • from markdown .md with Jupyter python code blocks, SugarTeX math and cross-references to ipynb notebook.
  • from Hydrogen/python notebook .py with Atom/Hydrogen code cells, Knitty markdown incerts (again with SugarTeX math and cross-references) to .ipynb notebook.

Examples are given for to .ipynb conversion but Pandoctools surely capable of conversion to .html, .pdf, .md.md or any Pandoc output format.

Extras:

  • If you need to capture Matplotlib plots please see matplotlibhelper (the approach showed in examples there can be used with other plot libraries).

Install

Windows:

Via conda:

  • Install Miniconda,

  • Install Git together with Bash. Git is needed for writing text conversion profiles in cross-platform bash language,

  • Fresh install preparations (incl. creating “myenv” conda environment):

    call activate root
    conda update conda
    conda create -n myenv python=3 pip setuptools
    
    call activate myenv
    conda update python pip setuptools
    
  • Pandoctools installation:

    conda install -c defaults -c conda-forge "pip>=10.0.1" "pandoc>=2.3.1" ^
    click pyyaml notebook jupyter future shutilwhich ^
    certifi websockets appdirs urllib3 tqdm ^
    jupyter_core traitlets ipython jupyter_client nbconvert pandocfilters ^
    pypandoc psutil nbformat pandoc-attributes pywin32
    
    pip install pandoctools pandoctools-ready
    
  • Install latest stable pandoc-crossref (compatible with pandoc version) to <miniconda-path>/envs/myenv/Library/bin,

  • Tips:

    • if pip install fails try to change codepage: chcp 1252,
    • If Pandoc errors try downgrade to "pandoc>=2.0,<2.1" and pandoc-crossref v0.3.0.1,
    • Should be "conda>=4.5.4" (conda update conda should be enough).

Via pip:

  • Install Git together with Bash. Git is needed for writing text conversion profiles in cross-platform bash language,

  • :

    pip install pandoctools pandoctools-ready
    
  • Install latest stable pandoc-crossref (compatible with pandoc version) to virtual environment’s .\Scripts folder.

Unix:

Via conda:

  • Install Miniconda,

  • Fresh install preparations (incl. creating “myenv” conda environment):

    source activate root
    conda update conda
    conda create -n myenv python=3 pip setuptools
    
    source activate myenv
    conda update python pip setuptools
    
  • Pandoctools installation:

    conda install -c defaults -c conda-forge "pip>=10.0.1" "pandoc>=2.3.1" \
    click pyyaml notebook jupyter future shutilwhich \
    certifi websockets appdirs urllib3 tqdm \
    jupyter_core traitlets ipython jupyter_client nbconvert pandocfilters \
    pypandoc psutil nbformat pandoc-attributes
    
    pip install pandoctools pandoctools-ready
    
  • Install latest stable pandoc-crossref (compatible with pandoc version) to <miniconda-path>/envs/myenv/bin,

  • Tips:

    • If Pandoc errors try downgrade to "pandoc>=2.0,<2.1" and pandoc-crossref v0.3.0.1,
    • Should be "conda>=4.5.4" (conda update conda should be enough).

Via pip:

  • :

    pip install pandoctools pandoctools-ready
    
  • Install latest stable pandoc-crossref (compatible with pandoc version) to virtual environment’s ./bin folder.

Useful tips (reload imported modules in Hydrogen, R kernel, LyX)

Useful tips

Alternatives to R Markdown (Markdown-based Literate Programming)

Alternatives to R Markdown

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
pandoctools-1.3.11.tar.gz (50.8 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page