Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Profile manager of text processing pipelines: Pandoc filters, any text CLI filters. Atom+Markdown+Pandoc+Jupyter workflow, export to ipynb.

Project description

Pandoctools

Pandoctools is a combination of tools that help write reproducible markdown reports. They rely on Pandoc and Jupyter kernels.

Introduction articles:

“Glueing” part of pandoctools is a profile manager of text processing pipelines. It stores short crossplatform bash scripts that define chain operations over text. They are mostly Pandoc filters but any CLI text filter is OK.

Update instructions

(Update instructions to v.2.6)

  • v2.6 is not backward compatible but profiles can be easily fixed. Uninstall Pandoctools before updating. Update your custom bash scripts as names and logic changed. References: Default_args, Default (profile), Default_pipe.

Contents

Notable parts of Pandoctools

  • Pandoc, Jupyter, pandoc-crossref (dependence) - classical tools.
  • Pandoctools CLI app: profile manager of text processing pipelines. It stores short bash scripts - called profiles - that define chain operations over text. They are mostly Pandoc filters but any CLI text filter is OK. Profiles can be used to convert any document of choise in the specified manner.
  • Knitty (dependence): Knitty is a Pandoc filter and another CLI for Stitch/Knotr: reproducible report generation tool via Jupyter, Pandoc and Markdown. Insert python code (or other Jupyter kernel code) to the Markdown document and have code's results in the output document. Can even export to .html, .pdf, Jupyter .ipynb notebooks and any other Pandoc output formats. You can use ipynb-py-convert to convert .ipynb to .py to use with Knitty.
  • SugarTeX (dependence): SugarTeX is a more readable LaTeX language extension and transcompiler to LaTeX.
  • Pyppdf (dependence): Pyppeteer PDF. Prints html output to pdf via patched Pyppeteer.
  • Prism.js and github-markdown-css (integrated): used for default to PDF conversion (but with borrowing from Default_args to custom profile you can use them with to HTML conversion too).
  • libsass-python: tweak and write css with more convenient sass or scss (see Default.sass).
  • (optional) Tabulate Helper converts tabular data like Pandas dataframe to GitHub Flavored Markdown pipe table.
  • (optional) Matplotlib Helper: custom helper to tune Matplotlib experience in Atom/Hydrogen and Pandoctools/Knitty.
  • (optional) Feather Helper: concise interface to cache numpy arrays and pandas dataframes.
  • (optional) pypugjs: Write HTML via Pug that is much more readable.

Pandoctools is a tool for converting markdown document. But we also need tools for writing markdown and deploying python/Jupyter code blocks.
And the best one for it is:

Examples

Here are examples that demonstrate converting documents:

  • from markdown .md with Jupyter python code blocks, SugarTeX math and cross-references to .ipynb notebook and to PDF.
  • from Hydrogen/python notebook .py with Atom/Hydrogen code cells, Knitty markdown incerts (again with SugarTeX math and cross-references) to .ipynb notebook and to PDF.

Examples are given for to .ipynb and to .pdf conversion but Pandoctools surely capable of conversion to .html, .md.md or any Pandoc output format.

Extras:

  • If you need to capture Matplotlib plots please see matplotlibhelper (the approach showed in examples there can be used with other plot libraries).
  • If you need to autonumber sections see pandoc-crossref or this SE question
  • If you need criticmarkup support please consider using git repository with git-time-machine for tracking changes, <!-- html comments --> for adding notes, pigments for highlighting text.

Install

If you have an antivirus then the first or two runs may fail - there may be errors like "Permission denied" because of the antivirus checking all the components.

Short instructions:

  • Either (1.1) install 64-bit Miniconda3 and:
    (on Unix:)
    conda install -c defaults -c conda-forge pandoctools
    
    (on Windows:)
    conda install -c defaults -c conda-forge pandoctools git-bash
    
    (or install 64-bit Git Bash instead of the conda package)
  • Or (1.2) install 64-bit Python and:
    pip install pandoctools
    
    (if on Windows install 64-bit Git Bash)
  • Then (2):
    pandoctools-ready
    
  • But it's recommended to create a dedicated environment for the Pandoctools. See below.

Via conda

  • Create "pandoctools" conda environment (do not set custom prefix unless you want to set root_env in the config):
    (on Unix):
    cd $root_miniconda_prefix
    source ./bin/activate base
    conda config --add channels conda-forge
    conda config --add channels defaults
    conda update conda
    
    conda create -n pandoctools pandoctools
    source activate pandoctools
    pandoctools-ready
    
    (on Windows):
    cd /d %root_miniconda_prefix%
    call .\Scripts\activate base
    conda config --add channels conda-forge
    conda config --add channels defaults
    conda update conda
    
    conda create -n pandoctools pandoctools git-bash
    call activate pandoctools
    pandoctools-ready
    
    (or install 64-bit Git Bash instead of the conda package; local Bash in the environment has priority)
  • Just in case: the right way to remove conda environment 'myenv' is to run:
    conda remove -n myenv --all
    conda env remove -n myenv
    
    (in this particular order)

Via pip

  • Create pandoctools venv environment:
    (on Unix):
    cd $root_python_prefix
    ./bin/python -m venv ./envs/pandoctools
    source ./envs/pandoctools/bin/activate
    
    pip install pandoctools
    pandoctools-ready
    
    (on Windows):
    cd /d %root_python_prefix%
    .\python -m venv .\envs\pandoctools
    call .\envs\pandoctools\Scripts\activate
    
    pip install pandoctools
    pandoctools-ready
    
    (then install 64-bit Git Bash)
  • In contrast with conda installation Jupyter notebooks in pip do not support activated python kernels (there is a strange bug).

Useful tips (reload imported modules in Hydrogen, Python kernel, R kernel, Typescript kernel)

Useful tips

Alternatives to R Markdown (Markdown-based Literate Programming)

Alternatives to R Markdown

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pandoctools, version 2.6.0.6
Filename, size File type Python version Upload date Hashes
Filename, size pandoctools-2.6.0.6.tar.gz (193.6 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page