rstdoc 1.6.0

rstdoc - support documentation in restructedText (rst)

rstdoc
======


Many companies use `DOCX`_ and thus produce an information barrier.
Working with text is more integrated in the (software) development process.
A final format can be `DOCX`_, but, at least during development, text is better.

`Sphinx`_ is an extension of `Docutils`_ used for many (software) projects,
but it does not support creation of `DOCX`_ files, which certain companies demand.
`Pandoc`_ does support `DOCX`_, but does not support the `Sphinx`_ extensions,
hence ``:ref:`` and the like cannot be used.

This python package supports working with `RST`_ as documentation format without depending on `Sphinx`_.

- link `RST`_ documents (``.rest``) using `substitutions`_ (generated in ``_links_xxx.rst``)
- create a ``.tags`` file to jump around in an editor that support `ctags`_
- `RST`_ handling with python: reformat/create `RST`_ tables
- postprocess `Pandoc`_'s conversion from `DOCX`_ to `RST`_
- preprocess `Pandoc`_'s conversion from `RST`_ to `DOCX`_
- Support in building with `WAF`_ (or ``Makefile``)

- expand `SimpleTemplate`_ template files ``.stpl``
- ``.tikz``, ``.svg``, ``.dot``, ``.uml``, ``.eps`` or ``.stpl`` thereof, and ``.pyg`` for Python-generated graphics, are converted to ``.png``
and placed into ``./_images`` or ``../_images``
- a ``gen`` file specifies how `RST`_ should be generated from source code files (see ``dcx.py``)

The conventions used are shown

- by the example produced via ``rstdcx --init samplerstdoc`` or ``rstdcx --stpl sampletemplated``
- by the documentation sources that can be found at
https://github.com/rpuntaie/rstdoc/tree/master/src/doc

``pip install rstdoc`` installs:

+-----------+--------------+--------------------------------------------+
| Module | Script | Description |
+===========+==============+============================================+
| dcx | rstdcx | create ``.tags``, labels and links |
+-----------+--------------+--------------------------------------------+
| fromdocx | rstfromdocx | Convert DOCX to RST using Pandoc |
+-----------+--------------+--------------------------------------------+
| listtable | rstlisttable | Convert RST grid tables to list-tables |
+-----------+--------------+--------------------------------------------+
| untable | rstuntable | Converts certain list-tables to paragraphs |
+-----------+--------------+--------------------------------------------+
| reflow | rstreflow | Reflow paragraphs and tables |
+-----------+--------------+--------------------------------------------+
| reimg | rstreimg | Rename images referenced in the RST file |
+-----------+--------------+--------------------------------------------+
| retable | rstretable | Transforms list tables to grid tables |
+-----------+--------------+--------------------------------------------+

- ``html``, ``docx``, ``pdf``, ... uses `Pandoc`_.

- ``rst_html``, ``rst_pdf``, ... uses `rst2html`_, ...

- ``sphinx_html``, ``sphinx_pdf``, ... uses `Sphinx`_.
`Sphinx`_ provides a nice entry point via the `sphinx bootstrap theme`_.


.. _`editors`: http://build-me-the-docs-please.readthedocs.io/en/latest/Using_Sphinx/ToolsForReStructuredText.html
.. _`Emacs`: http://docutils.sourceforge.net/docs/user/emacs.html
.. _`ctags`: http://ctags.sourceforge.net/FORMAT
.. _`Sphinx`: http://www.sphinx-doc.org/en/stable/
.. _`Docutils`: http://docutils.sourceforge.net/
.. _`Pandoc`: https://pandoc.org/
.. _`RST`: http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html
.. _`DOCX`: http://www.ecma-international.org/publications/standards/Ecma-376.htm
.. _`SimpleTemplate`: https://bottlepy.org/docs/dev/stpl.html#simpletemplate-syntax
.. _`waf`: https://github.com/waf-project/waf
.. _`sphinx bootstrap theme`: https://github.com/ryan-roemer/sphinx-bootstrap-theme
.. _`rst2html`: http://docutils.sourceforge.net/0.6/docs/user/tools.html
.. _`substitutions`: http://docutils.sourceforge.net/0.6/docs/ref/rst/restructuredtext.html#substitution-definitions



.. _`rstdcx`:

rstdcx
======

Support script to create documentation (PDF, HTML, DOCX)
from restructuredText (RST, reST) using either

- `Pandoc <https://pandoc.org>`__
- `Sphinx <http://www.sphinx-doc.org>`__
- Docutils `configurable <http://docutils.sourceforge.net/docs/user/config.html>`__

Usage
-----

``rstdoc`` installs the ``rstdcx`` command line tool that calls ``dcx.py``.
It

- processes ``gen`` files (see examples produced by --rest)

- handles `.stpl <https://bottlepy.org/docs/dev/stpl.html>`__ files

- creates ``.tags`` to jump around with the editor

- creates links files like ``_links_pdf.rst``, ``_links_docx.rst``, ``_links_sphinx.rst``

- forwards known files to either Pandoc, Sphinx or Docutils

See example at the end of ``dcx.py``.

It is supposed to be used with a build tool. ``make`` and ``waf`` examples are included.

- Initialize example tree.
This copies ``dcx.py`` into the example tree to be independent from possible future changes.

$ ./dcx.py --rest tmp #.rest files OR
$ ./dcx.py --stpl tmp #.rest.stpl files

- Only create .tags and _links_xxx.rst::

$ cd tmp/src/doc
$ ./dcx.py

- Create the docs (and .tags and _links_xxx.rst) with **make**::

$ make html #OR
$ make epub #OR
$ make latex #OR
$ make docx #OR
$ make pdf

The latter two are done by Pandoc, the others by Sphinx.

- Create the docs (and .tags and _links_xxx.rst) with **waf** (preferred):

Instead of using ``make`` one can load ``dcx.py`` in `waf <https://github.com/waf-project/waf>`__.
``waf`` also considers all recursively included files,
such that a change in any of them results in a rebuild of the documentation.
All files can have an additional ``.stpl`` extension to use `SimpleTemplate <https://bottlepy.org/docs/dev/stpl.html#simpletemplate-syntax>`__.

$ waf configure #copies the latest version of waf in here
$ waf --docs docx,sphinx_html,rst_odt

- ``rst_xxx`` via `rst2xxx.py <http://docutils.sourceforge.net/docs/user/tools.html>`__
- ``sphinx_xxx`` via `Sphinx <http://www.sphinx-doc.org>`__ and
- just ``xxx`` via `Pandoc <https://pandoc.org>`__.


The following image language files should be parallel to the ``.rest`` files.
They are automatically converted to ``.png`` and placed into ``./_images`` or ``../_images``.

- ``.tikz`` or ``.tikz.stpl``.
This needs LaTex.

- `.svg <http://svgpocketguide.com/book/>`__ or ``.svg.stpl``

- ``.dot`` or ``.dot.stpl``

This needs `graphviz <https://graphviz.gitlab.io/gallery/>`__.

- `.uml <http://plantuml.com/command-line>`__ or ``.uml.stpl``

This needs `plantuml <http://plantuml.com/command-line>`__ .
Provide either

- ``plantuml.bat`` with e.g. ``java -jar "%~dp0plantuml.jar" %*`` or
- ``plantuml`` sh script with ``java -jar `dirname $BASH_SOURCE`/plantuml.jar "$@"``

- ``.eps`` or ``.eps.stpl`` embedded postscript files.

This needs `inkscape <https://inkscape.org/en/>`__.

- ``.pyg`` contains python code that produces a graphic.
If the python code defines a ``save_to_png`` function,
then that is used, which allows to use whatever python library you want
(`graph_tool <https://graph-tool.skewed.de/static/doc/quickstart.html>`__,
`igraph <http://igraph.org/python/doc/tutorial/tutorial.html>`__,...)
Else the following is tried

- ``pyx.canvas.canvas`` from the `pyx <http://pyx.sourceforge.net/manual/graphics.html>`__ library or
- ``cairocffi.Surface`` from `cairocffi <https://cairocffi.readthedocs.io/en/stable/overview.html#basic-usage>`__
- ``pygal.Graph`` from `pygal <https://pygal.org>`__
- `matplotlib <https://matplotlib.org>`__. If ``matplotlib.pyplot.get_fignums()>1`` the figures result ``<name><fignum>.png``

Conventions
-----------

- Files

- main docs end in ``.rest``
- ``.rst`` are included and ignored by Sphinx (see conf.py).
- ``.txt`` are literally included (use :literal: option).
- templates ``x.rest.stpl`` and ``y.rst.stpl`` are rendered separately.
- ``some.rst.tpl`` are template included
Template lookup is done in in ``.`` and ``..`` with respect to the current file.

- with ``%include('some.rst.tpl',param="test")`` with optional parameters
- with ``%globals().update(include('utility.rst.tpl'))`` if it contains only definitions

- ``.. _`id`:`` are *reST targets*.
reST targets should not be template-generated.
The template files should have a higher or equal number of targets than the generated file,
in order for tags to jump to the template original.
If one wants to generate also reST targets, then this should happen in a previous step,
e.g. with ``gen`` files mentioned above.

- References use replacement `substitutions`_ : ``|id|``.

- Add ``.. include:: _traceability_file.rst`` to ``index.rest`` or another ``.rest`` file to get traceability information generated

See the example created with ``--rest`` of ``--stpl`` at the end of this file and the sources of the documentation of
`rstdoc <https://github.com/rpuntaie/rstdoc>`__.

.. _`substitutions`: http://docutils.sourceforge.net/docs/ref/rst/directives.html#replacement-text


``rstdcx`` CLI
--------------

Without parameters: creates |substitution| links and .tags ctags for reST targets.

With two or three parameters: process files through Pandoc, Sphinx, Docutils (.rest);
Inkscape (.eps, .svg), Dot (.dot), Planuml (.uml), latex (.tex,.tikz)
are converted to .png into ``./_images`` or ``../_images``.
Any of the files can be a SimpleTemplate templates (xxx.yyy.stpl).

Configuration is in ``conf.py`` or ``../conf.py``.

Examples with the files generated with the ``--stpl tmp``:

.. code-block:: sh

cd tmp/src/doc
rstdcx #expand .stpl and produce _links_xxx.rst and .tags

#expand stpl and append substitutions (for simple expansion use ``stpl <file> .``)
rstdcx dd.rest.stpl - rest # expand to stdout, appending dd.html substitutions, to pipe to Pandoc
rstdcx dd.rest.stpl - html. # as before
rstdcx dd.rest.stpl - docx. # expand to stdout, appending dd.docx substitutions, to pipe to Pandoc
rstdcx dd.rest.stpl - newname.docx. # expand template, appending substitutions for target newname.docx
rstdcx dd.rest.stpl - html # expand to stdout, already process through Pandoc to produce html on stdout
rstdcx dd.rest.stpl # as before
rstdcx sy.rest.stpl - rst_html # expand template, already process through Docutils to produce html on stdout
stpl sy.rest.stpl | rstdcx - - sy.html. # appending sy.html substitutions, e.g. to pipe to Pandoc
stpl dd.rest.stpl | rstdcx - - dd.html # appending tp.html substitutions and produce html on stdout via Pandoc
rstdcx dd.rest.stpl dd.rest # expand into dd.rest, appending substitutions for target dd.html
rstdcx dd.rest.stpl dd.html html # expand template, process through Pandoc to produce dd.html
rstdcx dd.rest.stpl dd.html # as before
rstdcx dd.rest.stpl dd.html rst_html # expand template, already process through Docutils to produce dd.html
rstdcx dd.rest.stpl dd.docx # expand template, process through Pandoc to produce dd.docx
rstdcx dd.rest.stpl dd.odt pandoc # expand template, process through Pandoc to produce dd.odt
rstdcx dd.rest.stpl dd.odt # as before
rstdcx dd.rest.stpl dd.odt rst_odt # expand template, process through Docutils to produce dd.odt
rstdcx dd.rest.stpl dd.odt rst # as before

#Sphinx is not file-oriented
#but with rstdcx you need to provide the files to give Sphinx ``master_doc`` (normally: index.rest)
#Directly from ``.stpl`` does not work with Sphinx
rstdcx index.rest ../../build/index.html sphinx_html # via Sphinx the output directory must be different

#convert the graphics and place the into _images or ../_images
#if no _images directory exists they will placed into the same folder
rstdcx egcairo.pyg
rstdcx egdot.dot.stpl
rstdcx egeps.eps
rstdcx egeps1.eps
rstdcx egother.pyg
rstdcx egplt.pyg
rstdcx egpygal.pyg
rstdcx egpyx.pyg
rstdcx egsvg.svg.stpl
rstdcx egtikz.tikz
rstdcx egtikz1.tikz
rstdcx eguml.uml

#convert graphics to a png here (even if _images directory exists)
rstdcx eguml.uml eguml.png



.. _`rstfromdocx`:

rstfromdocx
===========

rstfromdocx: shell command
fromdocx: rstdoc module

Convert DOCX to RST in a subfolder of current dir, named after the DOCX file.
It also creates ``conf.py``, ``index.py`` and ``Makefile``
and copies ``dcx.py`` into the folder.

See rstdcx_ for format conventions for the RST.

There are options to post-process through::

--listtable (--join can be provided)
--untable
--reflow (--sentence True, --join 0)
--reimg

``rstfromdocx -lurg`` combines all of these.

To convert more DOCX documents into the same
RST documentation folder, proceed like this:

- rename/copy the original DOCX to the name you want for the ``.rest`` file
- run ``rstfromdocx -lurg doc1.docx``; instead of -lurg use your own options
- check the output in the ``doc1`` subfolder
- repeat the previous 2 steps with the next DOCX files
- create a new folder, e.g. ``doc``
- merge all other folders into that new folder

``fromdocx.docx_rst_5`` creates 5 different rst files with different postprocessing.

See rstreflow_ for an alternative proceeding.


.. _`rstlisttable`:

rstlisttable
============

rstlisttable: shell command
listable: rstdoc module

Convert RST grid tables to list-tables.

Usage
-----

#. Convert grid tables in a file to list-tables. The result is output to stdout::

$ listtable.py input.rst

#. Convert several files::

$ listtable.py input1.rst input2.rst
$ listtable.py *.rst

#. Use pipe (but ``cat`` might not keep the encoding)::

$ cat in.rst | listtable.py - | untable.py - > out.rst

Options
-------
-j, --join e.g.002. Join method per column: 0="".join; 1=" ".join; 2="\\n".join



.. _`rstuntable`:

rstuntable
==========

rstuntable: shell command
untable: rstdoc module

Convert tables of following format to paragraphs with an ID.
The '-' in ID is removed and the ID is made lower case.
**Bold** is removed.

.. list-table::
:widths: 50 50
:header-rows: 0

* - **ID-XY-00**
- text goes here

* - **ID-XY-01**
- text again goes here


If the first entry does contain no word chars or spaces between words,
then the table stays. For a different behavior replace paragraph23.

A file produced from a docx using pandoc or ``fromdocx.py`` will
need a pre-processing via ``rstlisttable`` to convert grid tables to ``list-table`` tables.
This is done in one step with ``rstfromdocx -lu doc.rst``.


.. _`rstreflow`:

rstreflow
=========

rstreflow: shell command
reflow: rstdoc module

Reflow tables and paragraphs in a rst document produced from a docx.

Post-process a docx in this order::

rstfromdocx doc.docx
rstlisttable doc/doc.rst > doc/tmp.rst
rstuntable doc/tmp.rst > doc/tmp1.rst
rstreflow doc/tmp1.rst > doc/tmp2.rst
rstreimg doc/tmp2.rst > doc/tmp3.rst
rm doc/doc.rst
mv doc/tmp3.rst doc/doc.rst
rm doc/tmp*

Check the intermediate results.

Else one can also do inplace::

rstfromdocx doc.docx
rstlisttable -i doc/doc.rst
rstuntable -i doc/doc.rst
rstreflow -i doc/doc.rst
rstreimg -i doc/doc.rst

.. note:: DOCX to RST using Pandoc

``rstfromdocx -lurg doc.rst`` converts a docx to RST and
does all the post-processing in one step.

It is adviced, though, to compare the output with the original and do some manual
corrections here and there.


.. _`rstreimg`:

rstreimg
========

rstreimg: shell command
reimg: rstdoc module

Reimg renames the images in the rst file and the files themselves.
It uses part of the document name and a number as new names.

This is useful, if more RST documents converted from DOCX
should be combined in one directory and
the names of the images have no meaning (image13,...).


.. _`rstretable`:

rstretable
==========

rstretable: shell command
retable: rstdoc module

Transforms list tables to grid tables.

This file also contains the code from
the Vim plugin `vim-rst-tables-py3`_, plus some little fixes.
``rstdoc`` is used by the Vim plugin `vim_py3_rst`_, which replaces `vim-rst-tables-py3`_.

.. _`vim-rst-tables-py3`: https://github.com/ossobv/vim-rst-tables-py3
.. _`vim_py3_rst`: https://github.com/rpuntaie/vim_py3_rst