Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

include files to rst for documentation purposes

Project description

rst_include

Pypi Status Python Version license maintenance

Build Status Codecov Status Better Code Maintainability snyk security

since You can not include files into RST files on github and PyPi, You can replace those imports with this software.

That means You can locally write Your RST documents (for instance with pycharm) and use there the .. include: option to include other RST Files or code snippets into Your Document. Afterwards You can run this software to create a monolithic README.rst that can be viewed on Github or Pypi

You might also include Text/Code from Jupyter Notebooks (sorry, no pictures at the moment, but it is not very hard to do that)

This has many advantages like :

  • dont repeat Yourself, create standard blocks to include into Your documentation
  • include tested code snippets from Your code files into Your documentation, to avoid untested or outdated documentation
  • include other RST Files
  • very simple usage, throwing exit codes to detect errors on documentation at travis build-time
  • commandline or programmatic interface, You can even use it in the travis.yml
  • commandline interface supporting shellscript, cmd, pipes, config-files

This README was also created with rst_include, You might look at ./docs/README_template.rst , build_docs.sh, build_docs.cmd and build_docs.py as examples. (they all do the same, just different versions)

The travis.yml builds the Documentation on every run, so You can be sure that there are no Errors. rst_include does only work on python > 3.6

100% code coverage, mypy static type checking, tested under Linux, OsX, Windows and Wine, automatic daily builds and monitoring



Installation and Upgrade

From source code:

# normal install
python setup.py install
# test without installing
python setup.py test

via pip latest Release:

# latest Release from pypi
# under Linux You have to use sudo, or it will not be installed as a commandline application
# [sudo] means, that the command "sudo" is optional for Linux if You want to use it from bash commandline
[sudo] pip3 install rst_include

# test without installing
[sudo] pip3 install rst_include --install-option test

via pip latest Development Version:

# upgrade all dependencies regardless of version number (PREFERRED)
[sudo] pip3 install --upgrade https://github.com/bitranox/rst_include/archive/master.zip --upgrade-strategy eager
# normal install
[sudo] pip3 install --upgrade https://github.com/bitranox/rst_include/archive/master.zip
# test without installing
[sudo] pip3 install https://github.com/bitranox/rst_include/archive/master.zip --install-option test

via requirements.txt:

# Insert following line in Your requirements.txt:
# for the latest Release:
rst_include
# for the latest Development Version :
https://github.com/bitranox/rst_include/archive/master.zip

# to install and upgrade all modules mentioned in requirements.txt:
[sudo] pip3 install --upgrade -r /<path>/requirements.txt

via python:

# for the latest Release
[sudo] python3 -m pip install upgrade rst_include

# for the latest Development Version
[sudo] python3 -m pip install upgrade https://github.com/bitranox/rst_include/archive/master.zip

Basic Usage

since rst_include is registered as a console script command with Your current python interpreter, You have to use the command “rst_include” (not “rst_include.py”)

  • issue command :
# issue command on shell or windows commandline
$> rst_include [OPTIONS]

# or, if python/bin is not in Your python path :
# on Windows
$> c:\python37\scripts\rst_include [OPTIONS]
# on Linux/oSX
$> /python37/bin/rst_include [OPTIONS]

# issue command with python interpreter
$> python -m rst_include [OPTIONS]
  • get help :
# get help on shell or windows commandline
$> rst_include -h
usage: rst_include [-h] {include,replace} ...

Process .rst File Includes

positional arguments:
  {include,replace}
    include          include rst includes
    replace          string replace

optional arguments:
  -h, --help         show this help message and exit

check the documentation on github
# get help on shell or windows commandline for include
$> rst_include include -h
usage: rst_include include [-h] [-s [source]] [-t [target]]
                           [-se [source encoding]] [-te [target encoding]]
                           [-i] [-q] [-c [configfile.py]]

optional arguments:
  -h, --help            show this help message and exit
  -s [source], --source [source]
                        default: stdin
  -t [target], --target [target]
                        default: stdout
  -se [source encoding], --source_encoding [source encoding]
                        default: utf-8-sig
  -te [target encoding], --target_encoding [target encoding]
                        default: utf-8
  -i, --inplace         inplace - target file = sourcefile
  -q, --quiet           quiet
  -c [configfile.py], --config [configfile.py]
                        If no filename is passed, the default conf_rst_inc.py
                        is searched in the current directory
# get help on shell or windows commandline for string replace
$> rst_include replace -h
usage: rst_include replace [-h] [-s [source]] [-t [target]]
                           [-se [source encoding]] [-te [target encoding]]
                           [-i] [-q]
                           old new [count]

positional arguments:
  old                   old
  new                   new
  count                 count

optional arguments:
  -h, --help            show this help message and exit
  -s [source], --source [source]
                        default: stdin
  -t [target], --target [target]
                        default: stdout
  -se [source encoding], --source_encoding [source encoding]
                        default: utf-8-sig
  -te [target encoding], --target_encoding [target encoding]
                        default: utf-8
  -i, --inplace         inplace - target file = sourcefile
  -q, --quiet           quiet
  • replace the include statements in source.rst and save it to target.rst via commandline parameters :
# replace the include statements on shell or windows commandline
# path can be relative or absolute path
# examples :

# relativ path
$> rst_include include -s ./source.rst -t ./target.rst

# absolute path
$> rst_include include -s /project/docs/source.rst -t /project/docs/target.rst

# on linux via pipe
$> cat /project/docs/source.rst | rst_include include > /project/docs/target.rst

# on Windows via pipe
$> type /project/docs/source.rst | rst_include include > /project/docs/target.rst
  • replace include statements on multiple files via config.py :
# replace the include statements on shell or windows commandline
# path to the config file can be absolute or relative path
# option -c or --config :

# will try to load the default conf_rst_inc.py from the current directory
$> rst_include include -c

# will load another config file another directory
$> rst_include include -c ./conf_this_project.py

Structure of the configuration file:

the files are processed in the given order, by that way You can even realize nested .. include:: blocks.

You might also specify the encoding for source and target files

from rst_include import *

# set config here
rst_conf = RstConf()

# paths absolute, or relative to the location of the config file
# the notation for relative files is like on windows or linux - not like in python.
# so You might use ../../some/directory/some_document.rst to go two levels back.
# avoid absolute paths since You never know where the program will run.
rst_conf.l_rst_files = [RstFile(source='./rst_include/tests/test1_no_includes_template.rst',
                                target='./rst_include/tests/test1_no_includes_result.rst',
                                # default = utf-8-sig because it can read utf-8 and utf-8-sig
                                source_encoding='utf-8-sig',
                                # default = utf-8
                                target_encoding='utf-8'
                                ),
                        RstFile(source='./rst_include/tests/test2_include_samedir_template.rst',
                                target='./rst_include/tests/test2_include_samedir_result.rst'),
                        RstFile(source='./rst_include/tests/test3_include_subdir_template.rst',
                                target='./rst_include/tests/test3_include_subdir_result.rst'),
                        RstFile(source='./rst_include/tests/test4_include_nocode_template.rst',
                                target='./rst_include/tests/test4_include_nocode_result.rst')]

Additional You can easily replace (also multiline) text strings :

# replace text strings easily
# examples :

$> rst_include replace -s ./source.rst -t ./target.rst "{template_string}" "new content"

# multiline example
# note ${IFS} is the standard bash seperator
$> rst_include replace --inplace -s ./source.txt "line1${IFS}line2" "line1${IFS}something_between${IFS}line2"

piping under Linux:

# piping examples
$> rst_include include -s ./source.rst | rst_include replace -t ./target.rst "{template_string}" "new content"
# same result
$> cat ./source.rst | rst_include include | rst_include replace "{template_string}" "new content" > ./target.rst

# multiline example
$> cat ./text.txt | rst_include replace "line1${IFS}line2" "line1${IFS}something_between${IFS}line2" > ./text.txt

Example Build Script Python

import argparse
import errno
import logging
import os
import sys
from rst_include import *
from rst_include.libs import lib_log
import subprocess


# CONSTANTS & PROJECT SPECIFIC FUNCTIONS
codeclimate_link_hash = "ff3f414903627e5cfc35"


def project_specific(repository_slug, repository, repository_dashed):
    # PROJECT SPECIFIC
    logger = logging.getLogger('project_specific')
    logger.info('create help documentation files {dir}'.format(dir=os.path.abspath(os.path.curdir)))
    subprocess.run('{sys_executable} ./rst_include/rst_include.py -h > ./docs/rst_include_help_output.txt'.format(sys_executable=sys.executable), shell=True, check=True)
    subprocess.run('{sys_executable} ./rst_include/rst_include.py include -h > ./docs/rst_include_help_include_output.txt'.format(sys_executable=sys.executable), shell=True, check=True)
    subprocess.run('{sys_executable} ./rst_include/rst_include.py replace -h > ./docs/rst_include_help_replace_output.txt'.format(sys_executable=sys.executable), shell=True, check=True)


def parse_args(cmd_args=sys.argv[1:]):
    # type: ([]) -> []
    parser = argparse.ArgumentParser(
        description='Create Readme.rst',
        epilog='check the documentation on github',
        add_help=True)

    parser.add_argument('travis_repo_slug', metavar='TRAVIS_REPO_SLUG in the form "<github_account>/<repository>"')
    args = parser.parse_args(cmd_args)
    return args, parser


def main(args):
    logger = logging.getLogger('build_docs')
    logger.info('create the README.rst')
    travis_repo_slug = args.travis_repo_slug
    repository = travis_repo_slug.split('/')[1]
    repository_dashed = repository.replace('_', '-')

    project_specific(travis_repo_slug, repository, repository_dashed)

    """
    paths absolute, or relative to the location of the config file
    the notation for relative files is like on windows or linux - not like in python.
    so You might use ../../some/directory/some_document.rst to go two levels back.
    avoid absolute paths since You never know where the program will run.
    """

    logger.info('include the include blocks')
    rst_inc(source='./docs/README_template.rst',
            target='./README.rst')

    # please note that the replace syntax is not shown correctly in the README.rst,
    # because it gets replaced itself by the build_docs.py
    # we could overcome this by first replacing, and afterwards including -
    # check out the build_docs.py for the correct syntax !
    logger.info('replace repository related strings')
    rst_str_replace(source='./README.rst',
                    target='',
                    old='bitranox/rst_include',
                    new=travis_repo_slug,
                    inplace=True)
    rst_str_replace(source='./README.rst',
                    target='',
                    old='rst_include',
                    new=repository,
                    inplace=True)
    rst_str_replace(source='./README.rst',
                    target='',
                    old='rst-include',
                    new=repository_dashed,
                    inplace=True)

    rst_str_replace(source='./README.rst',
                    target='',
                    old='ff3f414903627e5cfc35',
                    new=codeclimate_link_hash,
                    inplace=True)

    logger.info('done')
    sys.exit(0)


if __name__ == '__main__':
    lib_log.setup_logger()
    main_logger = logging.getLogger('main')
    try:
        _args, _parser = parse_args()

        main(_args)
    except FileNotFoundError:
        # see https://www.thegeekstuff.com/2010/10/linux-error-codes for error codes
        sys.exit(errno.ENOENT)      # No such file or directory
    except FileExistsError:
        sys.exit(errno.EEXIST)      # File exists
    except TypeError:
        sys.exit(errno.EINVAL)      # Invalid Argument
    except ValueError:
        sys.exit(errno.EINVAL)      # Invalid Argument

Example Build Script DOS Batch

REM
REM rst_include needs to be installed and python paths set correctly
@echo off
cls

REM # You might also use Environment Variable here, or as commandline parameter
REM # this is just an example, I use actually the build_readme.py python file myself
REM # I do not recommend cmd files anymore - why it it is so much easier under python ...
REM # I am sure there is a more elegant was to do it on batch files, this is only an example

SET repository_slug="bitranox/rst_include"
SET repository="rst_include"
SET codeclimate_link_hash="ff3f414903627e5cfc35"

REM # get dashed repository name for pypi links
echo %repository% | rst_include replace "_" "-" > temp.txt
set /p repository_dashed= < temp.txt
del temp.txt


REM paths absolute, or relative to the location of the config file
REM the notation for relative files is like on windows or linux - not like in python.
REM so You might use ../../some/directory/some_document.rst to go two levels back.
REM avoid absolute paths since You never know where the program will run.

echo 'create the sample help outputs'
rst_include -h > ./docs/rst_include_help_output.txt
rst_include include -h > ./docs/rst_include_help_include_output.txt
rst_include replace -h > ./docs/rst_include_help_replace_output.txt

echo "import the include blocks"
rst_include include -s ./docs/README_template.rst -t ./README.rst

REM please note that the replace syntax is not shown correctly in the README.rst,
REM because it gets replaced itself by the build_docs.py
REM we could overcome this by first replacing, and afterwards including -
REM check out the build_docs.cmd for the correct syntax !

echo "replace repository_slug"
rst_include --inplace replace -s ./docs/README_template.rst bitranox/rst_include %repository_slug%
echo "replace repository"
rst_include --inplace replace -s ./docs/README_template.rst rst_include %repository%
echo "replace repository_dashed"
rst_include --inplace replace -s ./docs/README_template.rst rst-include %repository_dashed%
echo "replace codeclimate_link_hash"
rst_include --inplace replace -s ./docs/README_template.rst ff3f414903627e5cfc35 %codeclimate_link_hash%

echo 'finished'

Example Build Script Shellscript

#!/bin/bash

### CONSTANTS
codeclimate_link_hash="ff3f414903627e5cfc35"
# TRAVIS_TAG

function include_dependencies {
    local my_dir="$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P )"  # this gives the full path, even for sourced scripts
    chmod +x "${my_dir}"/lib_bash/*.sh
    source "${my_dir}/lib_bash/lib_color.sh"
}

include_dependencies  # we need to do that via a function to have local scope of my_dir

function check_repository_name {
    if [[ -z ${TRAVIS_REPO_SLUG} ]]
        then
            clr_bold clr_red "ERROR no travis repository name set - exiting"
            exit 1
        fi
}

clr_bold clr_green "Build README.rst for repository: ${TRAVIS_REPO_SLUG}"

check_repository_name

repository="${TRAVIS_REPO_SLUG#*/}"                                 # "username/repository_name" --> "repository_name"
repository_dashed="$( echo -e "$repository" | tr  '_' '-'  )"       # "repository_name --> repository-name"

clr_green "create the sample help outputs"
rst_include -h > ./docs/rst_include_help_output.txt
rst_include include -h > ./docs/rst_include_help_include_output.txt
rst_include replace -h > ./docs/rst_include_help_replace_output.txt

clr_green "import the include blocks"
rst_include include -s ./docs/README_template.rst -t ./docs/README_template_included.rst

clr_green "replace repository strings"

# please note that the replace syntax is not shown correctly in the README.rst,
# because it gets replaced itself by the build_docs.py
# we could overcome this by first replacing, and afterwards including -
# check out the build_docs.sh for the correct syntax !

# example for piping
cat ./docs/README_template_included.rst \
    | rst_include replace "bitranox/rst_include" "${TRAVIS_REPO_SLUG}" \
    | rst_include replace "rst_include" "$rst_include" \
    | rst_include replace "rst-include" "$rst-include" \
    | rst_include replace "ff3f414903627e5cfc35" "$ff3f414903627e5cfc35" \
     > ./README.rst

clr_green "cleanup"
rm ./docs/README_template_included.rst

clr_green "done"
clr_green "******************************************************************************************************************"
clr_bold clr_green "FINISHED building README.rst"
clr_green "******************************************************************************************************************"

RST Includes Examples

simple code include

# simple text include, empty line after
.. include:: ./include1.py
    :code: python
    :number-lines: 10
    :start-line: 6
    :end-line: 23
    :start-after: # start marker
    :end-before: # end-marker
    :encoding: utf-8

text or RST file include

# simple text include, without code setting - it is imported as normal textfile, as it is.
# You might also include other rst files
.. include:: include3.py
    :start-line: 0       # working, also end-line, etc ... all others suppressed.
    :number-lines:       # not working without :code: setting

include jupyter notebooks

jupyter notebooks can be first converted to rst via nbconvert, see : https://nbconvert.readthedocs.io/en/latest/usage.html#convert-rst

pandoc is a requirement for nbconvert, see : https://pandoc.org/

# convert the attached test.ipynb to test.rst
$ jupyter nbconvert --to rst test.ipynb

unfortunately the pictures are not shown and needed to be extracted - a first hint might be : https://gist.github.com/sglyon/5687b8455a0107afc6f4c60b5f313670

I would prefer to exctract the pictures after the conversion to RST, and make it a module in rst_include. Filenames can be a hash of the picture data, in order to avoid web caching issues.


RST Include Parameters

taken from : http://docutils.sourceforge.net/docs/ref/rst/directives.html

Standard data files intended for inclusion in reStructuredText documents are distributed with the Docutils source code, located in the “docutils” package in the docutils/parsers/rst/include directory. To access these files, use the special syntax for standard “include” data files, angle brackets around the file name:

.. include:: <isonum.txt>    # not supported now

The current set of standard “include” data files consists of sets of substitution definitions. See reStructuredText Standard Definition Files for details.

The following options are recognized:

# Only the content starting from this line will be included.
# (As usual in Python, the first line has index 0 and negative values count from the end.)
# Combining start/end-line and start-after/end-before is possible.
# The text markers will be searched in the specified lines (further limiting the included content).
start-line : integer
# Only the content up to (but excluding) this line will be included.
# Combining start/end-line and start-after/end-before is possible.
# The text markers will be searched in the specified lines (further limiting the included content).
end-line : integer
# Only the content after the first occurrence of the specified text will be included.
# Combining start/end-line and start-after/end-before is possible.
# The text markers will be searched in the specified lines (further limiting the included content).
start-after : text to find in the external data file
# Only the content before the first occurrence of the specified text (but after any after text) will be included.
# Combining start/end-line and start-after/end-before is possible.
# The text markers will be searched in the specified lines (further limiting the included content).
end-before : text to find in the external data file
# The entire included text is inserted into the document as a single literal block.
literal : flag (empty)
# The argument and the content of the included file are passed to the code directive (useful for program listings).
# (New in Docutils 0.9)
code : formal language (optional)
# Precede every code line with a line number. The optional argument is the number of the first line (default 1).
# Works only with code or literal. (New in Docutils 0.9)
number-lines : [start line number]
# The text encoding of the external data file. Defaults to the document's input_encoding.
encoding : name of text encoding
# Number of spaces for hard tab expansion. A negative value prevents expansion of hard tabs.
# Defaults to the tab_width configuration setting.
tab-width : integer
With code or literal the common options :class: and :name: are recognized as well.
all other option in the format :<option>: are just passed through the codeblock

Requirements

following modules will be automatically installed :

pytest  # see : https://github.com/pytest-dev/pytest
typing  # see : https://pypi.org/project/typing/

Acknowledgements

  • special thanks to “uncle bob” Robert C. Martin, especially for his books on “clean code” and “clean architecture”

Contribute

I would love for you to fork and send me pull request for this project. - please Contribute


License

This software is licensed under the MIT license


Changelog

1.0.8

  • strict mypy typechecking
  • drop python 2.7 / 3.5 support
  • implement –inplace option
  • implement –quiet option
  • implement multiline string replacement
  • extend documentation

1.0.2

2019-04-28: fix import errors

1.0.1

2019-04-28: add empty line at the end of the assembled documentation, to be able to add CHANGES.rst with setup.py

1.0.0

2019-04-19: Initial public release, PyPi Release

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for rst-include, version 1.0.8
Filename, size File type Python version Upload date Hashes
Filename, size rst_include-1.0.8-py3-none-any.whl (33.5 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size rst_include-1.0.8.tar.gz (113.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page