moban 0.7.3

General purpose static text generator

Author:

C.W. and its contributors (See contributors.rst)

Issues:

http://github.com/moremoban/moban/issues

License:

MIT

Announcement

For existing moban users, python 2 support has been dropped. Please stay with versions lower than 0.7.0 if you are still using python 2.

From 2020 onwards, minimum requirement is Python 3.6

Introduction

moban started with bringing the high performance template engine (JINJA2) for web into static text generation.

moban can use other python template engine: mako, handlebars, velocity, haml, slim and tornado, can read other data format: json and yaml, and can access both template file and configuration file in any location: zip, git, pypi package, s3, etc.

It has been used in pyexcel and coala project to keep documentation consistent across the documentations of individual libraries in the same organisation.

And here is a list of other usages:

1. Django Mobans, templates for django, docker etc.

2. Math Sheets, generate custom math sheets in pdf

Vision

Any template, any data in any location

Support

If you like moban, please support me on, patreon or bounty source to maintain the project and develop it further.

With your financial support, I will be able to invest a little bit more time in coding, documentation and writing interesting extensions.

Credit

jinja2-fsloader is the key component to enable PyFilesystem2 support in moban v0.6x. Please show your stars there too!

Installation

You can install it via pip:

$ pip install moban

or clone it and install it:

$ git clone http://github.com/moremoban/moban.git
$ cd moban
$ python setup.py install

Quick start

$ export HELLO="world"
$ moban "{{HELLO}}"
Templating {{HELLO}}... to moban.output
Templated 1 file.
$ cat moban.output
world

Or

$ export HELLO="world"
$ echo "{{HELLO}}" | moban

Or simply

$ HELLO="world" moban "{{HELLO}}"

A bit formal example:

$ moban -c data.yml -t my.template
$ cat moban.output

Given data.yml as:

hello: world

and my.template as:

{{hello}}

moban.output will contain:

world

Please note that data.yml will take precedence over environment variables.

Moban in live action:

All use cases are documented here

Templates and configuration files over HTTP(S)

httpfs should be installed first.

$ moban -t 'https://raw.githubusercontent.com/moremoban/pypi-mobans/dev/templates/_version.py.jj2'\
  -c 'https://raw.githubusercontent.com/moremoban/pypi-mobans/dev/config/data.yml'\
  -o _version.py

In an edge case, if github repo’s public url is given, this github repo shall not have sub repos. This library will fail to translate sub-repo as url. No magic.

Templates and configuration files in a git repo

gitfs2 is optional since v0.7.0 but was installed by default since v0.6.1

You can do the following with moban:

$ moban -t 'git://github.com/moremoban/pypi-mobans.git!/templates/_version.py.jj2' \
        -c 'git://github.com/moremoban/pypi-mobans.git!/config/data.yml' \
        -o _version.py
Info: Found repo in /Users/jaska/Library/Caches/gitfs2/repos/pypi-mobans
Templating git://github.com/moremoban/pypi-mobans.git!/templates/_version.py.jj2 to _version.py
Templated 1 file.
$ cat _version.py
__version__ = "0.1.1rc3"
__author__ = "C.W."

Templates and configuration files in a python package

pypifs is optional since v0.7.0 but was installed by default since v0.6.1

You can do the following with moban:

$ moban -t 'pypi://pypi-mobans-pkg/resources/templates/_version.py.jj2' \
        -c 'pypi://pypi-mobans-pkg/resources/config/data.yml' \
        -o _version.py
Collecting pypi-mobans-pkg
....
Installing collected packages: pypi-mobans-pkg
Successfully installed pypi-mobans-pkg-0.0.7
Templating pypi://pypi-mobans-pkg/resources/templates/_version.py.jj2 to _version.py
Templated 1 file.
$ cat _version.py
__version__ = "0.1.1rc3"
__author__ = "C.W."

Work with S3 and other cloud based file systems

Please install fs-s3fs:

$ pip install fs-s3fs

Then you can access your files in s3 bucket:

$ moban -c s3://${client_id}:${client_secrect}@moremoban/s3data.yml \
        -o 'zip://my.zip!/moban.output' {{hello}}
$ unzip my.zip
$ cat moban.output
world

Where the configuration sits in a s3 bucket, the output is a file in a zip. The content of s3data.yaml is:

hello: world

CLI documentation

usage: moban [-h] [-c CONFIGURATION] [-t TEMPLATE] [-o OUTPUT]
             [-td [TEMPLATE_DIR [TEMPLATE_DIR ...]]] [-cd CONFIGURATION_DIR]
             [-m MOBANFILE] [-g GROUP] [--template-type TEMPLATE_TYPE]
             [-d DEFINE [DEFINE ...]] [-e EXTENSION [EXTENSION ...]] [-f]
             [--exit-code] [-V] [-v]
             [template]

Static text generator using any template, any data and any location.

positional arguments:
  template              string templates

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIGURATION, --configuration CONFIGURATION
                        the data file
  -t TEMPLATE, --template TEMPLATE
                        the template file
  -o OUTPUT, --output OUTPUT
                        the output file

Advanced options:
  For better control

  -td [TEMPLATE_DIR [TEMPLATE_DIR ...]], --template_dir [TEMPLATE_DIR [TEMPLATE_DIR ...]]
                        add more directories for template file lookup
  -cd CONFIGURATION_DIR, --configuration_dir CONFIGURATION_DIR
                        the directory for configuration file lookup
  -m MOBANFILE, --mobanfile MOBANFILE
                        custom moban file
  -g GROUP, --group GROUP
                        a subset of targets
  --template-type TEMPLATE_TYPE
                        the template type, default is jinja2
  -d DEFINE [DEFINE ...], --define DEFINE [DEFINE ...]
                        to supply additional or override predefined variables,
                        format: VAR=VALUEs
  -e EXTENSION [EXTENSION ...], --extension EXTENSION [EXTENSION ...]
                        to to TEMPLATE_TYPE=EXTENSION_NAME
  -f                    force moban to template all files despite of
                        .moban.hashes

Developer options:
  For debugging and development

  --exit-code           tell moban to change exit code
  -V, --version         show program's version number and exit
  -v                    show verbose, try -v, -vv, -vvv

Exit codes

By default:

• 0 : no error

• 1 : error occured

With –exit-code:

• 0 : no changes

• 1 : has changes

• 2 : error occured

Contributors

In alphabetical order:

• Ayan Banerjee

• Charlie Liu

• John Vandenberg

• Joshua Chung

• PRAJWAL M

• SerekKiri

Change log

0.7.3 - 2.5.2020

Added

1. Added continuous check in travis for setup.py descriptions. No impact to moban user.

0.7.2 - 1.5.2020

Added

1. Support for templates and configuration files over HTTP(S) protocol with httpfs! Yepee!

0.7.1 - 25.04.2020

Fixed

1. #365: regression was introduced by v0.6.5. If you uses mobanfile as data configuration file, you are very likely to have this show stopper. Please upgrade to this version.

0.7.0 - 18.01.2020

Removed

1. #360: make gitfs2 and pypifs optional.

2. #303: python 2.7 support is dropped.

Updated

1. #360: show friendlier error when unknown protocol exception was raised.

0.6.8 - 7.12.2019

Updated

1. since version 0.5.0, when rendering a single file or string, moban would report ‘Templated 1 of 0 files’, which should have been ‘Templated 1 file.’

Removed

1. python 3.4 support is gone because colorama requires Python ‘>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*’

0.6.7 - 1.12.2019

Updated

1. no verbose for error, -v for warning, -vv for warning+info, -vvv for warning+info+debug

2. #351, show template plugin name, i.e. ‘copying’ for copy instead of ‘templating’

Removed

1. Message: ‘Warning: Attempting to use environment vars as data…’ became warning log

2. Message: ‘Warning: Both data.yml and /…/.moban.cd/data.yml does not exist’ became warning log

3. with -v, you would see them in such a situation

0.6.6 - 10.11.2019

Added

1. support moban file inheritance. one base moban file and child repos can inherit and override

0.6.5 - 13.10.2019

Added

1. #335: support intermediate targets in moban file

0.6.4 - 4.10.2019

Updated

1. Command options have been grouped. –template_type became –template-type

2. Increment gitfs2 to version 0.0.2. gitfs#4

0.6.3 - 25.09.2019

Added

1. #260: jinja-cli parity: support command line pipe stream.

0.6.2 - 15.09.2019

Added

1. #322: Implicit targets with template extensions default to copy

2. #257: ‘-e’ to load extensions for template engines, i.e. jinja2

3. #333: command line template fails with version 0.6.1

0.6.1 - 10.09.2019

Fixed

1. #328: update backward compatibility

0.6.0 - 10.09.2019

Added

1. #205: support pyFilesystem2

2. #185: -v will enable moban application logging for development. And -V is for version.

3. #325: -vv show debug trace

4. #126: Allow mobanfile to include data from arbitrary config files

5. #256: jinja2-cli parity: ‘-d hello=world’ to define custom variable on cli

Updated

1. #275: fix moban 0.4.5 test failures on openSUSE Tumbleweed

0.5.0 - 14.07.2019

Updated

1. #277: Restored dependency git-url-parse, replacing incompatible giturlparse which was used during moban 0.4.x

2. #281: Fixed unicode support on Python 2.7

3. #274: Updated ruamel.yaml dependency pins to restore support for Python 3.4, and prevent installation of versions that can not be installed on Python 3.7

4. #285: Fixed CI testing of minimum requirements

5. #271: Fixed repository caching bug preventing branch switching

6. #292: Reformatted YAML files according to yamllint rules

7. #291: Fixed filename typos in README

8. #280: Added CI to ensure repository is in sync with upstream

9. #280: sync setup.py from pypi-mobans

0.4.5 - 07.07.2019

Updated

1. #271: support git branch change in later run.

0.4.4 - 26.05.2019

Updated

1. #265: Use simple read binary to read instead of encoding

0.4.3 - 16.03.2019

Removed

1. #253: symbolic link in regression pack causes python setup.py to do recursive include

Added

1. #209: Alert moban user when git is not available and is used.

Updated

1. #261: since moban group template files per template type, this fill use first come first register to order moban group

0.4.2 - 08.03.2019

Added

1. #234: Define template parameters on the fly inside targets section

2. #62: select a group target to run

Updated

1. #180: No longer two statistics will be shown in v0.4.x. legacy copy targets are injected into a normal targets. cli target is made a clear priority.

2. #244: version 0.4.2 is first version which would work perfectly on windows since 17 Nov 2018. Note that: file permissions are not used on windows. Why the date? because samefile is not avaiable on windows, causing unit tests to fail hence it lead to my conclusion that moban version between 17 Nov 2018 and March 2019 wont work well on Windows.

0.4.1 - 28.02.2019

Added

1. #235: user defined template types so that custom file extensions, template configurations can be controlled by moban user

2. #232: the package dependencies have been fine tuning to lower versions, most of them are dated back to 2017.

0.4.0 - 20.02.2019

Added

1. #165: Copy as plugins

Updated

1. #219: git clone depth set to 2

2. #186: lowest dependecy on ruamel.yaml is 0.15.5, Jun 2017

0.3.10 - 03.02.2019

Added

1. #174: Store git cache in XDG_CACHE_DIR

2. #107: Add -v to show current moban version

3. #164: support additional data formats

Updated

1. #178: UnboundLocalError: local variable ‘target’ referenced before assignment

2. #169: uses GitPython instead of barebone git commands

0.3.9 - 18-1-2019

Updated

1. #90: allow adding extra jinja2 extensions. jinja2.ext.do, jinja2.ext.loopcontrols are included by default. what’s more, any other template enigne are eligible for extension additions.

2. #158: Empty file base_engine.py is finally removed

0.3.8 - 12-1-2019

Updated

1. #141: disable file permissions copy feature and not to check file permission changes on windows.

2. #154: introduce first ever positional argument for string base template.

3. #157: the exit code behavior changed. for backward compactibility please use –exit-code. Otherwise, moban will not tell if there is any changes.

0.3.7 - 6-1-2019

Updated

1. #146: added a low-setup usage mode via environment variables to moban

2. #148: include test related files in the package for package validation when distributing via linux system, i.e. OpenSuse

0.3.6 - 30-12-2018

Updated

1. #143: moban shall report permission error and continue the rest of the copying task.

2. #122: Since 0.3.6, moban is tested on windows and macos too, using azure build pipelines. It is already tested extensively on travis-ci on linux os.

0.3.5 - 10-12-2018

Updated

1. #37: moban will report line number where the value is empty and the name of mobanfile. Switch from pyyaml to ruamel.yaml.

0.3.4.1 - 28-11-2018

Updated

1. #137: missing contributors.rst file

0.3.4 - 18-11-2018

Added

1. global variables to store the target and template file names in the jinja2 engine

2. moban-handlebars is tested to work well with this version and above

Updated

1. Template engine interface has been clarified and documented

0.3.3 - 05-11-2018

Added

1. alternative and expanded syntax for requires, so as to accomendate github submodule recursive

0.3.2 - 04-11-2018

Added

1. configuration dirs may be located by requires, i.e. configuration files may be in a python package or git repository.

0.3.1 - 02-11-2018

Added

1. #97: requires will clone a repo if given. Note: only github, gitlab, bitbucket for now

0.3.0 - 27-18-2018

Added

1. #89: Install pypi-hosted mobans through requires syntax

Updated

1. #96: Fix for FileNotFoundError for plugins

2. various documentation updates

Removed

1. #88: removed python 2.6 support

2. removed python 3.3 support

0.2.4 - 14-07-2018

Added

1. #32: option 1 copy a directory without its subdirectories.

2. #30: command line template option is ignore when a moban file is present

0.2.3 - 10-07-2018

Added

1. #76: running moban as a module from python command

2. #32: copy a directory recusively

3. #33: template all files in a directory

0.2.2 - 16-06-2018

Added

1. #31: create directory if missing during copying

Updated

1. #28: if a template has been copied once before, it is skipped in the next moban call

0.2.1 - 13-06-2018

Updated

1. templates using the same template engine will be templated as a group

2. update lml dependency to 0.0.3

0.2.0 - 11-06-2018

Added

1. #18: file exists test

2. #23: custom jinja plugins

3. #26: repr filter

4. #47: allow the expansion of template engine

5. #58: allow template type per template

Updated

1. #34: fix plural message if single file is processed

0.1.4 - 29-May-2018

Updated

1. #21: targets become optional

2. #19: transfer symlink’s target file’s file permission under unix/linux systems

3. #16: introduce copy key word in mobanfile

0.1.3 - 12-Mar-2018

Updated

1. handle unicode on python 2

0.1.2 - 10-Jan-2018

Added

1. #13: strip off new lines in the templated file

0.1.1 - 08-Jan-2018

Added

1. the ability to present a long text as multi-line paragraph with a custom upper limit

2. speical filter expand github references: pull request and issues

3. #15: fix templating syntax to enable python 2.6

0.1.0 - 19-Dec-2017

Added

1. #14, provide shell exit code

0.0.9 - 24-Nov-2017

Added

1. #11, recognize .moban.yaml as well as .moban.yml.

2. #9, preserve file permissions of the source template.

3. -m option is added to allow you to specify a custom moban file. kinda related to issue 11.

Updated

1. use explicit version name: moban_file_spec_version so that version can be used by users. #10 Please note: moban_file_spec_version is reserved for future file spec upgrade. For now, all files are assumed to be ‘1.0’. When there comes a new version i.e. 2.0, new moban file based on 2.0 will have to include ‘moban_file_spec_version: 2.0’

0.0.8 - 18-Nov-2017

Added

1. #8, verify the existence of custom template and configuration directories. default .moban.td, .moban.cd are ignored if they do not exist.

Updated

1. Colorize error messages and processing messages. crayons become a dependency.

0.0.7 - 19-Jul-2017

Added

1. Bring the visibility of environment variable into jinja2 templating process: #7

0.0.6 - 16-Jun-2017

Added

1. added ‘-f’ flag to force moban to template all files despite of .moban.hashes

Updated

1. moban will not template target file in the situation where the changes occured in target file than in the source: the template file + the data configuration after moban has been applied. This new release will remove the change during mobanization process.

0.0.5 - 17-Mar-2017

Added

1. Create a default hash store when processing a moban file. It will save unnecessary file write to the disc if the rendered content is not changed.

2. Added summary reports

0.0.4 - 11-May-2016

Updated

1. Bug fix #5, should detect duplicated targets in .moban.yml file.

0.0.3 - 09-May-2016

Updated

1. Bug fix #4, keep trailing new lines

0.0.2 - 27-Apr-2016

Updated

1. Bug fix #1, failed to save utf-8 characters

0.0.1 - 23-Mar-2016

Added

1. Initial release