e2j2 0.6.1

e2j2 is a commandline utility to render text/configuration files from jinja2 templatesfrom shell environment variables

e2j2 [ ]

Table of contents:

• e2j2 [Build Status Coverage Status]

  • What is e2j2?

  • Command line switches and config file

  • Jinja2 filter support

  • Example

  • Environment variable examples

    • Plain environment variable

    • Tag file

    • Tag json

    • Tag jsonfile

    • Tag base64

    • Tag consul

    • Tag list

    • Two pass rendering

    • Tag vault

    • Tag dns

• Changelog

  • 0.6.1 (2020-08-16)

  • Changed

  • 0.6.0 (2020-08-11)

  • Add

  • 0.5.2 (2020-07-30)

  • Fixed

  • Add

  • Note

  • 0.5.1 (2020-07-28)

  • Known issues

  • Fixed

  • 0.5.0 (2020-07-21)

  • Added

  • Breaking change:

  • 0.4.10 (2020-01-14)

  • Changed

  • 0.4.9 (2019-12-16)

  • Changed

  • 0.4.8 (2019-12-04)

  • Added

  • Fixed

  • 0.4.7 (2019-11-25)

  • Fixed

  • 0.4.6 (2019-11-24)

  • Added

  • Changed

  • 0.4.5 (2019-11-19)

  • Changed

  • 0.4.4 (2019-11-18)

  • Fixed

  • 0.4.3 (2019-11-15)

  • Added

  • Fixed

  • 0.4.2 (2019-11-14)

  • Fixed

  • 0.4.1 (2019-11-13)

  • Fixed

  • 0.4.0 (2019-11-12)

  • Added

  • Fixed

  • 0.3.0 (2019-10-25)

  • Changed

  • Breaking change:

  • 0.2.1 (2019-10-23)

  • Added

  • 0.2.0 (2019-07-23)

  • Added

  • Changed

  • 0.1.21 (2019-07-09)

  • Fixed

  • Added

  • 0.1.20 (2019-06-12)

  • Fixed

  • 0.1.19 (2019-06-10)

  • Added

  • Fixed

  • Changed

  • 0.1.18 (2019-05-17)

  • Fixed

  • 0.1.17 (2019-05-17)

  • Fixed

  • 0.1.16 (2019-05-16)

  • Fixed

  • 0.1.15 (2019-05-16)

  • Changed

    • Breaking change

  • 0.1.14 (2019-04-30)

  • 0.1.13 (2019-03-29)

  • 0.1.12 (2019-03-14)

  • 0.1.11 (2019-01-30)

  • 0.1.10 (2018-09-24)

  • 0.1.9 (2018-09-21)

  • 0.1.8 (2018-09-21)

  • 0.1.7 (2018-09-20)

  • 0.1.6 (2018-05-17)

  • 0.1.5 (2018-05-11)

  • 0.1.4 (2017-08-21)

    • Added

    • Fixed

  • 0.1.3 (2017-07-11)

    • Breaking change

    • Changed

    • Fixed

  • 0.1.2 (2017-05-17)

  • 0.1.1 (2017-05-17)

    • Fixed

    • Removed

    • Changed

  • 0.1.0 (2017-05-16)

    • Added

    • Changed

  • 0.0.2 (2017-05-16)

    • Added

    • Changed

  • 0.0.1 (2017-05-01)

What is e2j2?

e2j2 (environment to jinja2 variables) is a commandline tool which will render jinja2 templates to textfiles. all environment variables can be used in the jinja2 templates, within the environment variables you can use special tags which give you the option to insert json, json file paths, base64 hashes, consul kv keys.

e2j2 can be used within docker containers, or for other simple configuration templating tasks. Within a docker container you can simply add the j2 extention to a configuration file, or to multiple files within one or more folder structures and then run e2j2 as part of your docker entrypoint script.

Command line switches and config file

Most flags can be set on either the command line or in a json formatted config file. The following flags / configuration keys are supported:

Switch	Type	Config key	Type	Description
-h, –help				Show help text and exit
-v, –version				Show version number
-e, –ext, –extension	string	extension	string	Jinja2 file extention (default: .j2)
-f, –filelist	comma separated list	filelist	array	List of jinja2 templates
-s, –searchlist	comma separated list	searchlist	array	List of directories to search for templates (default: current directory)
-N, –noop				skip writing template to disk
-r, –recursive		recursive	boolean	Traverse recursively through the search list
–no-color, –nocolor		no_color	boolean	Disable ANSI color
-2, –twopass		twopass	boolean	Enable two pass rendering
–block_start	string	block_start	string	Block marker start (default: ‘{%’}
–block_end	string	block_end	string	Block marker start (default: ‘%}’}
–variable_start	string	block_start	string	Variable marker start (default: ‘{%’}
–variable_end	string	variable_end	string	Variable marker end (default: ‘%}’}
–comment_start	string	comment_start	string	Comment marker start (default: ‘{#’}
–comment_end	string	comment_end	string	Comment marker start (default: ‘#}’}
-w, –env_whitelist	comma separated list	env_whitelist	array	List of envars to include
-b, –env_blacklist	comma separated list	env_blacklist	array	List of envars to exclude
-P, –copy_file_permissions		copy_file_permissions	boolean	Copy file permissions and ownership from template to rendered file
-S, –stacktrace		stacktrace	boolean	Include stacktrace in error file
-c, –config	string			Config file path
–watchlist	comma separated list	watchlist	array	Watch listed envvars for changes and render template(s) on change
-R, –run	string	run	array	Run command after rendering templates
–splay	int	splay	integer	Random delay between 1 and X seconds between watchlist polls
–initial-run		render templates before starting watch

Jinja2 filter support

By default only the jinja2 builtin filters are supported this can be extended by installing the jinja2-ansible-filters module.

Example

lets assume we want to render the following server block in nginx, if we place the server configuration in a nginx include directory for example /etc/nginx/conf.d

server {
  server_name {{ NGINX.server_name }};
  listen 80;
  listen [::]:80;
  error_page 500 502 503 504 /50x.html;

  location / {
    index {{ NGINX.index_page }};
    root {{ NGINX.web_root }};
  }

  location ~ \.php$ {
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    fastcgi_pass unix:{{ NGINX.fpm_socket }};
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    include {{ NGINX.fcgi_params }};
    root {{ NGINX.web_root }};
    try_files $uri =404;
  }
}

if you then set the NGINX environment variable, running e2j2 will render the jinja2 template and place it in the same folder:

~> export NGINX='json:
{
"server_name": "www.myweb.com",
"index_page": "index.php",
"web_root": "/usr/local/www/myweb",
"fcgi_params": "/usr/local/etc/nginx/myweb-fcgi-params",
"fpm_socket": "/var/run/php-fpm/myweb.socket"
}'
~> e2j2

In: .
    rendering: nginx_vhost_config.conf.j2=>done => writing: nginx_vhost_config.conf=>done

~> cat nginx_vhost_config.conf
server {
  server_name www.myweb.com;
  listen 80;
  listen [::]:80;
  error_page 500 502 503 504 /50x.html;

  location / {
    index index.php;
    root /usr/local/www/myweb;
  }

  location ~ \.php$ {
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    fastcgi_pass unix:/var/run/php-fpm/myweb.socket;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    include /usr/local/etc/nginx/myweb-fcgi-params;
    root /usr/local/www/roundcube;
    try_files $uri =404;
  }
}

Environment variable examples

Plain environment variable

Example:

setting:

MYENVVAR='plain environment variable'

will render envvar-example.j2 to:

This is a plain environment variable

Tag file

Example:

setting:

MYFILEVAR=file:./file-example.txt

will render file-example.j2 to:

This is a file example

Tag json

Example:

setting:

MYJSONVAR='json:{"key": "json-example"}'

will render json-example.j2 to:

This is a json-example

Tag jsonfile

Example:

setting:

MYJSONFILEVAR='jsonfile:jsonfile-example.json'

will render jsonfile-example.j2 to:

This is a jsonfile example with subkey

Tag base64

Example:

Setting:

export MYBASE64VAR='base64:YmFzZTY0IGV4YW1wbGU='

will render base64-example.j2 to:

This is a base64 example

Tag consul

Configuration:

You can configure the consul tag by setting the CONSUL_CONFIG environment variable. The following config items are supported:

Item	Explanation	Default
url	consul url	http://127.0.0.1:8500
scheme	consul url scheme http or https	scheme from url
host	consul host	hostname from url
port	consul http(s) port	port from url
token	consul token	none

Global config example:

read -d '' CONSUL_CONFIG << EOF
{
   "url": "https://consul.foobar.tld",
   "token": "abcdef01-0123-abcd-1234-0123456789ab"
}
EOF

The ACL token can be configured by either the above configuration or by setting the CONSUL_TOKEN variable.

As an alternative for the global configuration it is also possible to configure / adjust the global configuration for each consul tag, by simply include the configuration when using the consul tag.

CONSUL_TOKEN and config key token can either contain the actual token or point to a file containing the token, use the file: tag to point to a file.

Tag config examples:

export MYCONSULVAR='consul:config={"url": "https://consul2.foobar.tld", "token": "012345678-0123-abcd-1234-0123456789ab"}:consulvar"

export MYCONSULVAR='consul:config={"url": "https://consul2.foobar.tld", "token": "file:/path/to/token"}:consulvar"

Consul example:

Setting:

key: consulvar in consul to value: consul example

and

export MYCONSULVAR='consul:consulvar'

will render consul-example.j2 to:

This is a consul example

Tag list

Example:

Setting:

export MYLIST='list:"first","second","third","fourth"'

will render list-example.j2 to:

"first"
"second"
"third"
"fourth"

Two pass rendering

Starting from version 0.1.12 e2j2 supports embedding jinja2 macros in environment variables.

Example:

Setting the following two environment variables:

export WORDPRESS='json:{"database": {"name": "mydb", "user": "mydb_user", "password": "{{ DBSECRET }}", "host": "localhost"}}'
export DBSECRET='file:./twopass-secret'

will render (by running: e2j2 -f twopass-example.j2 -2) to:

// ** MySQL settings - You can get this info from your web host ** //
/** The name of the database for WordPress */
define( 'DB_NAME', 'mydb' );

/** MySQL database username */
define( 'DB_USER', 'mydb_user' );

/** MySQL database password */
define( 'DB_PASSWORD', 'Db$ecr3t' );

/** MySQL hostname */
define( 'DB_HOST', 'localhost' );

In version 0.5.0 support was introduced for nested tag variables, so the above listed example can be simplified. The DBSECRET variable is no longer needed, if the WORDPRESS variable is changed to:

export WORDPRESS='json:{"database": {"name": "mydb", "user": "mydb_user", "password": "file:./twopass-secret", "host": "localhost"}}'

If needed you can escape e2j2 tags by using the escape tag

Tag vault

Configuration:

You can configure the vault tag by setting the VAULT_CONFIG environment variable. The following config items are supported:

Item	Explanation	Default
url	vault url	http://127.0.0.1:8200
scheme	vault url scheme http or https	scheme from url
host	vault host	hostname from url
port	vault http(s) port	port from url
backend	vault secret backend	raw
token	vault token	none

the following backends are supported:

backend	Description
raw	use plain GET request to secret store API
kv1	key/value version 1
kv2	key/value version 2

Global config example:

read -d '' VAULT_CONFIG << EOF
 {
   "url": "https://vault.foobar.tld:8200",
   "token": "s.xxxxxxxxxxxxxxxxxxxxxxx",
   "backend: "kv2"
 }
EOF

The Authentication token can be configured by either the above configuration or by setting the VAULT_TOKEN variable.

As an alternative for the global configuration it is also possible to configure / adjust the global configuration for each vault tag, by simply include the configuration when using the vault tag.

VAULT_TOKEN and config key token can either contain the actual token or point to a file containing the token, use the file: tag to point to a file.

Tag config example:

export MYVAULTVAR='vault:config={"backend": "kv2", "token": "s.xxxxxxxxx"}:kv/my-secret"

export MYVAULTVAR='vault:config={"backend": "kv2", "token": "file:/path/to/token"}:kv/my-secret"

Vault example:

Setting:

vault kv put secret/my-secret secret=topsecret
export MYVAULTVAR='vault:secret/my-secret'

will render vault-kv1-example.j2 (by running: e2j2 -f vault-example.j2) to:

** topsecret **
This is a vault example

Tag dns

Configuration:

You can configure the dns tag by setting the DNS_CONFIG environment variable. The following config items are supported:

Item	Explanation	Default
nameservers	overwrite nameservers	use system resolvers
port	overwrite dns port	53
type	record type (A, AAAA or SRV)	A

the supported record types will return a dict with the following keys:

Type	Keys
A	address
AAAA	address
SRV	target, port, weight, priority

DNS example:

Assuming a consul node running on localhost with the default dns port 8600.

Setting the DNS_CONFIG variable:

read -d '' DNS_CONFIG << EOF
 {
   "nameservers": ['127.0.0.1'],
   "port": 8600,
   "type": "SRV"
 }
EOF

Setting:

export MYDNSVAR='dns:consul.service.consul'

will render dns-example.j2 (by running: e2j2 -f dns-example.j2) to:

My consul node:
node1.node.dc1.consul. listening on port 8300

Changelog

0.6.1 (2020-08-16)

Changed

• Nested tags is no longer enabled by default, use –nested-tags to enable this feature

0.6.0 (2020-08-11)

Add

• add marker sets there is support for the following marker sets

1. {%, %}, {{, }}, {#, #}

2. {%, %}, {=, =}, {#, #}

3. <%, %>, <=, =>, <#, #>

4. [%, %], [=, =], [#, #]

5. {%, %), (=, =), (#, #)

• start and end markers for the tag config block are now configurable (–config-start, –config-end)

0.5.2 (2020-07-30)

Fixed

• merge config in tag with existing config

Add

• add escape tag, this tag can be used for escaping e2j2 tags in strings

Note

• command line flags with underscores are now deprecated and will be removed in later version

0.5.1 (2020-07-28)

Known issues

• potential conflicts with twopass rendering if strings start with text similar to e2j2 tags

• flatten option for dict values ignores existing config values

Fixed

• only check for nested tags in strings

0.5.0 (2020-07-21)

Added

• Add support for rendering tags in json objects and lists in combination with twopass rendering:

  So now it is possible to render base64 encoded strings within a json tag within the same environment variable.

  example:

  with:

  export MY_NESTED_VAR=’json:{“ip_regex”: “base64:Xig/Oig/OjI1WzAtNV18MlswLTRdWzAtOV18WzAxXT9bMC05XVswLTldPylcLil7M30oPzoyNVswLTVdfDJbMC00XVswLTldfFswMV0/WzAtOV1bMC05XT8pJA==”}’

  the template

  {{ MY_NESTED_VAR }}

  will render to:

  {“ip_regex”: “^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$”}

• Add python 3.9 to the list of supported python versions

Breaking change:

python 2.7.x is no longer supported! python 3.4.x is no longer supported!

0.4.10 (2020-01-14)

Changed

consul and vault tokens can now point to a file (use the file: tag to point to a file)

0.4.9 (2019-12-16)

Changed

• follow symlinks when recursive flag is set

0.4.8 (2019-12-04)

Added

• support jinja2 filters provided by the jinja2-ansible-filters library if installed

Fixed

• fix handling consul keys containing newlines and double quote characters

0.4.7 (2019-11-25)

Fixed

• searchlist is required even if E2J2_SEARCHLIST is set

0.4.6 (2019-11-24)

Added

• add flag to run command after initial template run

• add config option to json, jsonfile, consul, dns, and vault tag to flatten the returned json object

Changed

• disable run on initial template run

0.4.5 (2019-11-19)

Changed

• add test run to watch, only apply changes if test is successful

0.4.4 (2019-11-18)

Fixed

• config var missing in get_var call

• fix message repeat filter in stdout function

• support ipv6 address in nameserver field of dns tag

0.4.3 (2019-11-15)

Added

• add –splay flag, to configure delay of watchlist polls

• alias flags for no color, white/black list and jinja2 markers

Fixed

• stacktrace on ctrl-c

• warning message in yellow with –no-color flag

• multiple script runs (one for each template), should be just one

0.4.2 (2019-11-14)

Fixed

• fixed incompatibility issue with python 2.7 and watchlist option

0.4.1 (2019-11-13)

Fixed

• set default value for filelist flag to empty list

0.4.0 (2019-11-12)

Added

• add dns tag (currently includes support of record types A, AAAA, and SRV)

• config file support

• watch for changes on consul, vault and dns tags and render on change (Experimental)

• add run flag, this flag can be used to execute shell commands after rendering templates (Experimental)

Fixed

• show only line numbers on a limited set up exceptions

0.3.0 (2019-10-25)

Changed

• show line numbers on rendering errors

• envvar tag errors are now disabled as warnings as they are non fatal

Breaking change:

• envvar tag errors will result in undefined variables previously they contained an error message

0.2.1 (2019-10-23)

Added

• add –stacktrace flag to include a python stack trace in the error file (thanks: Sylvia van Os/TheLastProject)

Changed - show jinja2 render error in console output (thanks: Marco Verleun)

Fixed - Fix for pip install issues - Pass exit code to calling shell

0.2.0 (2019-07-23)

Added

• vault secrets support

• per tag config (for the consul and vault tags)

• add support for token variables (for the consul and vault tags)

Changed

• Add json schema validation for consul and vault config

0.1.21 (2019-07-09)

Fixed

• Fix issue with included templates in other directories

Added

• Add option to “copy” file ownership and file permissions

0.1.20 (2019-06-12)

Fixed

• fix “only render one file” issue

0.1.19 (2019-06-10)

Added

• add option to whitelist / blacklist environment variables

Fixed

• set proper exit codes (0=success / 1=failure)

Changed

• add unittests

• add functional tests

0.1.18 (2019-05-17)

Fixed

• Fix bug in consul parser

0.1.17 (2019-05-17)

Fixed

• Fix bug with include statements in jinja2 template

0.1.16 (2019-05-16)

Fixed

• Fix bug with default jinja marker

0.1.15 (2019-05-16)

Changed

• add options to overwrite the default jinja2 markers (block start/end, variable start/end and comment start/end)

Breaking change

• removed “hack” for twopass rendering, you can use the new marker for handling conflicting characters in your template

0.1.14 (2019-04-30)

Fixed

• preserve newlines at end of file

• CVE-2019-10906, module now requires jinja2>=2.10.1

0.1.13 (2019-03-29)

Fixed

• Fix for ignored raw tag with two-pass rendering

0.1.12 (2019-03-14)

Added

• add support for two pass rendering

0.1.11 (2019-01-30)

Changed

• Changed behavior of nested keys with the consul tag.

example: in consul we have an key key1/key2/123 envvar KEY=consul:key1/key2 => {{ KEY }} will now render to 123 and no longer to {“key2”: 123}

0.1.10 (2018-09-24)

Added

• add file tag, this tag will place the content of the file into the variable

0.1.9 (2018-09-21)

Fixed

• fix import issue on python 2.7

0.1.8 (2018-09-21)

Changed

• handle hashrocket characters in json tag (as produced by hiera)

0.1.7 (2018-09-20)

Added

• Add list tag which will parse a comma separated list

0.1.6 (2018-05-17)

Changed

• Improved error messages

• Handle json decode errors

0.1.5 (2018-05-11)

Added

• add no-color option

0.1.4 (2017-08-21)

Added

• add version option

• add file list option

Fixed

• Fix bug with empty consul key values

0.1.3 (2017-07-11)

Breaking change

• the consul tags will now use default jinja2 objects, this breaks the previous behaviour of separation by underscores

Changed

• remove dependency click, use argparse instead.

Fixed

• make sure that bytes are casted to strings (for base64 and consul tags)

0.1.2 (2017-05-17)

Fixed

• additional fix for install issue on python2 (added MANIFEST.in)

0.1.1 (2017-05-17)

Fixed

• add README.rst and CHANGELOG.rst as package_data this fixes install issue with python 2.x

Removed

• remove dependency colorama

Changed

• move methods to separate helper files

0.1.0 (2017-05-16)

Added

• add short options for extention (-e) searchlist (-s) and noop (-N)

• add (MIT) license

Changed

• e2j2 is now packaged as pip package

• split script and module, script will be installed in /usr/bin or /usr/local/bin

0.0.2 (2017-05-16)

Added

• Add recurse flag

Changed

• Searchlist is no longer a required option e2j2 will use the current directory as default

• Recursion is no longer on by default

• Improve error handling, e2j2 will now report failures and render *.err files which will make debugging errors much easier

0.0.1 (2017-05-01)

Initial release