e2j2 is a commandline utility to render text/configuration files from jinja2 templatesfrom shell environment variables
Project description
e2j2 [ ]
What is e2j2?
e2j2 (environment to jinja2 variables) is a commandline tool which will render jinja2 templates to textfiles. all environment variables can be used in the jinja2 templates, within the environment variables you can use special tags which give you the option to insert json, json file paths, base64 hashes, consul kv keys.
e2j2 can be used within docker containers, or for other simple configuration templating tasks. Within a docker container you can simply add the j2 extention to a configuration file, or to multiple files within one or more folder structures and then run e2j2 as part of your docker entrypoint script.
Command line switches and config file
Most flags can be set on either the command line or in a json formatted config file. The following flags / configuration keys are supported:
Switch |
Type |
Config key |
Type |
Description |
---|---|---|---|---|
-h, –help |
Show help text and exit |
|||
-e, –ext, –extension |
string |
extension |
string |
Jinja2 file extention (default: .j2) |
-f, –filelist |
comma separated list |
filelist |
array |
List of jinja2 templates |
-s, –searchlist |
comma separated list |
searchlist |
array |
List of directories to search for templates (default: current directory) |
-N, –noop |
skip writing template to disk |
|||
-r, –recursive |
recursive |
boolean |
Traverse recursively through the search list |
|
–no-color |
no_color |
boolean |
Disable ANSI color |
|
-2, –twopass |
twopass |
boolean |
Enable two pass rendering |
|
–block_start |
string |
block_start |
string |
Block marker start (default: ‘{%’} |
–block_end |
string |
block_end |
string |
Block marker start (default: ‘%}’} |
–variable_start |
string |
block_start |
string |
Variable marker start (default: ‘{%’} |
–variable_end |
string |
variable_end |
string |
Variable marker end (default: ‘%}’} |
–comment_start |
string |
comment_start |
string |
Comment marker start (default: ‘{#’} |
–comment_end |
string |
comment_end |
string |
Comment marker start (default: ‘#}’} |
-w, –env_whitelist |
comma separated list |
env_whitelist |
array |
List of envars to include |
-b, –env_blacklist |
comma separated list |
env_blacklist |
array |
List of envars to exclude |
-P, –copy_file_permissions |
copy_file_permissions |
boolean |
Copy file permissions and ownership from template to rendered file |
|
-S, –stacktrace |
stacktrace |
boolean |
Include stacktrace in error file |
|
-C, –config |
string |
Config file path |
||
–watchlist |
comma separated list |
watchlist |
array |
Watch listed envvars for changes and render template(s) on change |
-R, –run |
string |
run |
array |
Run command after rendering templates |
–splay |
int |
splay |
integer |
Random delay between 1 and X seconds between watchlist polls |
Jinja2 filter support
By default only the jinja2 builtin filters are supported this can be extended by installing the jinja2-ansible-filters module.
Example
lets assume we want to render the following server block in nginx, if we place the server configuration in a nginx include directory for example /etc/nginx/conf.d
server {
server_name {{ NGINX.server_name }};
listen 80;
listen [::]:80;
error_page 500 502 503 504 /50x.html;
location / {
index {{ NGINX.index_page }};
root {{ NGINX.web_root }};
}
location ~ \.php$ {
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_pass unix:{{ NGINX.fpm_socket }};
fastcgi_split_path_info ^(.+\.php)(/.+)$;
include {{ NGINX.fcgi_params }};
root {{ NGINX.web_root }};
try_files $uri =404;
}
}
if you then set the NGINX environment variable, running e2j2 will render the jinja2 template and place it in the same folder:
~> export NGINX='json:
{
"server_name": "www.myweb.com",
"index_page": "index.php",
"web_root": "/usr/local/www/myweb",
"fcgi_params": "/usr/local/etc/nginx/myweb-fcgi-params",
"fpm_socket": "/var/run/php-fpm/myweb.socket"
}'
~> e2j2
In: .
rendering: nginx_vhost_config.conf.j2=>done => writing: nginx_vhost_config.conf=>done
~> cat nginx_vhost_config.conf
server {
server_name www.myweb.com;
listen 80;
listen [::]:80;
error_page 500 502 503 504 /50x.html;
location / {
index index.php;
root /usr/local/www/myweb;
}
location ~ \.php$ {
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_pass unix:/var/run/php-fpm/myweb.socket;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
include /usr/local/etc/nginx/myweb-fcgi-params;
root /usr/local/www/roundcube;
try_files $uri =404;
}
}
Environment variable examples
Plain environment variable
Example:
setting:
MYENVVAR='plain environment variable'
will render envvar-example.j2 to:
This is a plain environment variable
Tag file
Example:
setting:
MYFILEVAR=file:./file-example.txt
will render file-example.j2 to:
This is a file example
Tag json
Example:
setting:
MYJSONVAR='json:{"key": "json-example"}'
will render json-example.j2 to:
This is a json-example
Tag jsonfile
Example:
setting:
MYJSONFILEVAR='jsonfile:jsonfile-example.json'
will render jsonfile-example.j2 to:
This is a jsonfile example with subkey
Tag base64
Example:
Setting:
export MYBASE64VAR='base64:YmFzZTY0IGV4YW1wbGU='
will render base64-example.j2 to:
This is a base64 example
Tag consul
Configuration:
You can configure the consul tag by setting the CONSUL_CONFIG environment variable. The following config items are supported:
Item |
Explanation |
Default |
---|---|---|
url |
consul url |
|
scheme |
consul url scheme http or https |
scheme from url |
host |
consul host |
hostname from url |
port |
consul http(s) port |
port from url |
token |
consul token |
none |
Global config example:
read -d '' CONSUL_CONFIG << EOF { "url": "https://consul.foobar.tld", "token": "abcdef01-0123-abcd-1234-0123456789ab" } EOF
The ACL token can be configured by either the above configuration or by setting the CONSUL_TOKEN variable.
As an alternative for the global configuration it is also possible to configure / adjust the global configuration for each consul tag, by simply include the configuration when using the consul tag.
CONSUL_TOKEN and config key token can either contain the actual token or point to a file containing the token, use the file: tag to point to a file.
Tag config examples:
export MYCONSULVAR='consul:config={"url": "https://consul2.foobar.tld", "token": "012345678-0123-abcd-1234-0123456789ab"}:consulvar"
export MYCONSULVAR='consul:config={"url": "https://consul2.foobar.tld", "token": "file:/path/to/token"}:consulvar"
Consul example:
Setting:
key: consulvar in consul to value: consul example
and
export MYCONSULVAR='consul:consulvar'
will render consul-example.j2 to:
This is a consul example
Tag list
Example:
Setting:
export MYLIST='list:"first","second","third","fourth"'
will render list-example.j2 to:
"first" "second" "third" "fourth"
Two pass rendering
Starting from version 0.1.12 e2j2 supports embedding jinja2 macros in environment variables.
Example:
Setting the following two environment variables:
export WORDPRESS='json:{"database": {"name": "mydb", "user": "mydb_user", "password": "{{ DBSECRET }}", "host": "localhost"}}' export DBSECRET='file:./twopass-secret'
will render (by running: e2j2 -f twopass-example.j2 -2) to:
// ** MySQL settings - You can get this info from your web host ** // /** The name of the database for WordPress */ define( 'DB_NAME', 'mydb' ); /** MySQL database username */ define( 'DB_USER', 'mydb_user' ); /** MySQL database password */ define( 'DB_PASSWORD', 'Db$ecr3t' ); /** MySQL hostname */ define( 'DB_HOST', 'localhost' );
Tag vault
Configuration:
You can configure the vault tag by setting the VAULT_CONFIG environment variable. The following config items are supported:
Item |
Explanation |
Default |
---|---|---|
url |
vault url |
|
scheme |
vault url scheme http or https |
scheme from url |
host |
vault host |
hostname from url |
port |
vault http(s) port |
port from url |
backend |
vault secret backend |
raw |
token |
vault token |
none |
the following backends are supported:
backend |
Description |
---|---|
raw |
use plain GET request to secret store API |
kv1 |
key/value version 1 |
kv2 |
key/value version 2 |
Global config example:
read -d '' VAULT_CONFIG << EOF { "url": "https://vault.foobar.tld:8200", "token": "s.xxxxxxxxxxxxxxxxxxxxxxx", "backend: "kv2" } EOF
The Authentication token can be configured by either the above configuration or by setting the VAULT_TOKEN variable.
As an alternative for the global configuration it is also possible to configure / adjust the global configuration for each vault tag, by simply include the configuration when using the vault tag.
VAULT_TOKEN and config key token can either contain the actual token or point to a file containing the token, use the file: tag to point to a file.
Tag config example:
export MYVAULTVAR='vault:config={"backend": "kv2", "token": "s.xxxxxxxxx"}:kv/my-secret"
export MYVAULTVAR='vault:config={"backend": "kv2", "token": "file:/path/to/token"}:kv/my-secret"
Vault example:
Setting:
vault kv put secret/my-secret secret=topsecret export MYVAULTVAR='vault:secret/my-secret'
will render vault-kv1-example.j2 (by running: e2j2 -f vault-example.j2) to:
** topsecret ** This is a vault example
Tag dns
Configuration:
You can configure the dns tag by setting the DNS_CONFIG environment variable. The following config items are supported:
Item |
Explanation |
Default |
---|---|---|
nameservers |
overwrite nameservers |
use system resolvers |
port |
overwrite dns port |
53 |
type |
record type (A, AAAA or SRV) |
A |
the supported record types will return a dict with the following keys:
Type |
Keys |
---|---|
A |
address |
AAAA |
address |
SRV |
target, port, weight, priority |
DNS example:
Assuming a consul node running on localhost with the default dns port 8600.
Setting the DNS_CONFIG variable:
read -d '' DNS_CONFIG << EOF { "nameservers": ['127.0.0.1'], "port": 8600, "type": "SRV" } EOF
Setting:
export MYDNSVAR='dns:consul.service.consul'
will render dns-example.j2 (by running: e2j2 -f dns-example.j2) to:
My consul node: node1.node.dc1.consul. listening on port 8300
Changelog
0.4.10 (2020-01-14)
Changed
consul and vault tokens can now point to a file (use the file: tag to point to a file)
0.4.9 (2019-12-16)
Changed
follow symlinks when recursive flag is set
0.4.8 (2019-12-04)
Added
support jinja2 filters provided by the jinja2-ansible-filters library if installed
Fixed
fix handling consul keys containing newlines and double quote characters
0.4.7 (2019-11-25)
Fixed
searchlist is required even if E2J2_SEARCHLIST is set
0.4.6 (2019-11-24)
Added
add flag to run command after initial template run
add config option to json, jsonfile, consul, dns, and vault tag to flatten the returned json object
Changed
disable run on initial template run
0.4.5 (2019-11-19)
Changed
add test run to watch, only apply changes if test is successful
0.4.4 (2019-11-18)
Fixed
config var missing in get_var call
fix message repeat filter in stdout function
support ipv6 address in nameserver field of dns tag
0.4.3 (2019-11-15)
Added
add –splay flag, to configure delay of watchlist polls
alias flags for no color, white/black list and jinja2 markers
Fixed
stacktrace on ctrl-c
warning message in yellow with –no-color flag
multiple script runs (one for each template), should be just one
0.4.2 (2019-11-14)
Fixed
fixed incompatibility issue with python 2.7 and watchlist option
0.4.1 (2019-11-13)
Fixed
set default value for filelist flag to empty list
0.4.0 (2019-11-12)
Added
add dns tag (currently includes support of record types A, AAAA, and SRV)
config file support
watch for changes on consul, vault and dns tags and render on change (Experimental)
add run flag, this flag can be used to execute shell commands after rendering templates (Experimental)
Fixed
show only line numbers on a limited set up exceptions
0.3.0 (2019-10-25)
Changed
show line numbers on rendering errors
envvar tag errors are now disabled as warnings as they are non fatal
Breaking change:
envvar tag errors will result in undefined variables previously they contained an error message
0.2.1 (2019-10-23)
Added
add –stacktrace flag to include a python stack trace in the error file (thanks: Sylvia van Os/TheLastProject)
Changed - show jinja2 render error in console output (thanks: Marco Verleun)
Fixed - Fix for pip install issues - Pass exit code to calling shell
0.2.0 (2019-07-23)
Added
vault secrets support
per tag config (for the consul and vault tags)
add support for token variables (for the consul and vault tags)
Changed
Add json schema validation for consul and vault config
0.1.21 (2019-07-09)
Fixed
Fix issue with included templates in other directories
Added
Add option to “copy” file ownership and file permissions
0.1.20 (2019-06-12)
Fixed
fix “only render one file” issue
0.1.19 (2019-06-10)
Added
add option to whitelist / blacklist environment variables
Fixed
set proper exit codes (0=success / 1=failure)
Changed
add unittests
add functional tests
0.1.18 (2019-05-17)
Fixed
Fix bug in consul parser
0.1.17 (2019-05-17)
Fixed
Fix bug with include statements in jinja2 template
0.1.16 (2019-05-16)
Fixed
Fix bug with default jinja marker
0.1.15 (2019-05-16)
Changed
add options to overwrite the default jinja2 markers (block start/end, variable start/end and comment start/end)
Breaking change
removed “hack” for twopass rendering, you can use the new marker for handling conflicting characters in your template
0.1.14 (2019-04-30)
Fixed
preserve newlines at end of file
CVE-2019-10906, module now requires jinja2>=2.10.1
0.1.13 (2019-03-29)
Fixed
Fix for ignored raw tag with two-pass rendering
0.1.12 (2019-03-14)
Added
add support for two pass rendering
0.1.11 (2019-01-30)
Changed
Changed behavior of nested keys with the consul tag.
example: in consul we have an key key1/key2/123 envvar KEY=consul:key1/key2 => {{ KEY }} will now render to 123 and no longer to {“key2”: 123}
0.1.10 (2018-09-24)
Added
add file tag, this tag will place the content of the file into the variable
0.1.9 (2018-09-21)
Fixed
fix import issue on python 2.7
0.1.8 (2018-09-21)
Changed
handle hashrocket characters in json tag (as produced by hiera)
0.1.7 (2018-09-20)
Added
Add list tag which will parse a comma separated list
0.1.6 (2018-05-17)
Changed
Improved error messages
Handle json decode errors
0.1.5 (2018-05-11)
Added
add no-color option
0.1.4 (2017-08-21)
Added
add version option
add file list option
Fixed
Fix bug with empty consul key values
0.1.3 (2017-07-11)
Breaking change
the consul tags will now use default jinja2 objects, this breaks the previous behaviour of separation by underscores
Changed
remove dependency click, use argparse instead.
Fixed
make sure that bytes are casted to strings (for base64 and consul tags)
0.1.2 (2017-05-17)
Fixed
additional fix for install issue on python2 (added MANIFEST.in)
0.1.1 (2017-05-17)
Fixed
add README.rst and CHANGELOG.rst as package_data this fixes install issue with python 2.x
Removed
remove dependency colorama
Changed
move methods to separate helper files
0.1.0 (2017-05-16)
Added
add short options for extention (-e) searchlist (-s) and noop (-N)
add (MIT) license
Changed
e2j2 is now packaged as pip package
split script and module, script will be installed in /usr/bin or /usr/local/bin
0.0.2 (2017-05-16)
Added
Add recurse flag
Changed
Searchlist is no longer a required option e2j2 will use the current directory as default
Recursion is no longer on by default
Improve error handling, e2j2 will now report failures and render *.err files which will make debugging errors much easier
0.0.1 (2017-05-01)
Initial release
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.