metapensiero.tool.tinject 0.3

Automate creation of sources

author:

Lele Gaifax

contact:

[e-mail]

license:

GNU General Public License version 3 or later

This is a simple tool that takes a YAML declarative configuration of actions, each describing the set of steps needed to accomplish it, and realizes one or more action possibly interpolating a set of dynamic values inserted by the user.

I’ve written it with goal of taking away, as much as possible, the tediousness of creating new content in a PatchDB setup.

There is a myriad of similar tools around, but most of them are either targeted at bootstrapping a hierarchy of files and directories (that is, a one-shot operation, for example to create the skeleton of a new Python package, like cookiecutter to mention one) or belong to a very specific domain (an example of this category may be ZopeSkel).

To fulfill my goal I needed a tool able to perform an arbitrary list of operations, primarily:

• create a directory

• create a file with a particular content

• change the content of an existing file

It obviously need to be parametric in almost every part: the pathnames and the actual content of the files must be determined by a template interpolated with a set of values. This is what Jinja2 excels at. Since the values are mostly volatile (that is, almost all of them change from one invocation to the other), I used the inquirer library to prompt the user.

Contents

• Configuration

  • Global section

    • Jinja2 environment

    • File headers

    • Initial steps

  • Actions

  • Steps

• Sample session

• Changes

  • 0.3 (2016-05-22)

  • 0.2 (2016-05-19)

  • 0.1 (2016-04-26)

Configuration

The YAML configuration file may be composed by one or two documents: an optional global section and a mandatory one containing the list of available actions, each of them being basically a list of steps.

Global section

The first document contains a set of optional global settings, in particular:

• the options to instantiate the Jinja2 environment

• a set of file headers, one for each kind of file

• a list of steps to be performed at every invocation

Jinja2 environment

To avoid clashes with the syntax used by Python’s format() minilanguage, the default environment settings are the following:

block_start_string

The start of a block is <<, instead of the default {%

block_end_string

The end of a block is >>, instead of the default %}

variable_start_string

A variable is introduced by «, instead of the classic {{

variable_end_string

A variable ends with », and not with the classic }}

extensions

The extension jinja2_time.TimeExtension is automatically selected

keep_trailing_newline

By default the trailing newline is not discarded

Obviously any Jinja2 template defined by the configuration must adhere to these settings. If you don’t like them, or if you want define different ones, you can put an entry like the following in the globals section of the configuration:

jinja:
  block_start_string: "[["
  block_end_string: "]]"
  variable_start_string: "<<"
  variable_end_string: ">>"

File headers

This is a map between a file suffix and a template that will be automatically injected in the render’s context as the variable header. For example, should you need to generate a Pascal unit, you could insert the following:

headers:
  pas: |
    (*« "*" * 70 »
     * Project: «package_name»
     * Created: «timestamp»
     *« "*" * 70 »)

Initial steps

This is a list of steps to be executed unconditionally at startup. In particular it may be used to gather some values from arbitrary places to initialize the answers, to prompt the user for a common set of values and to define custom steps. As an example, you could say:

steps:
  - python:
      script: |
        import os, pwd, sys

        class InitCustomAnswersWithoutPrompt(Step):
            def __init__(self, state, config):
                super().__init__(state, config)
                self.name = config['name']
                self.value = config['value']

            def announce(self):
                self.state.announce('*', "Inject %s=%s", self.name, self.value)

            def __call__(self):
                return {self.name: self.value, python_version: sys.version}

        register_step('initcustom', InitCustomAnswersWithoutPrompt)

        myself = pwd.getpwuid(os.getuid())
        state.answers['author_username'] = myself.pw_name
        state.answers['author_fullname'] = myself.pw_gecos.split(',')[0]

  ## Here you can execute the new kind of operation defined above

  - initcustom:
      name: "myextravar"
      value: "thevalue"

Actions

An action is identified by a unique name and carries an optional description, an optional set of prompts specific to the action and a list of one or more steps.

The following is a complete example:

create_top_level_setup_py:
  description: Create top level setup.py

  prompt:
    - package_name:
        message: The name of the Python package

  steps:
    - createdir:
        directory: src

    - createfile:
        directory: src
        filename: setup.py
        content: |
          # Hi, I'm the setup.py file for «package_name»

Steps

A step is some kind of operation that must be carried out. The name of the step identifies the kind of operation, and its value is used to specify the needed parameters. So, in the example above, we have two steps, createdir and createfile, each requiring its specific arguments map.

This is the list of available operation kinds:

changefile

Perform some quite simple changes to the content of an existing file.

Required configuration:

directory

The directory containing the file to be changed

filename

The name of the existing file within the given directory

changes

A list of tweaks: there are currently just two types, one that add some content before a given marker, and one to add the content after it

Example:

- changefile:
    directory: src
    filename: listofitems.txt
    changes:
      - add: "«newitemname»\n"
        before: ";; items delimiter"
      - add: "«newitemname»\n"
        after: ";; reversed insertion order"

createdir

Create a directory and its parents.

Required configuration:

directory

The directory to be created

Example:

- createdir:
    directory: src/my/new/package

createfile

Create a new file with a particular content.

Required configuration:

directory

The directory contained the file to be created

filename

The name of the new file

content

A Jinja2 template that will be rendered and written to the new file

Example:

- createfile:
    directory: "«docs_dir»/«schema_name»/tables"
    filename: "«table_name».sql"
    description: Structure of table «schema_name».«table_name»
    ## The template may be either inline or included from an external file
    content: !include 'table.sql'

prompt

Ask the user for some information bits.

Required configuration: a list of dictionaries, each representing an inquirer’s question.

Example:

- prompt:
    - name_of_the_variable:
        message: Tell me the value
        default: "default value"

    - different_kind_of_input:
        message: Select the variant
        kind: list
        choices:
          - Big
          - Medium
          - Small

python

Execute an arbitrary Python script.

Required configuration:

script

The code of the script

The script is executed with a context containing the class Step, the function register_step and the global state of the program.

See the initial steps above for an example.

repeat

Repeat a list of substeps

Required configuration:

steps

The list of substeps to repeat

Optional configuration:

description

A message string, emitted at the start, if given

answers

The name of variable holding a list of answers, when one substep is a prompt

count

The number of iterations

again_message

When the number of iterations is not known (no count), the step will explicitly ask confirmation about looping again, at the end of each substeps execution

See examples/repeat.yml for an example.

Sample session

Create a new schema with a new table:

$ tinject --verbose apply examples/patchdb.yml new_schema new_table

* Execute Python script
[?] Author fullname (author_fullname): Lele Gaifax
[?] Author username (author_username): lele
[?] Author email (author_email): «author_username»@example.com
[?] Fully qualified package name (package_name): package.qualified.name
[?] Timestamp (timestamp): << now 'local', '%a %d %b %Y %H:%M:%S %Z' >>
[?] Year (year): << now 'local', '%Y' >>
[?] Distribution license (license): GNU General Public License version 3 or later
[?] Copyright holder (copyright): © «year» «author_fullname»
[?] Root directory of Sphinx documentation (docs_dir): docs/database
[?] Root directory of SQLAlchemy model sources (model_dir): src/«package_name|replace(".","/")»

=====================
 Create a new schema
=====================
[?] Name of the new schema (schema_name): public

* Create directory docs/database/public

* Create file docs/database/public/index.rst

* Create directory docs/database/public/tables

* Create file docs/database/public/tables/index.rst

* Create directory src/package/qualified/name/entities/public

* Create file src/package/qualified/name/entities/public/__init__.py

* Create directory src/package/qualified/name/tables/public

* Create file src/package/qualified/name/tables/public/__init__.py

====================
 Create a new table
====================
[?] Schema name of the new table (schema_name): public
[?] Name of the new table (table_name): things
[?] Description of the new table (table_description): The table ``«schema_name».«table_name»`` contains...
[?] Name of the corresponding entity (entity_name): Thing

* Create file docs/database/public/tables/things.rst

* Create file docs/database/public/tables/things.sql

* Change file docs/database/public/tables/index.rst

  - add “things” before “.. tinject toctree m…”

* Create file src/package/qualified/name/entities/public/thing.py

* Change file src/package/qualified/name/entities/public/__init__.py

  - add “from .thing import T…” before “## tinject import ma…”

  - add “mapper(Thing, t.thin…” before “## tinject mapper ma…”

* Create file src/package/qualified/name/tables/public/things.py

* Change file src/package/qualified/name/tables/public/__init__.py

  - add “from .things import …” before “## tinject marker, p…”

Verify:

$ cat src/package/qualified/name/entities/public/__init__.py
# -*- coding: utf-8 -*-
# :Project:   package.qualified.name -- Entities in schema public
# :Created:   lun 25 apr 2016 11:54:15 CEST
# :Author:    Lele Gaifax <[e-mail]>
# :License:   GNU General Public License version 3 or later
# :Copyright: © 2016 Lele Gaifax
#

from sqlalchemy.orm import mapper

from ...tables import public as t

from .thing import Thing
## tinject import marker, please don't remove!

mapper(Thing, t.things, properties={
})
## tinject mapper marker, please don't remove!

Add another table:

$ tinject --verbose apply examples/patchdb.yml new_table

* Execute Python script
[?] Author fullname (author_fullname): Lele Gaifax
[?] Author username (author_username): lele
[?] Author email (author_email): «author_username»@example.com
[?] Fully qualified package name (package_name): package.qualified.name
[?] Timestamp (timestamp): << now 'local', '%a %d %b %Y %H:%M:%S %Z' >>
[?] Year (year): << now 'local', '%Y' >>
[?] Distribution license (license): GNU General Public License version 3 or later
[?] Copyright holder (copyright): © «year» «author_fullname»
[?] Root directory of Sphinx documentation (docs_dir): docs/database
[?] Root directory of SQLAlchemy model sources (model_dir): src/«package_name|replace(".","/")»

====================
 Create a new table
====================
[?] Schema name of the new table (schema_name): public
[?] Name of the new table (table_name): thangs
[?] Description of the new table (table_description): The table ``«schema_name».«table_name»`` contains...
[?] Name of the corresponding entity (entity_name): Thang

* Create file docs/database/public/tables/thangs.rst

* Create file docs/database/public/tables/thangs.sql

* Change file docs/database/public/tables/index.rst

  - add “thangs” before “.. tinject toctree m…”

* Create file src/package/qualified/name/entities/public/thang.py

* Change file src/package/qualified/name/entities/public/__init__.py

  - add “from .thang import T…” before “## tinject import ma…”

  - add “mapper(Thang, t.than…” before “## tinject mapper ma…”

* Create file src/package/qualified/name/tables/public/thangs.py

* Change file src/package/qualified/name/tables/public/__init__.py

  - add “from .thangs import …” before “## tinject marker, p…”

Verify:

$ cat src/package/qualified/name/entities/public/__init__.py
# -*- coding: utf-8 -*-
# :Project:   package.qualified.name -- Entities in schema public
# :Created:   lun 25 apr 2016 11:54:15 CEST
# :Author:    Lele Gaifax <[e-mail]>
# :License:   GNU General Public License version 3 or later
# :Copyright: © 2016 Lele Gaifax
#

from sqlalchemy.orm import mapper

from ...tables import public as t

from .thing import Thing
from .thang import Thang
## tinject import marker, please don't remove!

mapper(Thing, t.things, properties={
})
mapper(Thang, t.thangs, properties={
})
## tinject mapper marker, please don't remove!

Changes

0.3 (2016-05-22)

• New ability to repeat a list of substeps

0.2 (2016-05-19)

• First release on PyPI

0.1 (2016-04-26)

• Initial effort