polyvers 0.0.2a9.post0

Bump sub-project PEP-440 versions in Git monorepos independently.

version:

0.0.2a9.post0

updated:

2018-05-23T05:17:33.218516

Documentation:

https://polyvers.readthedocs.io

repository:

https://github.com/JRCSTU/polyvers

pypi-repo:

https://pypi.org/project/polyvers/, https://pypi.org/project/polyversion/

keywords:

version-management, configuration-management, versioning, git, monorepos, tool, library

copyright:

2018 JRC.C4(STU), European Commission (JRC)

license:

EUPL 1.2

A python 3.6+ command-line tool to manage PEP-440 version-ids of dependent sub-projects hosted in a Git monorepos, independently.

When bumping the version of sub-project(s), polyvers does the following:

• help you decide the next version of sub-projects, selectively and independently;

• add x2 tagged commits for each project bumped:

  • one in-trunk “Version” commit tags (pvtags) like foo-proj-v0.1.0, and another out-of-trunk (leaf) “Release” commit like foo-proj-r0.1.0;

• engrave the new versions in the source code of bumped-project(s) and all dependent sub-projects, but this happening only in the “leaf” version-commit;

• build packages out of the later (optionally);

• enforce (customizable) validation rules and run (extensible) hooks.

Additional capabilities and utilities:

• polyversion library code to extract sub-project’s version from past tags; provided as a separate subproject here, for not depending on the full development tool.

• It is still possible to use plain version tags (vtags) like v0.1.0, assuming you have a single project (called hereinafter a mono-project)

Table of Contents

• Quickstart

  • 1. Install the tool

  • 2. Initialize project

  • 3. Bump versions

  • 4. Engrave version in the sources

  • 5. Bump sub-projects selectively

• Features

  • PEP 440 version ids

  • Monorepos

  • Out-of-trunk (leaf) “Release” commit

  • Marking dependent versions across sub-projects

  • Lock release trains as “developmental”

  • Other Features

  • Features TODO

    • pre/post release hooks

    • Lock release trains as “developmental”

  • Known Limitations, Drawbacks & Workarounds

• Similar Tools

• Credits

• Changes

  • 0.0.2a9.post0 (2018-05-23): polyvers

  • 0.0.2a8 (2018-05-19): polyvers

    • 0.0.2a7 (2018-05-18)

  • 0.0.2a6 (2018-05-18)

  • 0.0.2a5 (2018-05-18)

    • 0.0.2a4 (2018-05-18)

    • 0.0.2a3 (2018-05-17)

    • 0.0.2a2 (2018-05-17)

  • 0.0.2a1 (2018-05-17)

    • 0.0.2a0 (2018-05-16)

  • 0.0.1a0 (2018-01-29)

Quickstart

1. Install the tool

And you get the polyvers command:

$ pip install polyvers
...
$ polyvers --version
0.0.0
$ polyvers --help
...

$ polyvers status
polyvers: Neither `setup.py` nor `.polyvers(.json|.py|.salt)` config-files found!

Note

Actually two projects are installed:

• polyvers cmd-line tool, for developing python monorepos,

• polyversion: the base python library used by projects developed with polyvers tool, so that their sources can discover their subproject-version on runtime from Git.

2. Initialize project

Assuming our monorepo project /monorepo.git/ contains two sub-projects:

/monorepo.git/
    +--setup.py:  setup(name='mainprog', ...)
    +--mainprog/__init__.py
    +--...
    +--core-lib/
        +--setup.py: setup(name='core', ...)
        +--core/__init__.py
        +--...

…we let the tool auto-discover the mapping of project folders ↔ project-names and create a traitlets configuration YAML-file named as /monorepo.git/.polyvers.py:

$ cd monorepo.git

$ polyvers init --monorepo
Created new config-file '.polyvers.yaml'.

$ cat .polyvers.yaml
...
PolyversCmd:
  projects:
  - pname: mainprog     # name extracted from `setup.py`.
    basepath: .         # path discovered by the location of `setup.py`
  - pname: core
    basepath: core-lib
...

$ git add .polyvers.yaml
$ git commit -m 'add polyvers config-gile'

And now we can use the polyvers command to inspect the versions of all sub-projects:

$ polyvers status
- mainprog
- core

Indeed there are no tags in in git-history for the tool to derive and display project-versions, so only project-names are shown. With --all option more gets displayed:

$ polyvers status -a
- pname: mainprog
  basepath: .
  gitver:
  history: []
- pname: core
  basepath: core-lib
  gitver:
  history: []

..where gitver would be the result of git-describe.

3. Bump versions

We can now use tool to set the same version to all sub-projects:

$ polyvers bump 0.0.0 -f noengraves   # all projects implied, if no project-name given
00:52:06       |WARNI|polyvers.bumpcmd.BumpCmd|Ignored 1 errors while checking if at least one version-engraving happened:
  ignored (--force=noengraves): CmdException: No version-engravings happened, bump aborted.
00:52:07       |NOTIC|polyvers.bumpcmd.BumpCmd|Bumped projects: mainprog-0.0.0 --> 0.0.0, core-0.0.0 --> 0.0.0

The --force=noengraves disables a safety check that requires at least one file modification for engraving the current version in the leaf “Release” commit (see next step).

 $ polyvers status
 - mainprog-v0.0.0
 - core-v0.0.0

 $ git lg    # Ok, augmented `lg` output a bit here...HEAD --> UPPER branch.
 COMMITS BRANCH TAGS                 REMARKS
 ======= ====== ==================== ========================================
      O  latest mainprog-r0.0.0      - x2 tags on "Release" leaf-commit
     /          core-r0.0.0            outside-of-trunk (not in HEAD).
    O    MASTER mainprog-v0.0.0      - x2 tags on "Version" commit
    |           core-v0.0.0            for bumping both projects to v0.0.0
    O                                - Previous commit, before version bump.

.. Hint::
 Note the difference between ``ABC-v0.0.0`` vs ``ABC-r0.0.0`` tags.

In the source code, it's only the "release" commit that has *engraved* version-ids:

.. code-block:: console

 $ cat mainprog/mainprog/__init__.py    # Untouched!
 import polyvers

 __title__     = "mainprog"
 __version__ = polyvers.version('mainprog')
 ...

 $ git checkout  latest
 $ cat mainprog/mainprog/__init__.py
 import polyvers

 __title__     = "mainprog"
 __version__ = '0.0.0'
 ...

 $ git checkout  -  # to return to master.

4. Engrave version in the sources

Usually programs report their version somehow when run, e.g. with `cmd --version. With polyvers we can derive the latest from the tags created in the previous step, using a code like this, usually in the file /mainprog/mainprog/__init__.py::

import polyvers

__title__ = "mainprog"
__version__ = polyvers.version('mainprog')
...

…and respectively /core-lib/core/__init__.py::

__version__ = polyvers.version('core')

5. Bump sub-projects selectively

Now let’s add another dummy commit and then bump ONLY ONE sub-project:

$ git commit  --allow-empty  -m "some head work"
$ polyvers bump ^1 mainprog
00:53:07       |NOTIC|polyvers.bumpcmd.BumpCmd|Bumped projects: mainprog-0.0.0 --> 0.0.1

$ git lg
COMMITS BRANCH TAGS                 REMARKS
======= ====== ==================== ========================================
     O  latest mainprog-r0.0.1.dev0 - The latest "Release" leaf-commit.
    /                                 branch `latest` was reset non-ff.
   O    MASTER mainprog-v0.0.1.dev0 - The latest "Version" commit.
   O                                - some head work
   | O         mainprog-r0.0.0      - Now it's obvious why "Release" commits
   |/          core-r0.0.0            are called "leafs".
   O           mainprog-v0.0.0
   |           core-v0.0.0
   O

$ git checkout latest
$ cat mainprog/mainprog/__init__.py
import polyvers

__title__     = "mainprog"
__version__ = '0.0.1.dev0'
...

$ cat core/core/__init__.py
import polyvers

__title__ = "core"
__version__ = '0.0.0+mainprog.0.0.1.dev0'
...
$ git checkout -

Notice how the the “local” part of PEP-440 (statring with +...) is used by the engraved version of the un-bumped core project to signify the correlated version of the bumped mainprog. This trick is uneccesary for tags because they apply repo-wide, to all sub-projects.

Features

PEP 440 version ids

While most versioning tools use Semantic versioning, python’s distutils native library supports the quasi-superset, but more versatile, PEP-440 version ids, like that:

• Pre-releases: when working on new features:

  X.YbN               # Beta release
  X.YrcN  or  X.YcN   # Release Candidate
  X.Y                 # Final release

• Post-release:

  X.YaN.postM         # Post-release of an alpha release
  X.YrcN.postM        # Post-release of a release candidate

• Dev-release:

  X.YaN.devM          # Developmental release of an alpha release
  X.Y.postN.devM      # Developmental release of a post-release

Monorepos

When your single project succeeds, problems like these are known only too well:

Changes in web-server part depend on core features that cannot go public because the “official” wire-protocol is freezed.

While downstream projects using core as a library complain about its bloated transitive dependencies (asking why flask library is needed??).

So the time to “split the project has come. But from lerna:

Splitting up large codebases into separate independently versioned packages is extremely useful for code sharing. However, making changes across many repositories is messy and difficult to track, and testing across repositories gets complicated really fast.

So a monorepo [1] [2] is the solution. But as Yarn put it:

OTOH, splitting projects into their own folders is sometimes not enough. Testing, managing dependencies, and publishing multiple packages quickly gets complicated and many such projects adopt tools such as …

Polyvers is such a tool.

[1]

<https://medium.com/@maoberlehner/monorepos-in-the-wild-33c6eb246cb9

[2]

http://www.drmaciver.com/2016/10/why-you-should-use-a-single-repository-for-all-your-companys-projects/

Out-of-trunk (leaf) “Release” commit

Even in single-project repos, sharing code across branches may cause merge-conflicts due to the version-ids “engraved” in the sources. In monorepos, the versions proliferate, and so does the conflicts.

Contrary to similar tools, static version-ids are engraved only in out-of-trunk (leaf) commits, and only when the sub-projects are released. In-trunk code is never touched, and version-ids are reported, on runtime, based on Git tags (like git-describe), so they are always up-to-date.

Marking dependent versions across sub-projects

TODO: When bumping the version of a sub-project the “local” part of PEP-440 on all other the dependent sub-projects in the monorepo signify their relationship at the time of the bump.

Lock release trains as “developmental”

TODO: Specific branches can be selected always to be published into PyPi only as PEP-440’s “Developmental” releases, meanining that users need pip install --pre to install from such release-trains. This is a safeguard to avoid accidentally landing half-baked code to users.

Other Features

• Highly configurable using traitlets, with sensible defaults; it’s possible to run without any config file in single-project repos.

• Always accurate version reported on runtime when run from git repos (never again wonder with which version your experimental-data were produced).

Features TODO

pre/post release hooks

Possible to implement hooks as setuptools plugins. to run, for example, housekeeping commands on all subprojects like pip install -e <project> and immediately start working in “develop mode”.

This functionality would also allow to validate tests before/after every bump:

## Pre-release hook
#
pytest tests


## Post-release hook
#
rm -r dist/* build/*;
python setup.py sdist bdist_wheel
twine upload dist/*whl -s

Lock release trains as “developmental”

Specific branches can be selected always to be published into PyPi only as PEP-440’s “Developmental” releases, meanining that users need pip install --pre to install from such release-trains. This is a safeguard to avoid accidentally landing half-baked code to users.

Known Limitations, Drawbacks & Workarounds

• PEP440 Epoch handling is not yet working.

• Version-bump’s grammar is not yet as described in “GRAMMAR” section of command’s doc:

  $ polyvers config desc --class BumpCmd
  BumpCmd(_SubCmd)
  ----------------
  Increase or set the version of project(s) to the (relative/absolute) version.
  SYNTAX:
      polyvers config desc [OPTIONS] <version> [<project>]...
  - If no project(s) specified, increase the versions on all projects.
  - Denied if version for some projects is backward-in-time (or has jumped parts?);
    use --force if you might.
  VERSION: - A version specifier, either ABSOLUTE, or RELATIVE to the current
  version og each project:
    - *ABSOLUTE* PEP-440 version samples:
      - Pre-releases: when working on new features:
          X.YbN               # Beta release
          X.YrcN  or  X.YcN   # Release Candidate
          X.Y                 # Final release
  ...

• WARNING: when you build your package for distribution (wheel, correct?) remember to switch to the out-of-trunk (leaf) “Release” commit. This is particularly important if your setup.py file use polyversion() to derive its version.. Because if it fails for whatever reason (git command is missing, project not located in a git-repo, miss-configuration, etc).

  Check also that if you provide a default argument to facilitate development, then you may actually build a package(wheel, ok?) with that “default” version. So, always check you package’s version before uploading it to pypi.

• (not related to this tool) In setup.py script, the kw-argument package_dir={'': <sub-dir>} arg is needed for py_modules to work when packaging sub-projects (also useful with find_packages(), check this project’s sources). But <sub-dir> must be relative to launch cwd, or else, pip install -e <subdir> and/or python setup.py develop break.

• (not related to this tool) When building projects with python setup.py bdist_wheel, you have to clean up your build directory, or else, the distribution package will contain the sources from all previous subprojects. That applies also when rebuilding a project between versions.

• (not related to this tool) If you don’t place a setup.py file at the root of your git-repo, then it becomes more cumbersome to pip install directly from remote URLs, like this:

  pip install -e git+https://repo_url/#egg=pkg&subdirectory=pkg_dir

  You may use package_dir argument to setup() function (see setuptools-docs).

• Set branch latest as default in GitHub to show engraved sub-project version-ids.

Similar Tools

• The original bumpversion project; development stopped after 2015: https://github.com/peritus/bumpversion

• bump2version: active clone of the original: https://github.com/c4urself/bump2version

• releash: another monorepos managing tool, that publishes also to PyPi: https://github.com/maartenbreddels/releash

• Git Bump using git-hooks: https://github.com/arrdem/git-bump

• Search other 34 similar projects in GitHub.

• https://github.com/korfuri/awesome-monorepo

• Lerna: A tool for managing JavaScript projects with multiple packages.

• Pants: a build system designed for codebases that: - Are large and/or growing rapidly. - Consist of many subprojects that share a significant amount of code. - Have complex dependencies on third-party libraries. - Use a variety of languages, code generators and frameworks.

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Changes

0.0.2a9.post0 (2018-05-23): polyvers

• feat: add -C option to change project dir before running command.

• init command:

  • fix: were creating invalid .polyvers.yaml configuration-file unless --monorepo/--mono-project flags were given.

  • feat: include config-help in generated file only if the new --doc flag given.

  • feat: inform user of the projects auto-discovered and what type of config-file was generated.

• various fixes.

0.0.2a8 (2018-05-19): polyvers

• FIX(bump): was engraving all projects and not limiting to those specified in the command-line - command’s syntax slightly changed.

• chore: Stop increasing polyversion version from now on.

• doc: fix all sphinx errors and API reference.

0.0.2a7 (2018-05-18)

Interim release to embed re-LICENSED bin/pvlib.whl, from EUPLv1.2–>MIT

0.0.2a6 (2018-05-18)

• bump command:

  • feat: --amend now works

  • feat: --engrave-only.

  • feat: log PRETEND while doing actions.

  • feat: Log which files where engraved in the final message.

• fix(engrave): don’t waste cycles/log-messages on empty-matches (minor).

0.0.2a5 (2018-05-18)

Actually most changes happened in “interim” release v0.0.2a2, below.

• feat: make a standalone polyversion-lib wheel to facilitate bootstrap when installing & building from sources (and the lib is not yet installed).

• Add bin/package.sh that create the pvlib wheel as executable pvlib.run.

• doc: fix rtd & pypi sites.

0.0.2a4 (2018-05-18)

doc: bad PyPi landing page.

0.0.2a3 (2018-05-17)

The pvcmd was actually broken so far; was missing polyversion lib dependency!

0.0.2a2 (2018-05-17)

Interim release to produce executable wheel needed by next release.

0.0.2a1 (2018-05-17)

• 2nd release, own “mono-project” splitted into 2-project “monorepo”: - polyvers: cmdline tool - polyversion: library code for program-sources to derive version from git-tags

• init, status, bump and config commands work.

• Read/write YAML config file .polyvers.yaml at the git-root, and can automatically discover used configuration (from existing git tags or projects files).

• Support both --monorepo and --mono-project configurations.

• By default __init__.py, setup.py and README.rst files are engraved with bumped version.

0.0.2a0 (2018-05-16)

broken

0.0.1a0 (2018-01-29)

• First release on PyPI as mono-project