jarn.mkrelease 5.3

Python package releaser

mkrelease is a no-frills Python package releaser. It is designed to take the cumber out of building and distributing Python packages. The releaser supports source distributions, eggs, and wheels.

Motivation

After preparing a package for release (update version strings, dates) we typically have to:

1. Commit modified files.

2. Tag the release.

3. Build a source distribution and wheel.

4. Distribute the results via scp or upload them to an index server.

If we are doing this a lot, the need for automation becomes obvious.

Contents

• Installation

• Usage

• Options

• Arguments

• Configuration

• Upload with SCP

• Upload to Index Servers

• Using GnuPG

• Requirements

• Footnotes

• Related

• Changelog

Installation

mkrelease works with Python 2.7 - 3.11 and all released versions of setuptools.

Use pip install jarn.mkrelease to install the mkrelease script.

Since version 4.4 mkrelease requires twine for register and upload operations. Twine may be installed as a global utility on the system PATH or into the same environment as jarn.mkrelease. [1]

Use pip install jarn.mkrelease[twine] to install mkrelease + twine.

Since version 5.0 file-finder extensions are no longer installed by default. If you still want to use file-finders specify the filefinder extra.

Use pip install jarn.mkrelease[filefinder] to install mkrelease + file-finder extensions.

Use pip uninstall setuptools-subversion setuptools-hg setuptools-git to uninstall file-finder extensions.

Usage

mkrelease [options] [scm-sandbox|scm-url [rev]]

Options

-C, --no-commit

Do not commit modified files from the sandbox.

-T, --no-tag

Do not tag the release in SCM.

-P, --no-push

Do not push commits and tags upstream.

-R, --no-register

Do not register the release with dist-location.

-S, --no-upload

Do not upload the release to dist-location.

-n, --dry-run

Dry-run; equivalent to -CTPRS. This flag turns mkrelease into a simple package builder.

--svn, --hg, --git

Select the SCM type. Only required if the SCM type cannot be guessed from the argument.

-d dist-location, --dist-location=dist-location

An scp destination specification, an index server configured in ~/.pypirc, or an alias name for either. This option may be specified more than once.

-s, --sign

Sign the release with GnuPG.

-i identity, --identity=identity

The GnuPG identity to sign with. Implies -s.

-z, --zip

Release a zip archive.

-g, --gztar

Release a tar.gz archive (default).

-b, --egg

Release a binary egg.

-w, --wheel

Release a wheel file (default).

-m, --manifest-only

Ignore setuptools file-finder extensions and collect files via MANIFEST.in only.

-e, --develop

Allow setuptools build tags. Implies -T.

-q, --quiet

Suppress output of setuptools commands.

-t twine, --twine=twine

Override the twine executable used.

-c config-file, --config-file=config-file

Use config-file instead of the default ~/.mkrelease.

-l, --list-locations

List known dist-locations and exit.

-h, --help

Print the help message and exit.

-v, --version

Print the version string and exit.

--no-color

Disable output colors.

--non-interactive

Do not prompt for username and password if the required credentials are missing.

Arguments

scm-sandbox

A local SCM sandbox. Defaults to the current working directory.

scm-url [rev]

The URL of a remote SCM repository. The optional rev argument specifies a branch or tag to check out.

Configuration

mkrelease reads available index servers from the distutils configuration file ~/.pypirc. This file should contain your PyPI account information: [2]

[distutils]
index-servers =
    pypi

[pypi]
repository = https://upload.pypi.org/legacy/
username = fred
password = secret

Next, mkrelease reads its own configuration file ~/.mkrelease. It may be used to change defaults and to define location aliases:

[mkrelease]
# Release steps
commit = yes
tag = yes
push = yes
register = no
upload = yes

# One or more of: zip gztar egg wheel
formats = gztar wheel

# Setuptools options
manifest-only = yes
develop = no
quiet = no

# Sign with GnuPG
sign = no
identity =

# Default dist-location
dist-location =

[aliases]
# Map name to one or more dist-locations
customerA =
    jarn.com:/var/dist/customerA/
public =
    jarn.com:/var/dist/public/
world =
    pypi
    public

Upload with SCP

The simplest distribution location is a server directory reachable by ssh. Releasing a package means scp-ing it to the appropriate place on the server:

$ mkrelease -d customerA
$ mkrelease -d jarn.com:/var/dist/customerB/
$ mkrelease -d scp://jarn.com/var/dist/customerC/
$ mkrelease -d [e-mail]:eggs -C -e -q

To upload via sftp instead of scp, use the sftp URL scheme:

$ mkrelease -d sftp://jarn.com/var/dist/customerD/

Note: Unlike scp, the sftp client does not prompt for login credentials. This means that non-interactive login must be configured on the destination server or the upload will fail.

Upload to Index Servers

Another way of publishing a Python package is by uploading it to a dedicated index server like PyPI. Given the ~/.pypirc file from above, we can release to PyPI simply by typing:

$ mkrelease -d pypi

Index servers are not limited to PyPI though. There is TestPyPI, and there are alternative index servers like devpi. We extend our ~/.pypirc: [2]

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
repository = https://upload.pypi.org/legacy/
username = fred
password = secret

[testpypi]
repository = https://test.pypi.org/legacy/
username = fred
password = secret

We can now release to TestPyPI with:

$ mkrelease -d testpypi -C -e

Using GnuPG

Release a package and sign the distributions with GnuPG:

$ mkrelease -d pypi -s -i [e-mail]

The -i flag is optional and GnuPG will pick your default key if not given.

Requirements

The following commands must be available on the system PATH (you only need what you plan to use):

• svn

• hg

• git

• scp

• sftp

• gpg

• twine [1]

Footnotes

[1] (1,2)

The twine executable is determined by trying in order:

1. Value of --twine command line option, or

2. Value of TWINE environment variable, or

3. Value of twine configuration file setting, or

4. python -m twine if twine is importable, or

5. twine

[2] (1,2)

There are more secure ways to handle login credentials:

1. Generate an API token on PyPI, and/or

2. Make use of twine’s keyring support.

Related

Also see our Python documentation viewer jarn.viewdoc.

Changelog

5.3 - 2023-09-23

• Update tox.ini for latest tox. [stefan]

• Filter deprecated config warning. [stefan]

• Fix invalid escape sequences. [stefan]

• The rev part of Git URLs no longer defaults to ‘master’. [stefan]

5.2 - 2023-01-20

• Support setuptools >= 65.6.0. [stefan]

5.1 - 2022-11-13

• Support Python 2.7 - 3.11. [stefan]

• Do not require setup.py or setup.cfg if pyproject.toml exists. [stefan]

• Filter some deprecation warnings. [stefan]

• Disable twine progress bars in non-interactive mode. [stefan]

5.0 - 2022-02-26

• NEW: File finder extensions are no longer installed by default. [stefan]

• NEW: push is now on by default [stefan]

• NEW: register is now off by default [stefan]

• NEW: formats now defaults to gztar + wheel [stefan]

• Do not require setup.py if setup.cfg exists. [stefan]

• Switch to the nose2 test runner for its fast multiprocessing plugin. [stefan]

4.4 - 2022-02-26

• Support Python 2.7 - 3.10. [stefan]

• Support local distutils in setuptools >= 60.0.0. [stefan]

• Add output colors. [stefan]

• Require twine for register and upload operations. [stefan]

• Move metadata to setup.cfg. [stefan]

• Move tests out of jarn.mkrelease namespace. [stefan]

• Include tests in sdist but not in wheel. [stefan]

4.3 - 2019-01-28

• Fix issue #10: Can no longer run from a zc.buildout. [stefan]

• Support python -m jarn.mkrelease. [stefan]

4.2 - 2019-01-25

• Drop Python 2.6 support, add Python 3.7. [stefan]

• Update wheel and keyring dependencies. [stefan]

• Add a complete config file example to the README. [stefan]

• Accept dist-location as alias for distdefault in config files. [stefan]

• Convert dashes to underscores in config parser optionxform. [stefan]

4.1 - 2017-10-06

• Add -m option to skip setuptools extensions. [stefan]

• Never read existing SOURCES.txt files. [stefan]

• Allow –egg as alias for –binary. [stefan]

4.0.1 - 2017-07-20

• Fix conditional include of keyring. [stefan]

4.0 - 2017-07-20

• Add support for source distributions in gztar format. [stefan]

• Add support for wheel files. [stefan]

• Allow multiple archive formats per release. [stefan]

• Protect against bad or incomplete locale settings. [stefan]

• The secondary thread would sometimes not terminate in Python 2. [stefan]

3.10 - 2017-02-01

• Support setuptools >= 33.1.0. [stefan]

3.9 - 2017-01-31

• Catch up with late-2016 PyPI API change. [stefan]

• Add -R option to skip the register step. [stefan]

• Support Python 2.6 - 3.6 without 2to3. [stefan]

• Handle return code changes in Mercurial 3.1.1. [stefan]

3.8 - 2013-11-21

• Support Python 3.x. [stefan]

3.7 - 2012-08-22

• Fix compilation of Python source files when the -b option is given. [stefan]

• Run check command as part of sdist and register commands. [stefan]

• Add SFTP support. [stefan]

• Allow sftp:// and scp:// URLs as dist-locations. [stefan]

3.6 - 2012-07-11

• Handle return code changes in Mercurial 2.1. [stefan]

• Add setuptools-subversion dependency. [stefan]

• Support Subversion 1.7 with the help of setuptools-subversion. [stefan]

3.5 - 2011-11-25

• Allow multiple values for the distdefault config file option. [stefan]

• Defer list-locations until after all arguments have been parsed. [stefan]

• Make tests run twice as fast by avoiding Subversion checkouts. [stefan]

3.4 - 2011-11-10

• Warn if -p is given but no upstream location is found. [stefan]

• Always push to default in Mercurial. [stefan]

• Avoid reading empty lines from terminating subprocesses. [stefan]

• Fix bug in handling of distbase. [stefan]

3.3 - 2011-10-31

• Add setuptools to the PYTHONPATH for subprocesses. [stefan]

• Unset any PYTHONPATH while executing SCM commands. [stefan]

• Support Git’s short-form ssh:// URLs. [stefan]

• Add -c option to specify a config file other than ~/.mkrelease. [stefan]

3.2 - 2011-10-21

• Fix the environment passed to subprocesses; Mercurial did not appreciate the mangled PYTHONPATH. [stefan]

• Allow to specify the branch or tag to check out from Git and Mercurial repositories. [stefan]

• Adapt to new status output in Subversion 1.6. [stefan]

• Always include distdefault in list-locations. [stefan]

• Detect Subversion repos from file:// URLs. [stefan]

• Detect bare Git repos from file:// URLs. [stefan]

3.1 - 2011-07-19

• Pass the PYTHONPATH to subprocesses so mkrelease works in zc.buildout environments. [stefan]

• Improve SCM detection in situations where one or more SCMs are nested. [stefan]

• Add support for relative file: URLs. [stefan]

• Depend on lazy instead of carrying a local implementation. [stefan]

3.0.10 - 2011-07-07

• Add -l option to list known dist-locations (i.e. servers and aliases). [stefan]

• Drop support for server URLs as dist-locations. Server URLs are not unique. [stefan]

• Update the Mercurial test repository so tagging tests don’t fail under Mercurial 1.8. [stefan]

3.0.9 - 2010-12-31

• Rename [defaults] configuration file section to [mkrelease]. [stefan]

• Various internal code cleanups. [stefan]

3.0.8 - 2010-08-13

• Avoid underscores in dependency names. [stefan]

• Handle return code changes in Mercurial 1.6. [stefan]

3.0.7 - 2010-07-07

• Improve documentation and error messages. [stefan]

3.0.5 - 2010-03-23

• Allow per-server configuration of -s and -i defaults. [stefan]

• Support the codespeak.net Subversion repository layout. [stefan]

3.0.4 - 2010-03-16

• Status checks didn’t use the same path restrictions as commits (Mercurial and Git.) [stefan]

3.0.3 - 2010-03-16

• Change how we check for existing tags in Subversion repositories. [stefan]

• Make sandbox-status checks more robust in all three SCMs. [stefan]

3.0.2 - 2010-03-12

• Add support for Git 1.7. [stefan]

3.0.1 - 2010-02-07

• Stop when -d pypi is given but no configuration can be found. [stefan]

• Use gnu_getopt to parse the command line. [stefan]

3.0 - 2010-01-15

• Switch to -n for dry-run to be consistent with other tools. [stefan]

• Rename –skip-* long options to –no-* for the same reason. [stefan]

• Fix a bug in Mercurial and Git sandbox detection. [stefan]

• Prepare for standalone distutils. [stefan]

2.0.4 - 2010-01-10

• Improve Git support to handle remotes other than origin. [stefan]

• Fix SCM detection in ssh:// URLs. [stefan]

2.0.3 - 2010-01-03

• Add -b option for releasing binary eggs. [stefan]

• Don’t choke on dirty sandboxes when dry-running. [stefan]

2.0.2 - 2009-08-29

• Filter meta files (.svn*, .hg*, .git*) and never include them in releases. [stefan]

• Make sure to clean up all temporary files. [stefan]

2.0.1 - 2009-07-24

• Fixed bug which could cause mkrelease to issue eggs with faulty manifest files (Symptom: data files not installed). [stefan]

• The -e flag now implies -T. We never want to tag a development release. [stefan]

2.0 - 2009-07-16

• Allow command line options to appear after the argument. As in: mkrelease src/mypackage -q -d pypi. [stefan]

2.0b2 - 2009-07-09

• Improve user feedback in the SCM-detection part. [stefan]

• Document the -e flag. [stefan]

• Drop global configuration file for YAGNI. [stefan]

• Allow to set default values for -s and -i in ~/.mkrelease. [stefan]

2.0b1 - 2009-07-03

• By default, ignore all version number extensions (dev-r12345) that may be configured in setup.cfg. Passing the -e flag disables this safeguard. [witsch, stefan]

• Delete any existing signature file before signing anew. This keeps GnuPG from complaining about existing (but left-over) files. [stefan]

2.0a2 - 2009-06-27

• Drop configurable Python and use sys.executable. This also means we now require Python 2.6. [stefan]

• Force setuptools to only use file-finders for the selected SCM type. This is required to support multi-SCM sandboxes (think git-svn). [stefan]

• Treat Subversion sandboxes just like the others and avoid the temporary checkout step. [stefan]

• Remove the -u flag for being pointless. [stefan]

2.0a1 - 2009-06-14

• Added support for Mercurial and Git. [stefan]

• Added 250+ unit tests. [stefan]

1.0.2 - 2009-06-13

• Documented long options. [stefan]

• Print a “Tagging …” line before tagging. [stefan]

1.0 - 2009-05-14

• Print help and version to stdout, not stderr. [stefan]

1.0b4 - 2009-04-30

• Since distutils commands may return 0, successful or not, we must check their output for signs of failure. [stefan]

• Allow to pass argument list to main(). [stefan]

1.0b3 - 2009-03-23

• No longer depend on grep. [stefan]

• Use subprocess.Popen instead of os.system and os.popen. [stefan]

• Protect against infinite alias recursion. [stefan]

• Drop -z option and always create zip files from now on. [stefan]

1.0b2 - 2009-03-19

• Checkin everything that’s been modified, not just “relevant” files. [stefan]

• Expand aliases recursively. [stefan]

1.0b1 - 2009-03-18

• The distbase and distdefault config file options no longer have default values. [stefan]

• Read index servers from ~/.pypirc and allow them to be used with -d. [stefan]

• The -d option may be specified more than once. [stefan]

• Dropped -p option. Use -d pypi instead. [stefan]

• Dropped -c option. If your have non-standard SVN repositories you must tag by hand. [stefan]

0.19 - 2009-02-23

• Absolute-ize the temp directory path. [stefan]

0.18 - 2009-01-26

• Include README.txt and CHANGES.txt in long_description. [stefan]

• Rid unused imports and locals. [stefan]

0.17 - 2009-01-23

• Add -c option to enable codespeak support. The codespeak.net repository uses branch and tag instead of branches and tags. [gotcha, stefan]

0.16 - 2009-01-13

• Fold regex construction into find and make find a method. [stefan]

• Update README.txt. [stefan]

0.15 - 2009-01-13

• Support for reading default options from a config file. [fschulze, stefan]

0.14 - 2009-01-08

• Add -s and -i options for signing PyPI uploads with GnuPG. [stefan]

• Stop execution after any failing step. [stefan]

0.13 - 2009-01-05

• Stop execution when the checkin step fails. [stefan]

0.12 - 2009-01-02

• setup.cfg may not exist. [stefan]

0.11 - 2008-12-02

• Add setup.cfg to list of files we check in. [stefan]

0.10 - 2008-10-21

• Don’t capitalize GetOptError messages. [stefan]

0.9 - 2008-10-16

• Add -v option to print the script version. [stefan]

0.8 - 2008-10-16

• Lift restriction where only svn trunk could be released. [stefan]

0.7 - 2008-10-09

• Fix PyPI upload which must happen on the same command line as sdist. [stefan]

0.6 - 2008-10-08

• Update README.txt. [stefan]

0.5 - 2008-10-08

• Also locate and checkin HISTORY.txt to support ZopeSkel’ed eggs. [stefan]

0.4 - 2008-10-08

• Use svn checkout instead of svn export because it makes a difference to setuptools. [stefan]

• Add -p option for uploading to PyPI instead of dist-location. [stefan]

0.3 - 2008-10-06

• Also locate and checkin version.txt. [stefan]

0.2 - 2008-10-01

• Add -z option to create zip archives instead of the default tar.gz. [stefan]

0.1 - 2008-10-01

• Initial release [stefan]