Skip to main content

Automatically update Python Eggs on application startup.

Project description

Automatically update Python Eggs on application startup – or, you know, whenever, really.

Overview

The idea behind this module is to support self-updating Python applications, namely command-line tools. Transmute probes remote repositories for updated components, fetches updates and adds them to sys.path, making them available for use in the application.

Components are assumed to be standard Python Eggs. Repositories are then simple containers for these eggs. Currently PyPI and S3 “folders” are supported as repositories. (Mostly for testing purposes, local directories are also supported as repositories).

Under the hood, pkg_resources (from setuptools) is used to parse and fulfill requirements, based on listings of available eggs obtained from each repository. Once updated packages are made available, their modules can be imported or the application can be re-launched with an adjusted environment to pick up updated modules.

The application writer controls the packages to update, repositories each package will be grabbed from, and when an update is actioned.

The philosophy has been that an absent or flaky network should not prevent (or significantly delay) an application from running on top of outdated packages, if they’ve been cached locally. That said, there are currently no provisions for testing and verifying a successful update or rolling back a failed update.

Example

This is a script that requests the ‘hello’ package to be updated from the dist folder in the current working directory:

import transmute

transmute.require([ 'hello' ], sources=[ 'dist' ])
transmute.update()

import hello
hello.greet('world')

Bootstrapping an application with bootstrap.py

The submodule in `transmute/bootstrap.py <https://github.com/comoyo/python-transmute/blob/master/transmute/bootstrap.py>`__ can be used on its own to bootstrap other Python modules and applications. It is capable of downloading packages from PyPI. In this way, transmute itself can be loaded and further used to download additional packages.

The script provides a bunch of hooks where users can place their code. In particular, main() can be filled in to fetch application specific packages and actually launch the application. At the point it is called transmute has been added to sys.path (after downloading from PyPI, if needed).

def main():
    import transmute
    import transmute.s3

    transmute.require([ 'foobar' ],
            sources=[ 's3://foobar-repository/eggs/foobar' ])
    transmute.update()

    import foobar.cli
    return foobar.cli.run()

It can also be used as a placeholder for a Python module. If the module itself is available from PyPI, the corresponding package name would be added to the requirements variable. For other use cases, I sense a pull request coming :-)

At the time of this writing, the only non-standard dependency of the script is the pkg_resources module from the setuptools package. The assumption here is that the module is more or less available everywhere. If this turns out to be a problem in practice, I suppose the script could be simplified to not require it.

Supported package formats

Currently, only standard Python eggs are supported. I don’t mind adding support for other formats, formats supported natively by Python are preferred.

In that regard, source tarballs look particularly interesting for pure Python packages, and seem to be more generally available from PyPI. Unpacking and importing the packages locally could be a way forward.

Python wheels also look interesting and gaining some traction.

Supported repositories

Local repositories

Just a directory with eggs in it. This is mostly useful for testing.

transmute.require([ 'foobar' ], sources=[ '/opt/basket' ])

PyPI

Transmute supports PyPI’s PyPIJSON interface.

transmute.require([ 'foobar' ], sources=[ transmute.PYPI_SOURCE ])

Amazon Simple Storage Service (S3)

Packages can be uploaded to a directory in S3.

While technically Amazon’s S3 doesn’t have the concept of a folder, the slash ('/') in S3 key names is abused to sustain the illusion of directories.

Credentials can be provided as environment variables. transmute recognizes a few fairly standard variables:

- AWS_CREDENTIAL_FILE
- AWS_ACCESS_KEY, AWS_SECRET_KEY, and (optionally) AWS_SECURITY_TOKEN
- AWS_DEFAULT_REGION and EC2_REGION

When running in an EC2 instance, transmute may also pick credentials from the IAM role associated with it.

import transmute.s3
transmute.require([ 'foobar' ], sources=[ 's3://bucket/key-prefix' ])

Missing a repository format?

I’m missing a pull request. :-)

Open issues

  • Logging is sorely missing. This can be helpful in debugging, but also to keep track of updates and possibly tie in to enabling rollbacks.

  • Rolling back a b0rked update.

  • Provide hooks for verifying an update before activating it.

  • We shouldn’t use the network on every run of a given command. Keeping track of metadata about repository queries would allow us to limit updates to daily or weekly schedules.

  • Currently MD5 hashes are used to verify integrity of downloaded packages, as advertised by repositories. It would be nice to be able to verify package signatures.

  • Your pet peeve?

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

transmute-3.tar.gz (13.9 kB view details)

Uploaded Source

Built Distribution

transmute-3-py2.7.egg (25.5 kB view details)

Uploaded Source

File details

Details for the file transmute-3.tar.gz.

File metadata

  • Download URL: transmute-3.tar.gz
  • Upload date:
  • Size: 13.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for transmute-3.tar.gz
Algorithm Hash digest
SHA256 41ff24a7064910ae37405e316d2c362bb7b88a42cefebc2c5986d17d067e290b
MD5 404399d3b5b9a643da2cf218d0c987f6
BLAKE2b-256 0327a467bbca02051dec8dbb0c6c31fe51a11760f6b9cfa528d2d516e34f5083

See more details on using hashes here.

File details

Details for the file transmute-3-py2.7.egg.

File metadata

  • Download URL: transmute-3-py2.7.egg
  • Upload date:
  • Size: 25.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for transmute-3-py2.7.egg
Algorithm Hash digest
SHA256 40f2c995e3a57949f8988848f33a58e87e26ea547681cf05bff20111c9a0703f
MD5 8f65ebc0b51ef58caa16a91aeb5041bb
BLAKE2b-256 63e8f3fdf08c96e6a4d18a90c7a8e5439df0af4afc607daebd5015334775a915

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page