Copy packages between python package repositories & more
A PY-thon RE-pository NE-twork tool
Python package repositories are nodes in a network.
Pyrene is a tool to transfer packages between repositories
- with an easy syntax: sources and targets are both defined as repo:package
- with package dependencies, thus the target will be self contained!
Pyrene is also an opinionated (read: simple, useful) configuration tool for pip.
As an application into a separate virtualenv using virtualenvwrapper:
mkvirtualenv app-pyrene pip install pyrene
creating this link is optional, but let’s us use pyrene outside its virtualenv:
ln -s ~/.virtualenvs/app-pyrene/bin/pyrene ~/bin
A simple network
The following simple network is intended to be a working reference setup for Pyrene.
Repositories are pypi, local, private as defined below:
is a pre-defined repo pointing to the public Python package repository at https://pypi.python.org/simple
It is used for publishing packages of public interest.
It is a volatile repo: project owners can DELETE published packages or even entire projects.
The obvious package names are mostly already taken.
is a pre-defined repo pointing to the local directory: ~/.pip/local
The user wants this to be the default repository for new package installs:
$ pyrene use local
This directory holds all packages needed for development, thus development can continue offline as well.
The developer has full control over the repository.
is accessible through a potentially restricted url https://packages.proprietary.com/simple
This repository holds all packages needed for deployments: both publicly available packages from pypi and in-house developed closed source ones.
The project/company has full control over the repo content.
It is not pre-defined, so define it:
$ pyrene ... Pyrene: http_repo private Pyrene[private]: set download_url=https://packages.proprietary.com/simple/ Pyrene[private]: set upload_url=https://packages.proprietary.com/pypi Pyrene[private]: set username=<me@company repo> Pyrene[private]: set password=<secret>
Here pyrene is used in shell mode, and Pyrene[...]: is a prompt for commands.
Import a public package for offline use
$ pyrene copy pypi:public-package local:
This also copies all the dependencies of public-package as well.
Publish an in-house package
$ cd in-house-pkg $ python setup.py sdist $ pyrene copy dist/in-house-pkg-1.0.0.tgz local: $ pyrene copy local:in-house-pkg private:
Deploy an application
# make pip use the `private` repository by default by [over]writing `~/.pip/pip.conf` $ pyrene use private $ mkvirtualenv app $ pip install application $ start-application ... # restore the dev-env $ deactivate $ rmvirtualenv app $ pyrene use local
This is an early release so if you have any problems (including usability), please tell them about at
shell with help, command completion and colors for configuration and interactive operations:
$ pyrene ... Pyrene: directory_repo local Pyrene[local]: set directory=/path/to/repo Pyrene[local]: copy pypi:someinterestingpkg==0.1 local: Pyrene[local]: work_on private Pyrene[private]: copy local:someinterestingpkg private: Pyrene[private]: help use ...
copy packages between repos (and directories):
$ pyrene copy source-repo:package-with-lots-of-dependencies destination-repo:
serve local directory repos over http (package sharing with coworkers?):
$ pyrene serve dir-repo
configure pip (by writing a ~/.pip/pip.conf file) to use a repo without --index-url or --find-links command line options:
$ pyrene use repo
The network configuration is persisted in the file ~/.pyrene (including passwords in plain text), which can be thought of as a combined ~/.pypirc and ~/.pip/pip.conf.
All operations are delegated to external tools (not to reinvent the eggs):
- downloading packages is delegated to pip
- uploading via http/https is delegated to twine
- serving a directory is delegated to pypiserver
Tools are heavily influenced to work in a certain way by
- giving them approriate command line options
- setting an environment variable
- writing a config file for them
all of them painful manually.
Repositories defined in ~/.pypirc are imported on first startup, and the use command replaces ~/.pip/pip.conf to make pip work without options.
- wheels are not downloaded anymore - local wheels can still be uploaded
- delegate http uploads to twine - simplifies code & license
- status command: show python packaging configuration status
- use command: makes backup of existing config before writing ~/.pip/pip.conf
- add known repos on first run:
- repositories from .pypirc (only sets upload-url)
- pypi as http://pypi.python.org
- local as ~/.pip/local
- new command: import_pypirc
- readline history
- show version on startup
- fix #1: existing package at remote http repo stops copy
- setup.cfg: fix keywords & classifiers, early release notice
- initial release - for testing