A zc.buildout extension to ease the development of large projects with lots of packages.
Project description
Introduction
Let Mr. Developer help you win the everlasting buildout battle!
(Remixed by Matt Hamilton, original from http://xkcd.com/303)
mr.developer is a zc.buildout extension that makes it easy to work with buildouts containing lots of packages, of which you only want to develop some. The basic idea comes from Wichert Akkerman’s plonenext effort.
Usage
Add mr.developer to the extensions entry in your [buildout] section:
[buildout] extensions = mr.developer
This enables additional [buildout] options:
- sources
This specifies the name of a section which lists the repository information for your packages. Defaults to sources.
- sources-dir
This specifies the default directory where your development packages will be placed. Defaults to src.
- auto-checkout
This specifies the names of packages which should be checked out during buildout. Packages already checked out are skipped. You can use * as a wildcard for all packages in sources.
- always-checkout
This defaults to false. If it’s true, then all packages specified by auto-checkout and currently in develop mode are updated during each buildout run. If set to force, then packages are updated even when they are dirty instead of asking interactively.
- always-accept-server-certificate
This defaults to false. If it’s true, invalid server certificates are accepted without asking (for subversion repositories).
The format of entries in the [sources] section is:
[sources] name = kind url [key=value ...]
Where individual parts are:
- name
The package name.
- kind
The kind of repository. Currently supported are svn, hg, git, bzr, darcs, cvs, or fs.
- url
The location of the repository. This value is specific to the version control system used.
- key=value
You can add options for each individual package with this. No whitespace is allowed in key, value, and around the equal sign. For a description of the options see below.
The per-package options are:
- Common options
The path option allows you to set the base directory where the package will be checked out. The name of the package will be appended to the base path. If path is not set, sources-dir is used.
With full-path you can set the directory where the package will be checked out. This is the actual destination, nothing will be added.
The update option allows you to specify whether a package will be updated during buildout or not. If it’s true, then it will always be updated. If it’s false, then it will never be updated, even if the global always-checkout option is set.
The egg option makes it possible to manage packages which are not eggs with egg=false. All commands like update work as expected, but the package isn’t added to the develop buildout option and the activate and deactivate commands skip the package.
- svn
The url is one of the urls supported by subversion.
You can specify a url with a revision pin, like http://example.com/trunk@123.
You can also set the rev or revision option, which is either a pin like with rev=123 or a minimum revision like rev=>123 or rev=>=123. When you set a minimum revision, the repository is updated when the current revision is lower.
- git
The branch option allows you to use a specific branch instead of master.
The rev option allows you to use a specific revision (usually a tag) instead of the HEAD.
The pushurl options allows you to explicitly separate push url from pull url, configured by git config.
Note that the branch and rev option are mutually exclusive.
- hg
The branch option allows you to use a specific branch instead of default.
The rev option allows you to force a specific revision (hash, tag, branch) to be checked out after buildout
- bzr
Currently no additional options.
- darcs
Currently no additional options.
- cvs
The cvs_root option can be used to override the setting of the $CVSROOT environment variable. The tag option forces checkout/update of the given tag instead of CVS HEAD.
- fs
This allows you to add packages on the filesystem without a version control system, or with an unsupported one. You can activate and deactivate packages, but you don’t get status info and can’t update etc.
The url needs to be the same as the name of the package.
Here’s an example of how your buildout.cfg may look like:
[buildout] extensions = mr.developer auto-checkout = my.package [sources] my.package = svn http://example.com/svn/my.package/trunk update=true some.other.package = git git://example.com/git/some.other.package.git
When you run buildout, the script bin/develop is created in your buildout directory. With this script you can perform various actions on packages, like checking out their source code, without the need to know where the repositories are located.
For help on what the script can do, run bin/develop help.
If you checked out the source code of a package, you must run buildout again. The new package will then be marked as a development egg and have its version pin cleared (if any). You can control the list of development eggs explicitely with the activate and deactivate commands.
Troubleshooting
Dirty SVN
You get an error like:
ERROR: Can't switch package 'foo' to 'https://example.com/svn/foo/trunk/' because it's dirty.
If you have not modified the package files under src/foo, then you can check what’s going on with status -v. One common cause is a *.egg-info folder which gets generated every time you run buildout and this shows up as an untracked item in svn status.
You should add .egg-info to your global Subversion ignores in ~/.subversion/config, like this:
global-ignores = *.o *.lo *.la *.al .libs *.so *.so.[0-9]* *.a *.pyc *.pyo *.rej *~ #*# .#* .*.swp .DS_Store *.egg-info
HTTPS Certificates
The best way to handle https certificates at the moment, is to accept them permanently when checking out the source manually.
Mercurial reports mismatching URL
This happens if you use lp:// URLs from launchpad. The problem is, that hg reports the actual URL, not the lp shortcut.
Commands
The following is a list of all bin/develop commands:
activate (a)
usage: develop activate [-h] [-a] [-c] [-d] package-regexp [package-regexp ...] Add packages to the list of development packages. positional arguments: package-regexp A regular expression to match package names. optional arguments: -h, --help show this help message and exit -a, --auto-checkout Only considers packages declared by auto-checkout. If you don't specify a <package-regexps> then all declared packages are processed. -c, --checked-out Only considers packages currently checked out. If you don't specify a <package-regexps> then all checked out packages are processed. -d, --develop Only considers packages currently in development mode. If you don't specify a <package-regexps> then all develop packages are processed.
checkout (co)
usage: develop checkout [-h] [-a] [-v] package-regexp [package-regexp ...] Make a checkout of the packages matching the regular expressions and add them to the list of development packages. positional arguments: package-regexp A regular expression to match package names. optional arguments: -h, --help show this help message and exit -a, --auto-checkout Only considers packages declared by auto-checkout. If you don't specify a <package-regexps> then all declared packages are processed. -v, --verbose Show output of VCS command.
deactivate (d)
usage: develop deactivate [-h] [-a] [-c] [-d] package-regexp [package-regexp ...] Remove packages from the list of development packages. positional arguments: package-regexp A regular expression to match package names. optional arguments: -h, --help show this help message and exit -a, --auto-checkout Only considers packages declared by auto-checkout. If you don't specify a <package-regexps> then all declared packages are processed. -c, --checked-out Only considers packages currently checked out. If you don't specify a <package-regexps> then all checked out packages are processed. -d, --develop Only considers packages currently in development mode. If you don't specify a <package-regexps> then all develop packages are processed.
help (h)
usage: develop help [-h] [--rst] [command] Show help on the given command or about the whole script if none given. positional arguments: command The command you want to see the help of. optional arguments: -h, --help show this help message and exit --rst Print help for all commands in reStructuredText format.
info
usage: develop info [-h] [-a] [-c] [-d] [--name] [-p] [--type] [--url] [package-regexp [package-regexp ...]] Lists informations about packages. positional arguments: package-regexp A regular expression to match package names. optional arguments: -h, --help show this help message and exit -a, --auto-checkout Only considers packages declared by auto-checkout. If you don't specify a <package-regexps> then all declared packages are processed. -c, --checked-out Only considers packages currently checked out. If you don't specify a <package-regexps> then all declared packages are processed. -d, --develop Only considers packages currently in development mode. If you don't specify a <package-regexps> then all declared packages are processed. Output options: The following options are used to print just the info you want, the order they are specified reflects the order in which the information will be printed. --name Prints the name of the package. -p, --path Prints the absolute path of the package. --type Prints the repository type of the package. --url Prints the URL of the package.
list (ls)
usage: develop list [-h] [-a] [-c] [-d] [-l] [-s] [package-regexp [package-regexp ...]] Lists tracked packages. positional arguments: package-regexp A regular expression to match package names. optional arguments: -h, --help show this help message and exit -a, --auto-checkout Only show packages in auto-checkout list. -c, --checked-out Only considers packages currently checked out. If you don't specify a <package-regexps> then all checked out packages are processed. -d, --develop Only considers packages currently in development mode. If you don't specify a <package-regexps> then all develop packages are processed. -l, --long Show URL and kind of package. -s, --status Show checkout status. The first column in the output shows the checkout status: '#' available for checkout ' ' in auto-checkout list and checked out '~' not in auto-checkout list, but checked out '!' in auto-checkout list, but not checked out 'C' the repository URL doesn't match
purge
usage: develop purge [-h] [-n] [package-regexp [package-regexp ...]] Remove checked out packages which aren't active anymore. Only 'svn' packages can be purged, because other repositories may contain unrecoverable files even when not marked as 'dirty'. positional arguments: package-regexp A regular expression to match package names. optional arguments: -h, --help show this help message and exit -n, --dry-run Don't actually remove anything, just print the paths which would be removed.
rebuild (rb)
usage: develop rebuild [-h] [-n] Run buildout with the last used arguments. optional arguments: -h, --help show this help message and exit -n, --dry-run Don't actually run buildout, just show the last used arguments.
reset
usage: develop reset [-h] [-a] [-c] [-d] [package-regexp [package-regexp ...]] Resets the packages develop status. This is useful when switching to a new buildout configuration. positional arguments: package-regexp A regular expression to match package names. optional arguments: -h, --help show this help message and exit -a, --auto-checkout Only considers packages declared by auto-checkout. If you don't specify a <package-regexps> then all declared packages are processed. -c, --checked-out Only considers packages currently checked out. If you don't specify a <package-regexps> then all checked out packages are processed. -d, --develop Only considers packages currently in development mode. If you don't specify a <package-regexps> then all develop packages are processed.
status (stat, st)
usage: develop status [-h] [-a] [-c] [-d] [-v] [package-regexp [package-regexp ...]] Shows the status of tracked packages, filtered if <package-regexps> is given. The first column in the output shows the checkout status: ' ' in auto-checkout list '~' not in auto-checkout list '!' in auto-checkout list, but not checked out 'C' the repository URL doesn't match The second column shows the working copy status: ' ' no changes 'M' local modifications or untracked files The third column shows the development status: ' ' activated '-' deactivated '!' deactivated, but the package is in the auto-checkout list 'A' activated, but not in list of development packages (run buildout) 'D' deactivated, but still in list of development packages (run buildout) positional arguments: package-regexp A regular expression to match package names. optional arguments: -h, --help show this help message and exit -a, --auto-checkout Only considers packages declared by auto-checkout. If you don't specify a <package-regexps> then all declared packages are processed. -c, --checked-out Only considers packages currently checked out. If you don't specify a <package-regexps> then all checked out packages are processed. -d, --develop Only considers packages currently in development mode. If you don't specify a <package-regexps> then all develop packages are processed. -v, --verbose Show output of VCS command.
update (up)
usage: develop update [-h] [-a] [-d] [-f] [-v] [package-regexp [package-regexp ...]] Updates all known packages currently checked out. positional arguments: package-regexp A regular expression to match package names. optional arguments: -h, --help show this help message and exit -a, --auto-checkout Only considers packages declared by auto-checkout. If you don't specify a <package-regexps> then all declared packages are processed. -d, --develop Only considers packages currently in development mode. If you don't specify a <package-regexps> then all develop packages are processed. -f, --force Force update even if the working copy is dirty. -v, --verbose Show output of VCS command.
Changelog
1.20 - 2012-02-26
Git: Added pushurl option [iElectric (Domen Kožar)]
Refactored thread locking. [shywolf9982]
Refactored search for git executable and version handling. [shywolf9982]
In the status command report unknown packages with ‘?’ when no package-regexp has been given. [maurits]
Added –force option to purge command. Especially helpful in purging non-subversion packages, which otherwise we refuse to remove. Fixes issue #71. [maurits]
Do not depend on elementtree if there is xml.etree (Python >= 2.5).
Don’t set locale anymore when calling svn. This may break if the output is localized instead of english, I couldn’t reproduce that anymore though. [fschulze, rochecompaan (Roché Compaan)]
Fix compatibility with mercurial v2.1 [janjaapdriessen (Jan-Jaap Driessen)].
1.19 - 2011-09-22
Git: Don’t stop buildout after renaming/adding git remotes, i.e. when actively working on a given package. [witsch (Andreas Zeidler)]
Bugfix: Honhour buildout:develop parameters even if ending with slash. [lukenowak]
Installation: Check presence of required modules instead of relying on version of python. [lukenowak (Łukasz Nowak)]
1.18 - 2011-08-16
Mercurial: Added support for branches. [posborne (Paul Osborne)]
Git: Added support for the Windows msysGit. [canassa (Cesar Canassa)]
Git: Added rev option that allows you to use a specific revision (usually a tag) instead of the HEAD. [maurits (Maurits van Rees)]
1.17 - 2011-01-26
Git: Default to branch master if no branch is given in the source. [stefan]
Brush up the README. [stefan]
Create the sources-dir if it is not present. [janjaapdriessen]
Only require argparse with Python < 2.7. [dobe]
Fixed issue #35 using bzr, similar to the fix #28 for hg in last version. [menesis]
Pass branch to bzr pull. [menesis]
Add support for darcs. [lelit, azazel]
1.16 - 2010-09-16
Fix NameError: global name 'source' is not defined when using gitsvn and running status command. [markvl]
Add handling of new -s command line option of zc.buildout 1.5, this fixes issue #29. [fschulze]
Don’t pass the PYTHONPATH onwards to mercurial, this fixes issue #28 [fschulze, Christian Zagrodnick]
Fix saving buildout options on Windows. Issue #24 [fschulze]
Only warn if the svn version is too old. [fschulze]
1.15 - 2010-07-25
Use always-checkout option from buildout config for update command. This fixes issue #18. [fschulze]
Fix OSError: [Errno 10] No child processes errors in Python 2.4 and 2.5. (Issue #12) [fschulze]
Fix CVS update. [sargo]
1.14 - 2010-05-15
Added bzr support. [tseaver]
Added git branch support. [shywolf9982, fschulze]
1.13 - 2010-04-11
Tell the user which packages are queued for update or checkout, so one can check which packages are still updating now that the output is only printed after everything is done due to parallel checkouts. [fschulze]
Added always-accept-server-certificate option. When set in the [buildout] section, then invalid certificates are always accepted for subversion repositories. [fschulze]
Added -v/--version option. [tomster, fschulze]
Use the much nicer argparse library instead of optparse. [fschulze]
1.12 - 2010-03-15
Fix svn checkout. [fschulze]
1.11 - 2010-03-14
Handle untrusted server certificates by asking the user what to do. [fschulze]
Properly handle user input for authorization by using locks to prevent problems with parallel checkouts. [fschulze]
Only checkout/update packages in auto-checkout or with update = true option when running buildout. [fschulze]
1.10 - 2010-02-06
Don’t store the buildout options if they contain a command. [fschulze]
Basic support for buildout offline mode (-o). Not all cases are handled yet. [fschulze]
Added full-path package option. [fschulze]
Added egg package option (Issue #6). [fschulze]
By setting always-checkout = force, all packages will be updated without asking when dirty. [fschulze]
The [path] part of sources is replaced by path=PATH syntax and throws a warning when used. [fschulze]
Per package options are now allowed before the URL. [fschulze]
Check svn version and output helpful error messages if it’s too old or can’t be determined (Issue #13). [fschulze]
Error messages instead of tracebacks when source definitions are wrong. [fschulze]
Fix checkout of packages (Issues #9 and #11). [fschulze]
Possibility to checkout/update tags instead of HEAD in CVS [sargo]
Tests for CVS integration [sargo]
Better checking of CVS package purity. [sargo]
1.9 - 2010-01-11
Added dry-run option to purge command. [fschulze]
Fix purging on windows. [kleist]
Fix compatibility with Python < 2.6. [fschulze, vincentfretin]
Fix all answer for update command. [fschulze]
1.8 - 2010-01-10
Added threading for parallel checkouts. [fschulze, jensens]
Ask whether to update dirty packages during checkout. [fschulze]
When you answered yes when asked whether to update a dirty package, then all further questions had been answered with yes as well, this is now fixed. [fschulze]
Added all option when asked to update dirty packages. [fschulze]
Added help for all commands to PyPI description. [fschulze]
Added option to help command which outputs the help for all commands in reStructuredText format. [fschulze]
Don’t abort after user answered no on whether to update a package, just skip that package. [fschulze]
1.7 - 2009-11-26
Fix a problem where a package wasn’t added to the develop packages on auto checkout. [fschulze]
1.6 - 2009-11-21
Filter the packages gathered from buildout:develop to ones declared in sources, otherwise things like “develop = .” break. [fschulze]
Added support for Concurrent Versions System (CVS). [sargo]
1.5 - 2009-11-19
Added global always-checkout and a per source update option. [fschulze]
Added purge command. [fschulze]
Ask user how to proceed when package is dirty. [fschulze]
Refactored package matching and made the command options consistent. Now you can update only the packages currently in development with -d and similar possibilities. [fschulze]
Fix duplicate logging output. [fschulze]
Fix parsing of buildout arguments when -t was used. [fschulze]
1.4 - 2009-11-16
Allow to set a minimal revision for svn repositories. If the current revision is lower, then the package is updated. [fschulze]
1.3 - 2009-11-15
Read the cfg used by last buildout run. This prevents unexpected behaviour, if you change mr.developer options like source declarations and don’t run buildout. Such changes are now picked up immediately. [fschulze]
Added tests and a buildout to run them easily. [fschulze]
1.2 - 2009-11-12
If a package is removed from auto-checkout and wasn’t explicitly activated, then it will be removed from the develop packages automatically. In existing buildouts with an older mr.developer, you have to reset the packages first. [fschulze]
Added * wild card support for auto-checkout. [fschulze]
Don’t bail on subversion URLs ending in a slash or a revision marker. [fschulze]
Removed old way of specifying sources with sources-svn and sources-git. [fschulze]
Exit immediately when there are issues during checkout when running as extension. [fschulze]
Use verbosity from buildout when running as extension. [fschulze]
Fix buildout_dir in develop script, so it is properly escaped on Windows. [fschulze]
Changed the output of list -s to match the one from status. [fschulze]
Added troubleshooting section to readme. [miohtama, fschulze]
All commands have a -h and --help option now to show their help. [fschulze]
1.1 - 2009-08-07
Use relative paths from the buildout directory for the develop option if possible. This fixes issues if your buildout path contains a space. [fschulze]
Warn when trying to activate or deactivate a package which isn’t checked out. [fschulze]
Don’t depend on elementree on Python >= 2.5, because it’s builtin there. [fschulze]
When checking out a source it will automatically be activated. [fschulze]
Use ‘sources’ as the default section name for source information. [fschulze]
Added support for filesystem packages without version control with the ‘fs’ type. [fschulze]
1.0.1 - 2009-05-05
Fixed case sensitivity of package names for several commands. [fschulze]
SVN externals no longer cause a modified status. [fschulze]
1.0 - 2009-05-02
Added info command to print various informations about packages. [fschulze]
Added reset command to reset the develop state of packages. This is useful when switching to a new buildout configuration. During the next buildout run the develop state is determined the same way as in a clean buildout. [fschulze]
Got rid of deprecation warning in Python 2.6 by removing unnecessary call of __init__ in working copy implementations. [fschulze]
0.15 - 2009-04-17
Added reminder to run buildout after activating or deactivating packages. [fschulze]
Added rebuild command to rerun buildout with the last used arguments. [fschulze]
0.14 - 2009-04-16
Fixed verbose output of checkout command. [fschulze]
Added -f option to update command to force updates even if the working copy is dirty. [fschulze]
0.13 - 2009-04-14
Added -a option to update command to only update the packages declared in the auto-checkout list. [fschulze]
Added activate and deactivate commands. This allows to select which packages are added to the develop option of zc.buildout. Enhanced the status command to show the additional informations. [fschulze]
Switched the meaning of ~ and C in status command. [fschulze]
0.12 - 2009-04-14
Added support for Mercurial (hg). [mj]
Refactored working copy logic, so it’s easier to add support for other version control systems. [fschulze]
Added verbose flag to checkout and update commands. [fschulze]
0.11 - 2009-04-06
Removed the nice os.path.relpath usage, because it’s only been introduced with Python 2.6. [fschulze]
0.10 - 2009-04-06
Added verbose flag to status command. [fschulze]
Deprecated sources-svn and sources-git in favour of just sources which allows more flexibility. [fschulze]
Changed status command to only check known paths and not the whole sources-dir path. [fschulze]
Add possibility to filter packages in update and status commands. [fschulze]
Tell the user at the end of the buildout run whether there have been any errors during automatic checkout. [fschulze]
Install the develop script as the first part instead of the last, so it can be used to fix problems. [fschulze]
0.9 - 2009-03-30
When installed as just an egg, then the resulting develop script can be called from anywhere and will try to find the correct develop script from the current working directory and execute it. [fschulze]
Fixed help text formatting. [fschulze]
0.8 - 2009-03-25
Added authentication support to subversion commands. [fschulze]
Added -a option to checkout command to limit matching to the packages declared by the auto-checkout option. If no further argument is given, then all the packages from auto-checkout are processed. [fschulze]
0.7 - 2009-03-24
Added update command to checkout script. [fschulze]
Added status command to checkout script. [fschulze]
Added status flag to list command to indicate packages with wrong URL. [fschulze]
If the working copy is clean, then checkout automatically switches svn packages. [fschulze]
Skip on checkout errors during buildout, so the develop script is generated and you get a chance to inspect and fix the problems. [fschulze]
Check remote url and stop checkout if it differs. [fschulze]
Added various options to the list command. [fschulze]
Instead of the checkout script, there is now a develop script with various commands. [fschulze]
0.6 - 2009-03-24
Added custom logging formatter for nicer output in the checkout script. [fschulze]
Removed the ‘-e’ option, regular expression matching is the default now. [fschulze]
Made it possible to specify more than one regular expression without the need to use a pipe symbol and quotes. [fschulze]
Added help text for the checkout script to pypi page. [fschulze]
Add a warning to rerun buildout after checkout. [fschulze]
0.5 - 2009-03-23
Make sure that the updated versions are actually used. [fschulze]
0.4 - 2009-03-22
Fixed logging, which fixes the info message output. [fschulze]
Skip checkout of existing packages. [fschulze]
0.3 - 2009-03-22
Fixed source distribution by adding a MANIFEST.in. [fschulze]
Added -e and -l options to checkout script. [fschulze]
0.2 - 2009-03-22
Added auto-checkout option (only works with subversion at the moment). [fschulze]
Added support for git. [fschulze]
Throw error when trying to checkout unknown package. [fschulze]
Fixed target directory for checkouts. [fschulze]
0.1 - 2009-03-19
Initial release
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.