treep 1.38

managing git projects structured in tree in python

What it is

treep is a project manager, where "project" refers to a collection of git repositories. It aims to:

• cloning and performing batch git actions on all repositories contained in the workspace
• having an overview of the workspace (e.g. which repositories are ahead of master ?)
• helping compiling the workspace even if this latest comprises package supporting different compilation methods (e.g. cmake, pip, catkin)

To use treep you need:

• to install treep
• to create a folder called 'treep_XXX' (replace XXX by anything of your choices) containing at least:
  • a file repositories.yaml in which the URLs of repositories are given
  • a file projects.yaml which defines projects (i.e. grouping of repositories that can be cloned/managed together)

installation

Treep supports python2.7 and python 3

pip install treep

configuration

Example of repositories.yaml file:

my_robot:
	path: src/robots/
    origin: robot
    branch: master
    compilation: catkin
my_controller:
    path: src/controllers/
    origin: [e-mail]:my_name/my_controller.git
    compilation: cmake
my_logger:
    path: src/utils
    origin: [e-mail]:my_name/my_logger.git
    compilation: pip

• path : relative path in which the repositories will be cloned by treep (e.g. /src/robots will resolve to <absolute path to treep_XXX>/../workspace/src/robot)
• origin : the URL of the repository, or a URL prefix defined in configuration.yaml (e.g. "robot", explained later in this file)
• branch (optional) : branch to checkout. Master by default
• compilation (optional) : will be used by treep to generate compilation scripts (explained later in this file). This need a file configuration.yaml to be also present.

Example of projects.yaml file:

ROBOT:
    parent_projects: []
    repos: [my_robot,my_controller]
FULL_ROBOT
    parent_projects: [ROBOT]
    repos: [my_logger]

This file declare 2 projects:

• ROBOT (which contains my_robot and my controller)
• FULL_ROBOT (which contains my_robot, my_controller and my_logger)

Example of configuration.yaml file

The repositories.yaml file above uses 4 tags that requires definition: catkin, cmake, pip and robot. They are to be defined:

origin_prefixes:
    robot: '[e-mail]:my_account/'
compilation:
    catkin:
        default: {install=True}
    pip:
        default: {pip_executable: 'pip', user=True}
    cmake:
        default: {clean: True, multiprocess: True, cmake_suffix: ""}

• declaring "robot" as origin_prefixes allowed to declare the origin of "my_robot" in repositories.yaml as just "robot". Treep will resolve the url of "my_robot" to [e-mail]:my_account/my_robot.git
• compilation supports natively "catkin", "pip" and "cmake". Setting compilation tags in configuration.yaml and using them in repositories.yaml will allow treep to generate compilation scripts for the workspace, as described later in this document

Usage

Once treep has been installed and the treep configuration folder has been created (in this example in ~/Software):

# we assume treep_xxx folder is in ~/Software
cd ~/Software 
treep

will display all arguments supported by treep. We hope they are self explanatory.

For example, assuming that treep_xxx/projects.yaml defines a project "FULL_ROBOT":

treep --projects # will display list of known projects, including "FULL_ROBOT"
treep --clone FULL_ROBOT # will clone all repos of "FULL_ROBOT"
treep --status # will display state of the workspace

Autocompletion

treep uses argcomplete to provide autocompletion. More info here:

https://github.com/kislyuk/argcomplete

In short, to activate autocompletion, run in a terminal (may need sudo rights):

activate-global-python-argcomplete

Generating yaml configuration files

repositories.yaml and projects.yaml files can be generated automatically using the command treep_to_yaml :

• clone all the repositories of interest in a folder called "workspace"
• from the folder containing "workspace", call "yaml_to_treep"

For example:

mkdir ~/Software
cd ~/Software
mkdir workspace
cd workspace
mkdir src/
cd src
clone [e-mail]:vincentberenz/playful_kinematics.git
clone [e-mail]:vincentberenz/lightargs.git
cd ~/Software
treep_to_yaml 

Using several configuration folders

Treep supports using several configuration folders at once, which then need to be in the same folder. Treep will behave as if using a single configuration folder which defines the repositories and projects of both folders.

Generating a compilation script for the workspace

natively supported

In the example above, repositories.yaml was defining:

my_robot:
    path: src/robots/
    origin: robot
    branch: master
    compilation: catkin

and configuration.yaml:

compilation:
    catkin:
        default: {install=True}

If "my_robot" has been cloned in the workspace, this will result in :

treep --compilation-script

to generate a file compilation.sh which will call the suitable bash commands to build the workspace (in this case using catkin_make). If other repositories with other tags (e.g pip, cmake) have been cloned, compilation.sh will also contain the commands to compile them.

custom made

It is possible to use custom developed python scripts for generating compilation.sh. For this you need to create in treep_xxx a file compilation.py:

def my_script_generator(workspace_path,
                        package_name,
                        package_path,
                        arg1="1",
                        arg2="2"):

    # here write code for returning the bash script string that
    # will perform compilation of the code in workspace_path/package_path

To create such script, you may take for example inspiration from : https://git-amd.tuebingen.mpg.de/amd-clmc/treep/blob/master/treep/cmake.py

This file compilation.py allows the file configuration.yaml to define for example:

compilation:
    my_script_generator:
        default: {}
        my_gen: {arg1:"5"}

which then would allow in repositories.yaml:

my_robot:
    path: src/robots/
    origin: robot
    branch: master
    compilation: my_script_generator
my_controller:
    path: src/controllers/
    origin: [e-mail]:my_name/my_controller.git
    compilation: my_gen

Credit

Treep has been developed and is maintained by the Max Planck Institute for Intelligent Systems, Tuebingen, Germany

Versions changes

version 1.38

• compilation: generated script made executable
• compilation: catkin, cmake and pip supported without need of configuration.yaml file
• interactive generation of compilation script
• call to treep with undefined parent project and/or repositories not crashing

version 1.37

• added --fix-origins

version 1.36

• accepts several yaml files for repositories, projects and configurations in a single configuration folder

version 1.35

• removed documentation, redirect to pypi webpage instead
• fixed treep not working using python 2.7.6 because of unsupported usage of exec

version 1.34

• fixed typos in README
• added this log of changes in README
• fixed bug in checkout branch
• pulling with orange output when pulling worked but did not update the repo for any valid reason

version 1.33

• updated README.md and fixed setup.py to use it for project description

version 1.32

• fixed bug in status, does not attempt to create status from non git repo anymore
• cloning now also cloning submodules
• fixed display of status and pull of repos (better tabulation)
• pulling, fetching and cloning now multiprocess
• added compilation script generation

version 1.31

• process calls: transforming output to unicode for support of python3
• files: yaml loader compatible both newer and older versions of pyyaml
• status : display of status message improved for python3
• status : does not fail anymore on detached head

version 1.3