fdbuild 0.3.0

CLI tool to fulfill tasks on single projects or vast project hierarchies

FDBuild

Configurable and extensible CLI tool to fulfill tasks sequentially on a list of projects or vast project hierarchies. Most commonly used for compiling many projects at once.

Features

Define Projects

• Define projects in a hierarchy of directories. Settings are inherited.
• Pick plugins to work on these projects.
• Simple YAML syntax allows to easily define projects and share settings. For example a toplevel settings file

  projects:
    - drm
    - kwinft
  source:
    plugin: git
  build:
    plugin: ninja
    threads: max
  

  with the specific project settings file drm/fdbuild.yaml:

  source:
    branch: main
    origin: https://gitlab.freedesktop.org/mesa/drm.git
  configure:
    plugin: meson
  install:
    path: /opt/drm
    sudo: false
  

• Customize settings in an fdbuild.yaml file next to each project and subtree.

Convenience

• Work all projects with a single command: fdbuild
• Depending on your working directory work a subtree with the very same command.
• Bootstrap a single project or whole hierarchies of projects via templates: fdbuild --init-with-template drm

Extensibility

• Several templates of popular open source projects are included.
• Work plugins for building most C and C++ software are included.
• Additional templates and work plugins can be installed as Python packages that are automatically discovered.

Getting Started

• Get the latest release via: pip install fdbuild.
• Create a project from a template: fdbuild --init-with-template linux-desktop-meta.
• Change into the new directory and run fdbuild.

Advanced Usage

Step Selection

You may skip steps in projects by the --no flag or do subset of steps with the --only flag.

Example

With the steps source, configure, build, install the following two commands are equivalent:

• fdbuild --no source,build
• fdbuild --only configure,install

Settings Override

Up to four key-value pairs may be overridden on invokation. This overrides the value for any project independent of its position in the projects hierarchy.

Example

In the above example we may override the install path and sudo usage as well as set the clear flag to do a clean build with:

fdbuild --set install:path:/usr/local --set1 install:sudo:true --set2 build:clear:true

Settings Hierarchies

Without settings override through command line a subproject always adheres to the nearest available settings value in an fdbuild.yaml file of its hierarchy.

Example

Take the following project tree:

PRJ_A
│   fdbuild.yaml
│
└───PRJ_B
|   │   fdbuild.yaml
|   └───PRJ_C
|   │       fdbuild.yaml
|   └───PRJ_D
|           fdbuild.yaml
│
└───PRJ_E
        fdbuild.yaml

To query a specific settings value for project PRJ_C FDBuild will first look in PRJ_A/PRJ_B/PRJ_C/fdbuild.yaml. If this settings file defines the value FDBuild will use it on project PRJ_C.

Otherwise FDBuild will look into PRJ_A/PRJ_B/fdbuild.yaml. And if this settings file also does not provide the value FDBuild will look into PRJ_A/fdbuild.yaml.

Likewise, for a settings value of project PRJ_D, FDBuild will first check its settings file PRJ_A/PRJ_B/PRJ_D/fdbuild.yaml, next the settings file PRJ_A/PRJ_B/fdbuild.yaml and last PRJ_A/fdbuild.yaml.

For project PRJ_E FDBuild will first look into PRJ_A/PRJ_E/fdbuild.yaml, and if the value is not found there look into PRJ_A/fdbuild.yaml.

If FDBuild is run in some directory the hierarchy unfolds recursively according to the projects lists in the working directory and all adjacent directories. For example assume that in the above tree FDBuild is run from directory PRJ_B, then the settings file of PRJ_A will be considered if and only if PRJ_B is listed in the projects entry of settings file PRJ_A/fdbuild.yaml.

Assume further that the parent directory of PRJ_A does not feature a FDBuild settings file listing PRJ_A as one of its projects. Then the lookup will stop here and PRJ_A becomes the toplevel directory of this project hierarchy defined through the settings files.

Settings References

Besides common YAML syntax you can use the following syntax for referencing other entries in the same settings file or in another settings file within the project hierarchy: @{path_to_other_entry}. If you need to use the sequence @{ without declaring a reference you have to write @@{.

The path to the other reference follows a certain syntax as well: a section or entry in a settings file is denoted by one leading $, project hierarchies are divided by /.

Example

• $section$entry - take value from same settings file at labels section: entry.
• prjA/prjB$entry - take value from label entry in settings file of project prjA/prjB relative to the settings file of the reference.
• /prjC/prjD$entry - take value from label entry in the settings file of project prjC/prjD, whereas prjC is a subproject at the toplevel of the project hierarchy.

Work Dependencies

Usually projects are being worked on in the order defined by the projects hierarchy and their position in the projects key.

This can be bypassed by setting dependencies between projects via the depends key in the fdbuild.yaml file of the project that should only be worked on after another one.

Example:

In the above example writing additionally

depends:
  - kwinft

to the drm/fdbuild.yaml file ensures that the "kwinft" project is processed before the "drm" project.

Extensions

With templates, stages, steps, and structurizers, FDBuild can adapt to any work flow. For users this means just installing another Python package.

For developers this means hooking into FDBuild's defined entry points and making their set of templates or plugin a Python package.

Templates

FDBuild can be made aware of additional templates by defining a paths() function for the entry point group fdbuild.templates.

The paths() function should then return a list of paths, where the template files of the package are stored.

Example

The official template collection of FDBuild provides:

• an entry point hooking up
• a paths() function returning a list with the path to
• a directory containing the templates

Plugins

Stage, step and structurizer plugins allow to extend FDBuild in order to organize work for any project imaginable.

Stages

A stage provides an abstract view on a task.

Defining a custom stage means finding a unique name describing its purpose and then creating for it an entry hook in the fdbuild.stages group.

The entry point must be a class inheriting fdbuild.core.stage.Stage. Its constructor should take an object of type fdbuild.core.step.Step as argument and forward it to the super constructor. The abstract functionality of the stage can then be declared at will as member functions of the class.

Steps

A step implements a stage. They may access the abstract functionality of a certain stage to conduct specific operations to fulfil the task at hand.

Defining a custom step means finding a unique name and then creating for it an entry hook in the fdbuild.steps group.

The entry point must be a class with its constructor taking an object of type fdbuild.core.step.Step as argument. The class must provide a work() member function that is then called by FDBuild to work on the stage.

Structurizers

A structurizer allows creating and updating complex project structures before the actual work starts on them.

Defining a custom structurizer means finding a unique name and then creating for it an entry hook in the fdbuild.structurizers group.

The entry point must be a class with its constructor taking an object of type fdbuild.core.structurizer.Structurizer as argument. The class must provide a structure(..) member function that takes an object of type fdbuild.core.log.StructureLog as argument and returns a map that describes all settings files of projects being structurized.

Status

This software has been used actively in production already for several years but without a proper release. Now we first release an unstable version to gather feedback and in particular stabilize the extensions API. Long-term the goal is a 1.0 release with stable settings syntax and extension API. But until then both may still change in backwards incompatible ways. Such changes will be listed in the changelog though.

Development

There are a number of open feature tickets. If you are interested in working on one, comment in the ticket or contact us in our Gitter/Matrix room.