pymoku-parts 0.12

Another build tool

Installation

Using Pip

pip install parts

Or using the self-contained executable

./parts

Introduction

Parts is a build dependency and execution tool heavily inspired by SCons, bitbake, GitLab-CI among others.

The motivation for parts was the scale of a particular project being managed which had components larger and more independent than a typical SCons or Makefile targets; but not requireing the external package and configuration management provided by bitbake. These existing solution either felt awkward or overkill.

Parts intends to provide a simple, lightweight dependency tree API and runtime to build these medium sized projects. The command line interface and interation will feel very familiar.

Example

Assuming a folder structure

./parts.def
./module/parts.def

# parts.def
mod = include('module')

default(mod['library'])  # use the instance of Module created in module/parts.def

# module/parts.def
import asyncio
from parts import BuildStep


class Module(BuildStep):
    async def do_step(self):
        self.update_progress(total=100, phase='running')
        self.log.info(f'Running Module: {self.name}')
        for i in range(100):
            self.update_progress(advance=1, phase=f'loop {i:d}')
            await asyncio.sleep(0.1)  # do some work
        self.log.info(f'Completed Module: {self.name}')


Module('library')  # automatically registers target with name 'library'

Running

parts --no-cache

will result in

[11:29:56] INFO     Building ['default']...                                parts
           INFO     Running Module: library                              library
[11:30:06] INFO     Completed Module: library                            library
╭──────────────────────────────────────────────────────╮
│ 0:00:10 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 │
╰──────────────────────────────────────────────────────╯

           INFO     Complete                                               parts

Design Goals

• parts.def use Python syntax with some shortcuts
• File paths are relative to the parts.def that references them
• BuildStep is the unit of work used by parts
• (sub)projects define all their BuildStep objects in a parts.def
• Build objects are useable by include()'ing another parts.def
• Build objects can be reproduced out of context if their input/output artefacts match (to enable distribution over multiple hosts)
• One parts.def can reference object instances created from an included parts.def
• BuildSteps can be used to communicate configuration options to depenent steps

Running parts

From within the toplevel of a project directory, next to a parts.def

parts [targets]

or by specifying the parts.def using --path

parts --path path/to/parts.def

Multiple targets can be specified simultaneously. If no target is provided then 'default' will be run, assuming it was provided by a call the default().

Listing available targets

parts --list

Graphing project dependencies

If graphviz is installed, parts can produce a graph for specific build targets to help visualize project dependencies.

parts --graph target

parts.def Reference

include()

# TODO

source()

# TODO

alias()

# TODO

copy()

# TODO

link()

# TODO

publish()

This will create a BuildStep that copies the given dependencies to the global ouput folder.

publish('a', 'b', BuildStep('c'))

will copy the outputs of task 'a', 'b' and the newly created 'c' to ./output.

alias('all', publish('a', 'b', 'c')

Will create a build target 'all' that publishes the outputs of 'a', 'b' and 'c'.

publish('a', dest='module_a')

Will publish ouputs of 'a' to the ./output/module_a folder.

arg()

# TODO

default()

# TODO

BuildStep

A unit of work in parts is represented by the abstract BuildStep which provides dependency linking with other BuildSteps as well as informative scheduling and execution of each task.

# TODO

Dependencies

# TODO

ProcessStep

# TODO

Elaborate vs Execution

Parts will elaborate all included config files to resolve their dependecies before beginning any execution.

This implies the inputs and outputs for each step must be defined during elaboration and will not change during execution.

While the execution of each step is parallelised (co-routines, though steps will likely produce their own processes), elaboration is serialized and not thread-safe.

Logging and Progress Bars

# TODO

ParseNodes

# TODO

State tracking

# TODO

phony targets

# TODO

parts.toml

Project-level configuration lives in parts.toml. All matching files are merged, with later locations overriding earlier ones section by section:

1. ~/.config/parts/runners.toml — legacy name, still accepted
2. ~/.config/parts/parts.toml — user-global
3. ./parts.runners.toml — legacy name, still accepted
4. ./parts.toml — project-local, takes precedence

So a user-global [runners.X] and a project-local [runners.Y] coexist, while a project-local [runners.X] overrides the user-global one.

The file currently has two sections:

• [parts] — overrides for the --build and --output CLI defaults. CLI flags still win when supplied.

  [parts]
  build  = "~/PyMoku-shared/build"
  output = "~/PyMoku-shared/output"
  

• [runners.<name>] — remote-runner rules (see below).

Remote Runners

By default, all ProcessStep commands run locally. Runners allow steps to be redirected to remote execution backends — SSH hosts, Docker containers, or cloud VMs — without changing parts.def files.

Each runner has a match pattern that selects which BuildStep classes it handles, and a type that determines the execution backend.

[runners.my-runner]
match = "VivadoStep|SynthStep"   # fnmatch patterns, pipe-separated
type = "ssh"                      # local, ssh, pool, docker, cloud

Match patterns are tested against each class in the step's MRO (so matching VivadoStep also catches its subclasses). First matching rule wins. Unmatched steps run locally.

Runner Types

local

Runs commands via local subprocess. This is the default when no runner matches.

ssh

Executes on a remote host via SSH. Input files are rsynced to the remote host, the command runs there, and declared output files are rsynced back. Absolute paths in the command are rewritten to match the remote mirror.

[runners.build-server]
match = "VivadoStep"
type = "ssh"
host = "build.example.com"
user = "builder"
key = "~/.ssh/id_ed25519"
workdir = "~/parts-builds"
setup = "source /opt/tools/settings.sh"  # run before each command
jobs = 8                                  # max concurrent jobs

[runners.build-server.env]
LICENSE_FILE = "/opt/licenses/license.lic"

pool

Distributes jobs across multiple SSH hosts. Each host has a concurrency limit; jobs are dispatched to the first host with a free slot.

[runners.farm]
match = "VivadoStep"
type = "pool"
setup = "source /opt/tools/settings.sh"

[[runners.farm.hosts]]
host = "build1.example.com"
user = "builder"
key = "~/.ssh/id_ed25519"
workdir = "~/parts-builds"
jobs = 4

[[runners.farm.hosts]]
host = "build2.example.com"
user = "builder"
key = "~/.ssh/id_ed25519"
workdir = "~/parts-builds"
jobs = 8

docker

Runs commands inside a local Docker container.

[runners.sim]
match = "CocoTB*"
type = "docker"
image = "cocotb:latest"

Custom runner types

External packages can register their own runner types via the parts.runners entry point group, or by calling register_runner_type() at import time.

Entry point (pyproject.toml):

[project.entry-points."parts.runners"]
myrunner = "mypackage.runner:create"

Where create(name, config) is a factory that returns a Runner instance.

Programmatic registration:

from pymoku.parts.runners import register_runner_type

def my_factory(name, config):
    return MyRunner(name, config)

register_runner_type('myrunner', my_factory)

Then use it in parts.toml:

[runners.example]
match = "SomeStep"
type = "myrunner"
custom_option = "value"

Cache Invalidation

The runner's state_hash() is mixed into each step's cache key. This means switching runners (e.g. from local to cloud, or changing SSH host) automatically invalidates the cache for affected steps.

Environment Variables

Environment variables are merged from three sources (later wins):

1. Runner config [runners.name.env] table
2. set_env() calls in parts.def (config_env)
3. ProcessStep extra_env kwarg

DockerStep with Remote Runners

When a DockerStep is matched by an SSH or cloud runner, the step's image attribute is forwarded. The remote runner wraps the command in docker run on the remote host, or passes it to the cloud API as a container job.