AlphaBuild's core
Project description
AlphaBuild
AlphaBuild is a simple monorepo build tool based on Make with out-of-the-box support for numerous tools in the Python, Bash, Jupyter Notebooks, Markdown, YAML ecosystems and with a strong focus on extensibility.
The way AlphaBuild works draws inspiration heavily from monorepo build tools such as Pants, Bazel, Buck. It can run at once multiple linters, formatters, type checkers, hermetic packers, testing frameworks, virtual environment managers etc.
- Platforms
- Usage
- AlphaBuild structure
- IDE Integration
- Example monorepo running AlphaBuild
- Common admin actions
- Out-of-the-box tools by language
- Comparison with Pants, Bazel, Pre-commit, Makefiles
- Limitations
Platforms
AlphaBuild works on Linux distributions, MacOS, WSL and Windows with Git Bash.
Make
and Bash
are AlphaBuild's only pre-requisites (Note: Bash with GNU utilities, not BSD utilities).
Linux, WSL
Since Linux is awesome and WSL follows its footsteps, AlphaBuild should just work there.
MacOS
AlphaBuild relies heavily on the GNU version of find
and egrep
, so, if you are running an OS which,
by default, uses BSD rather than GNU (pointing fingers to MacOS here, you may need to
brew install
findutils
and grep
, see: https://xenodium.com/gnu-find-on-macos/)
Also, AlphaBuild may not work if you have ancient versions of Make
/ Bash
, so try upgrading them if some commands
don't seem to work. Macs typically come equipped with ancient versions of Bash
.
Windows
On Windows, Make
and Bash
are not supported out-of-the-box, so it is recommended to use AlphaBuild on Windows within
Git Bash. Note that Git Bash does not come with Make pre-installed. Once you downloaded and installed Git Bash
run build-support/git-bash-integration/install_make.sh
running Git Bash as administrator. Alternatively, you may get
Make
from conda
.
Usage
Usually to format, lint, type-check, test, package, ... the code, one needs to run a bunch of commands in the terminal, setting the right flags and the right parameters. This Make-based build system helps with running these commands with a very simple interface:
make <goal> <optional-targets>
where
- goal = what tools we run
- targets = over which files we run these tools.
Goals - what tools we run
Goals mean what command line tools to run. This build system can run one more tools at once as follows:
- Single individual tool
- e.g.
make mypy
,make flake8
,make isort
- e.g.
- Multiple tools for a specific language
- e.g.
make fmt-py
runs all Python formatters e.g.isort
,black
,docformatter
,flynt
,autoflake
- e.g.
make fmt-sh
runsshfmt
- e.g.
make lint-py
runs Python all linters and all formatters in "verification" mode, that isflake8
+pylint
+ check whether the code is already formatted withisort
,black
,docformatter
,flynt
,autoflake
make fmt-md
,make lint-yml
,test-sh
,test-py
... work similarly
- e.g.
- Multiple tools for multiple languages
- e.g.
make fmt
runs all formatters for all supported languages (Python, Bash, Markdown, YAML, ...) - e.g.
make lint
runs all linters for all supported languages - e.g.
make test
runs all test suites for all supported languages
- e.g.
It is possible to run multiple goals at once like make lint test
. In addition, it is very easy to change the meaning
of goals that run more than one command since they are very simply defined in Make based on other goals. For example,
one can remove the shfmt
from bash linting simply by doing the below:
# Before
lint-sh: shellcheck shfmt-check # where shellcheck and shfmt-check run the respective commands
# After
lint-sh: shellcheck
Per-tool config files (e.g. mypy.ini
, pyproject.toml
) are found in build-support/<language>/tools-config/
.
Targets - what files we run the tools on
We have seen that Make gives us the power to run multiple terminal commands effortlessly. Using a Makefile like described above is standard practice in many projects, typically running the different tools over all their files. However, as projects grow, the need to run these tools at different granularities (e.g. in a specific directory, over a given file, on the diff between two branches, since we last committed etc). This is where targets come into play.
With default targets
The default targets per-language are defined at the top of the Makefile
in language specific variables
e.g. ONPY=py-project1/ py-project2/ script.py
and ONSH=scripts/
.
make lint
runs:- all Python linters on all directories (in the
$ONPY
) that contain Python/stub files. - all notebook linters on all directories (in
$ONNB
) that contain.ipynb
files. - all Bash linters (shellcheck) on all directories (in
$ONSH
) that contain Bash files. - a Haskell linter (hlint) on all directories (in
$ONHS
) that contain Haskell files. - a YAML linter (yamllint) on all directories (in
$ONYML
) that contain YAML files.
- all Python linters on all directories (in the
make lint
,make fmt -j1
,make type-check
work similarly- remember that the
$(ONPY)
,$(ONSH)
, ... variables are defined at the top of the Makefile and represent the default locations where AlphaBuild searches for files certain languages.
With specific targets
To specify manually the files/directories you want to run your tools on, AlphaBuild leverages the "on" syntax:
- file:
make lint on=app_iqor/server.py
runs all Python linters on the file, same asmake lint-py on=app_iqor/server.py
- directory:
make lint on=lib_py_utils
runs a bunch of linters on the directory, in this case, same asmake lint-py on=lib_py_utils/
. - files/directories:
make lint on="lib_py_utils app_iqor/server.py"
runs a bunch of linters on both targets. - globs:
make lint on=lib_*
- aliases:
make fmt on=iqor
where at the top of the Makefileiqor=app_iqor/iqor app_iqor/test_iqor
, this is the same asmake fmt on=app_iqor/iqor app_iqor/test_iqor
becauseiqor
is an alias forapp_iqor/iqor app_iqor/test_iqor
. Even though this example is simplistic, it is useful to alias combinations of multiple files/directories. It is recommended to set aliases as constants in the Makefile even though environment variables would also work. - same for
make fmt
,make test
,make type-check
.
With git revision targets
make fmt -j1 since=master
runs all formatters on the diff between the current branch and master.make fmt -j1 since=HEAD~1
runs all formatters on all files that changed since "2 commits ago".make lint since=--cached
runs all linters on all files that are "git added".- all goals that support the "on" syntax also support the "since" syntax
Mixed "on" and "since"
One can use the "on" and "since" syntaxes at the same time. For example:
make lint on=my_dir/ since=HEAD~2
will run all linters on all files inmy_dir/
that changed since "3 commits ago".
Constraints
Different languages may have different goals, for example Python can be packaged hermetically with Shiv, while Bash obviously can't.
The following goals must support the "on" and "since" syntax and ensure that they are only run if there are any targets for the language they target:
- format
- lint
- type-check
- test
If you want to learn more about the API of a specific goal, check the source code.
AlphaBuild structure
- Makefile: AlphaBuild's entry point, this is where all components come together.
- 3rdparty/: Files required to build environments of 3rd party dependencies (e.g. requirements.txt files, package.json or lock files)
- build-support/: Makefile library inspired by Pants/Bazel to run linters, formatters, test frameworks, type
checkers, packers etc. on a variety of languages (Python, Jupyter Notebooks, Bash, Haskell, YAML, Markdown)
- The flags used per-tool (e.g. setting the paths to config files) can be found in
build-support/make/config/<lang>.mk
- The core part of AlphaBuild lives in
build-support/make/core/
, this comprises the build-system backboneresolver.mk
and recipes to run lots of readily-available tools. This should be the same for all monorepos that use AlphaBuild. - By convention, repo-specific custom goals go in
build-support/make/extensions/
following the examples incore
. build-support/<other-programming-lang-than-make>/
contain things like config files for each tool and other files required for your custom AlphaBuild goals.
- The flags used per-tool (e.g. setting the paths to config files) can be found in
IDE Integration
- PyCharm / IntelliJ
- Windows: On Windows, it is advised to set Git Bash as the default "Terminal". In "settings" search for "terminal",
go to "Tools → Terminal" and set "Shell Path" to something like
C:\Program Files\Git\bin\bash.exe
. - PYTHONPATH: If you are writing Python, please mark the directories for each project as "Sources Roots", such that PyCharm discovers your imports.
- AlphaBuild hotkeys: It is easy to use AlphaBuild with hotkeys. For example, mapping
Alt+F
tomake fmt -j1 on=<current-file>
andAlt+L
tomake lint on=<current-file>
. To set this up, follow the example inbuild-support/ide-integration/windows/
to set up an external tool. Unix systems would be similar. Next, just map the new external tools to your hotkeys.
- Windows: On Windows, it is advised to set Git Bash as the default "Terminal". In "settings" search for "terminal",
go to "Tools → Terminal" and set "Shell Path" to something like
Example monorepo running AlphaBuild
To see AlphaBuild at work in a real-world example check https://github.com/cristianmatache/workspace out. Workspace extends AlphaBuild with support for Prometheus, Alertmanager and Grafana.
Common admin actions
Installation
To add this build system to an existing repo, one needs to simply copy over build-support/
, 3rdparty/
, Makefile
.
Create a conda environment, activate it and make env-default-replicate
, as a one-off, to set up the python, markdown
and bash environments (mostly pip/npm install-s).
CI/CD setup
Since all CI/CD pipelines essentially rely on running some scripts in a certain order, AlphaBuild can be called directly from any CI/CD pipeline regardless of CI/CD technology provider. AlphaBuild helps with ensuring that both the CI pipelines and developers run the exact same commands. Since, one can easily select targets within the repo, setting pipelines on a per sub-project basis, is effortless with AlphaBuild.
Upgrade
To upgrade an existing installation if new tools are added or changes are made to the target resolution infrastructure,
one would simply need to replace the lib-support/make/core
directory. To do that please run:
pip install alpha-build-core --target tmp/
tar -xvf tmp/alpha_build_core.tar.gz
rm -rf tmp/
Change goal definitions
Let's say, for example, you don't want to run pylint
as part of your python linting. You would simply go to the
Makefile
and change the definition of the lint-py
goal to not include pylint
.
Add goals
The goals that are available out of the box are found in build-support/make/core/<language>/
.
You can extend/replace the core goals for new languages and/or tools by writing .mk
code in
build-support/make/extensions/<language>/
following the examples in build-support/make/core/
.
For example, https://github.com/cristianmatache/workspace extends AlphaBuild with Prometheus and Alertmanager goals.
Update PYTHONPATH
The PYTHONPATH is set at the top of the Makefile
. For example, to add new directories to
the PYTHONPATH (i.e. to mark them as sources roots) set PY_SOURCES_ROOTS
at the top of the Makefile.
See/Change tools config
Let's say you want to change the way mypy
is configured to exclude some directory from checking. Then head to
build-support/make/config/python.mk
check what is the path to the mypy
config file, go there and update it.
All other tools work similarly.
Third party environments
- Exact reproduction of the default environment: The recipes to fully replicate the default environment
(mostly using
pip
,conda
andnpm
) are found inbuild-support/make/core/<langugage>/setup.mk
, where they use dependency files and lock files that can be found in3rdparty/
. In practice, runmake env-default-replicate
inside a conda environment. Also make sure you also havenpm
installed becausemarkdownlint
andbats
bash testing framework come fromnpm
(if you don't need them no need to worry aboutnpm
just exclude themarkdown
environment rule from the pre-requisites ofenv-default-replicate
) - Create/Upgrade/Edit default environment: If you want to edit the default environment, for example to add,
remove, constrain packages edit the
requirements.txt
not theconstraints.txt
file (in3rdparty/
). Theconstraints.txt
is only used for reproducibility. If you just want to upgrade your third party dependencies there is no need to temper with therequirements.txt
files. Then runmake env-default-upgrade
and check the lock files back into git. - Add a new environment: To add a new environment, first add the dependency files (e.g.
requirements.txt
) in3rdparty/<new-env-name>
, add a new goal inbuild-support/make/extensions
. For environment management over time, we strongly encourage maintaining the approach split between creation/upgrade/edit and exact reproduction of environments.
Nested Makefiles
Supposing you want to have another Makefile
for a specific project in the monorepo, just
import everything that you need from build-support/make/core/
and/or build-support/make/extensions/
. To change the
Now let's say you want to use a different config file for mypy
. You would have 2 options, either change the path
globally build-support/make/config/python.mk
or, if you just want different settings for your little project use
your inner Makefile
to overwrite the value of the corresponding variable (that points to the config file) with the
different path.
Generate requirements.txt for each sub-project
Run make reqs-py
.
Generate setup.py for each sub-project
Run build-support/python/packaging/generate_pip_install_files.py
Markdown badge
If you like AlphaBuild, wear the Markdown badge on your repo:
[![powered_by_alpha_build](https://img.shields.io/badge/Powered%20by%20-AlphaBuild-lightblue?style=flat&logo=CMake&logoColor=lightblue)
](https://github.com/cristianmatache/alpha-build)
Out-of-the-box tools by language
AlphaBuild has recipes to run the following tools. However, if you don't use some of them you don't need to have them
installed in your enviornments. For example, let's say you don't use bandit
then you don't need to have bandit
installed in your environment provided that you are not using the bandit
goal per-se or as part of a composite goal
like lint-py
or lint
.
- Python:
- Setup:
pip
/conda
- Type-check:
mypy
- Test:
pytest
- Format + Lint:
black
,docformatter
,isort
,autoflake
,flynt
,pre-commit
- Lint only:
flake8
,pylint
,bandit
- Package:
pipreqs
,shiv
- Setup:
- Jupyter:
- Setup:
pip
- Format + Lint:
jupyterblack
,nbstripout
- Lint only:
flake8-nb
- Setup:
- Bash:
- Setup:
npm
andconda
- Test:
bats
(bash testing:bats-core
,bats-assert
,bats-support
) - Format + Lint:
shfmt
- Lint only:
shellcheck
- Setup:
- Haskell:
- Lint:
hlint
- Lint:
- YAML:
- Setup:
pip
- Lint:
yamllint
,prettier
- Setup:
- Prometheus and Alertmanager YAML:
- Lint:
promtool check
,amtool check-config
- Lint:
- Markdown:
- Setup:
npm
- Format + Lint:
markdownlint
,prettier
- Setup:
- HTML + CSS:
- Setup:
npm
- Format + Lint:
prettier
- Setup:
- JavaScript:
- Setup:
npm
- Format + Lint:
prettier
- Setup:
- TypeScript:
- Setup:
npm
- Format + Lint:
prettier
- Setup:
It is very easy to extend this list with another tool, just following the existing examples.
Comparison with Pants, Bazel, Pre-commit and traditional Makefiles
Modern build tools like Pants or Bazel work similarly to AlphaBuild in terms of goals and targets, but they also add a caching layer on previous results of running the goals. While they come equipped with heavy machinery to support enormous scale projects, they also come with some restrictions and specialized maintenance and contribution requirements.
For example, Pants which, in my opinion, is the most suitable modern build tool for Python doesn't allow building
environments with arbitrary package managers (e.g. conda, mamba), does not work on Windows, prohibits inconsistent
environments (which is good but sometimes simply impossible in practice), does not yet support multiple environments.
Bazel, requires maintaining the dependencies between Python files twice, once as "imports" in the Python files
(the normal thing to do) and twice in some specific BUILD
files that must be placed in each directory (by contrast
Pants features autodiscovery). Maintaining the same dependencies in two places is quite draining. Of course, these tools
come with benefits like (remote) caching, incrementality and out-of-the-box support for hermetic packaging (e.g. PEXes),
remote execution etc. Moreover, playing with some new command line tools, or new programming languages / types of files
(e.g. Jupyter Notebooks, Markdown, YAML) may be challenging with these frameworks. The Pants community is very welcoming
and supportive towards incorporating new tools, so it would be good to give Pants a try first. However, if any of the
mentioned shortcomings is a hard requirement, Make seems like a good and robust alternative in the meanwhile which
withstood the test of time in so many settings. AlphaBuild's strengths are its flexibility, simplicity, transparency and
tooling richness. One can quickly hack/add a new tool, see the commands that run under the hood and does not need to
worry about BUILD files or the config language.
Since AlphaBuild is essentially a script manager (Python, Bash, Perl, anything) enhanced with advanced target/file/directory selection, AlphaBuild would allow an incremental adoption of large-scale build tools like Pants. For example, in the main Makefile, one could do:
# Makefile
lint-with-pants:
$(eval on := ::) # Default value if the user does not specify "on" in the terminal command like: make goal on=.
./pants lint $(on)
lint: lint-md lint-nb lint-yml lint-with-pants
such that running a command like the below would delegate most of the work to Pants while using AlphaBuild's core or custom capabilities not yet available in Pants (e.g. linting notebooks, markdown or YAML files).
make lint on=my-dir/
Pre-commit and typical usages of Make work exceptionally well on small projects but they don't really scale well to
multi-projects monorepos. The build system proposed here, already incorporates pre-commit
and is obviously compatible
with any existing Makefiles. This approach simply takes the idea of advanced target selection and ports it over to
classical techniques like pre-commit and Make.
Limitations
Since AlphaBuild is essentially a small-repo tool (Python Makefile) adapted to work on larger codebases (through target selection), there is a point from where it will no longer be able to scale up. Fortunately, that point is quite far away from medium-sized repos/teams.
In addition, AlphaBuild requires that the commands it builds are shorter than getconf ARG_MAX
characters.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for alpha_build_core-0.0.3b2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f4153a6cb61bc9d820787825237378dd4388af2a951ca896c52cdc940e587b5e |
|
MD5 | a430b30dbf0448017aff3264822f6dff |
|
BLAKE2b-256 | 3ff0da2571e1afbde190e351127eeea9d5be34dd1baec7ee2c1363d3b5c641c3 |