xcross 0.1

Zero-setup cross compilation.

xcross

Docker images and high-level scripts for plug-and-play C/C++ cross-compiling, inspired by rust-embedded/cross. xcross supports both bare-metal and OS-based compilation, with a wide variety of architectures and C-runtimes supported. Most images are <500MB, and <200MB compressed. xcross is ideal for:

• Testing cross-platform support in CI pipelines.
• Building and deploying cross-compiled programs.

Note that this project is similar to dockercross, however, xcross supports many more CPU architectures than dockcross. If you need Docker images of common architectures, dockcross may have better support.

Table of Contents

• Motivation
• Getting Started
  • xcross
  • run
  • Docker
• Travis CI Example
• Using xcross
• Sharing Binaries To Host
• Building/Running Dockerfiles
• Images
• Development Dependencies
• Toolchain Files
• Developing New Toolchains
• Platform Support
• License
• Contributing

Motivation

Unlike 10 years ago, we no longer live in an x86 world. ARM architectures are nearly ubiquitous in mobile devices, and popular in embedded devices, servers, and game systems. IBM's POWER and z/Architecture can run some high-end servers. PowerPC systems are popular in embedded devices, and used to be popular architectures for game systems, desktops, and servers. MIPS has been integral to autonomous driving systems and other embedded systems. RISC-V is rapidly being adopted for a wide variety of use-cases. The IoT market has lead to an explosion in embedded devices.

At the same time, modern software design builds upon a body of open source work. It is more important than ever to ensure that foundational libraries are portable, and can run on a wide variety of devices. However, few open source developers can afford a large selection of hardware to test code on, and most pre-packaged cross-compilers only support a few, common architectures.

Normally, cross compilers are limited by long compile times (to build the cross-compiler) and non-portable toolchain files, since the toolchain cannot be added to the path.

Using Docker images simplifies this, since the cross-compilers are pre-packaged in a compact image, enabling building, testing, and deploying cross-compiled code in seconds, rather than hours. Each image comes with a toolchain installed on-path, making it work with standard build tools without any configuration. And, xcross allows you to cross-compile code with zero setup required.

It just works.

Getting Started

This shows a simple example of building and running a C++ project on PowerPC64, a big-endian system:

xcross

xcross is a Python script to automate building transparently for custom targets, similar to Rust's cross.

# Clone our project locally.
git clone https://github.com/Alexhuszagh/cpp-helloworld.git
cd cpp-helloworld

# Add xcross to any command, and it just works.
xcross make --target=alpha-unknown-linux-gnu
file helloworld
# helloworld: ELF 64-bit LSB executable, Alpha (unofficial)

# We can also use environment variables for the target and dir.
export CROSS_TARGET=alpha-unknown-linux-gnu

# Let's try CMake. Here, we have to tell Docker where to 
# mount the directory, since we need to share the parent's files.
# Here we configure the project, build and run the executable.
mkdir build-alpha && cd build-alpha
xcross cmake ..
xcross make run
# Hello world!

# Let's try a raw C++ compiler. Here we build it using g++,
# and then run it using Qemu. It just works.
cd ..
xcross g++ helloworld.cc -o hello
xcross run hello

# We also support environment variable passthrough.
# Please note that if you use `$CXX`, it will evaluate
# on the host, so we must escape it.
xcross -E CXX=g++ '$CXX' helloworld.cc -o hello

run

run is a command inside the docker image that invokes Qemu with the correct arguments to execute the binary, whether statically or dynamically-linked. run hello is analogous to running ./hello as a native binary.

Docker

For more fine-tuned control, you can also build a project within a container:

# Pull the Docker image, and run it interactively, entering the container.
image=alpha-unknown-linux-gnu
docker pull "ahuszagh/cross:$image"
docker run -it "ahuszagh/cross:$image" /bin/bash

# Clone the repository, build and run the code in the container using CMake.
# These toolchains in general aren't necessary, but ensure
# CMake knows we're cross-compiling and how to link.
git clone https://github.com/Alexhuszagh/cpp-helloworld.git
cd cpp-helloworld
mkdir build && cd build
# Build a statically-linked executable.
cmake .. -DCMAKE_TOOLCHAIN_FILE=/toolchains/static.cmake
make
# Just works, as long as `add_custom_target` uses
# `${CMAKE_CROSSCOMPILING_EMULATOR} $<TARGET_FILE:..>`
# This uses Qemu as a wrapper, so running the executable
# only works on some architectures.
make run
# Can also run executables manually.
run hello

# Clean, and build a dynamically-linked executable.
rm -rf ./*
cmake .. -DCMAKE_TOOLCHAIN_FILE=/toolchains/shared.cmake
make
# Just works, since `run` has the proper library search path.
make run
run hello

# Can also use Makefiles normally. Here we prefer shared linking,
# which also adds position-independent code. These environment
# only add the `-fPIC` or `-static` flags: nothing else is modified.
cd ..
source /toolchains/shared
make
run helloworld

# Can also use static linking.
source /toolchains/static
make clean
make
run helloworld

# Can also just raw `c++` and `cc` commands.
# It's really that simple.
c++ helloworld.cc -fPIC
run a.out

c++ helloworld.cc -static
run a.out

Travis CI Example

A simple example of integrating cross images is as follows:

language: cpp
dist: bionic
services:
  - docker

# Use a matrix with both native toolchains and cross-toolchain images.
matrix:
  include:
    - arch: amd64
      os: linux

    - arch: amd64
      os: linux
      env:
        - TARGET="mips64"

before_install:
  - |
    if [ "$TARGET" != "" ] ; then
      wget https://raw.githubusercontent.com/Alexhuszagh/toolchains/main/bin/xcross
      chmod +x xcross
      docker pull ahuszagh/cross:"$TARGET"
    fi

script:
  - |
    mkdir build && cd build
    xcross=
    if [ "$TARGET" != "" ] ; then
      xcross=../xcross --target="$TARGET"
    fi
    $xcross cmake ..
    $xcross make -j 5
    $xcross run tests/test

Using xcross

Most of the magic happens via xcross, which allows you to transparently execute commands in a Docker container. Although xcross provides simple, easy-to-use defaults, it has more configuration options for extensible cross-platform builds. Most of these command-line arguments may be provided as environment variables.

WARNING By default, the root directory is shared with the Docker container, for maximum compatibility. In order to mitigate any security vulnerabilities, we run any build commands as a non-root user, and escape input in an attempt to avoid any script injections. If you are worried about a malicious build system, you may further restrict this using the --dir option.

Installing

xcross may be installed via PyPi via:

pip install xcross --user

Or xcross may be installed via git:

git clone https://github.com/Alexhuszagh/xcross
cd xcross
python setup.py install --user

Arguments

Fallthrough

All arguments that are not recognized, as follows, are passed to the Docker container, with a few caveats.

• Escape all control characters for the local shell.

If they are not properly escaped, they will be evaluated on the host, often giving unexpected results.

# This works in POSIX shells, since we have no environment variables that 
# might be evaluated, and we've escaped the `|`.
xcross echo "int main() { return 0; }" '|' c++ -x c++ -

# In Windows CMD, we need to escape the `|` as follows:
xcross echo "int main() { return 0; }" ^| c++ -x c++ -

# Escape environment variables in POSIX shells to evaluate them in the container.
xcross -E CXX=cpp '$CXX' main.c -o main

# This does not work in POSIX shells, since it evaluates `$CXX` in the local shell.
xcross -E CXX=cpp $CXX main.c -o main

• Any environment variables and paths should be passed in POSIX style.

Although non-trivial paths that exist on Windows will be translated to POSIX style, ideally you should not rely on this.

# Doesn't work, since we use a Windows-style path to an output file.
xcross c++ main.c -o test\basic

# This does work, since it uses a POSIX-style path for the output.
xcross c++ main.c -o test/basic

# This won't work, since we use a Windows-style environment variable.
# We don't know what this is used for, so we can't convert this.
xcross -E VAR1=cpp ^%VAR1^% main.c -o main

# Works in Windows CMD, since $X doesn't expand.
xcross -E VAR1=cpp $VAR1 main.c -o main

xcross Arguments

• --target, CROSS_TARGET: The target architecture to compile to.

# These two are identical, and build for Alpha on Linux/glibc
xcross --target=alpha-unknown-linux-gnu ...
CROSS_TARGET=alpha-unknown-linux-gnu xcross ...

• --dir, CROSS_DIR: The target architecture to compile to.

# These two are identical, and share only from the 
# current working directory.
xcross --dir=. ...
CROSS_DIR=. xcross ...

• -E, --env: Pass environment variables to the container.

If no value is passed for the variable, it exports the variable from the current environment.

# These are all identical.
xcross -E VAR1 -E VAR2=x -E VAR3=y
xcross -E VAR1 -E VAR2=x,VAR3=y
xcross -E VAR1,VAR2=x,VAR3=y

• --cpu, CROSS_CPU: Set the CPU model to compile/run code for.

If not provided, it defaults to a generic processor model for the architecture. If provided, it will set the register usage and instruction scheduling parameters in addition to the generic processor model.

# Build for the PowerPC e500mc CPU.
export CROSS_TARGET=ppc-unknown-linux-gnu
xcross --cpu=e500mc c++ helloworld.cc -o hello
xcross --cpu=e500mc run hello
CROSS_CPU=e500mc xcross run hello

In order to determine valid CPU model types for the cross-compiler, you may use either of the following commands:

# Here, we probe GCC for valid CPU names for the cross-compiler.
export CROSS_TARGET=ppc-unknown-linux-gnu
xcross cc-cpu-list
# 401 403 405 405fp ... e500mc ... rs64 titan

# Here, we probe Qemu for the valid CPU names for the emulation.
xcross run-cpu-list
# 401 401a1 401b2 401c2 ... e500mc ... x2vp50 x2vp7

These are convenience functions around gcc -mcpu=unknown and qemu-ppc -cpu help, listing only the sorted CPU types. Note that the CPU types might not be identical for both, so it's up to the caller to properly match the CPU types.

• --username, CROSS_USERNAME: The Docker Hub username for the Docker image.

This defaults to ahuszagh if not provided, however, it may be explicit set to an empty string. If the username is not empty, the image has the format $username/$repository:$target, otherwise, it has the format $repository:$target.

# These are all identical.
xcross --username=ahuszagh ...
CROSS_USERNAME=ahuszagh xcross ...

• --repository, CROSS_REPOSITORY: The name of the repository for the image.

This default to cross if not provided or is empty.

# These are all identical.
xcross --repository=cross ...
CROSS_REPOSITORY=cross xcross ...

• --docker, CROSS_DOCKER: The command for the Docker executable.

This default to docker if not provided or is empty.

# These are all identical.
xcross --docker=docker ...
CROSS_DOCKER=docker xcross ...

Sharing Binaries To Host

In order to build projects and share data back to the host machine, you can use Docker's --volume to bind a volume from the host to client. This allows us to share a build directory between the client and host, allowing us to build binaries with inside the container and test/deploy on the host.

# On a Linux distro with SELinux, you may need to turn it
# to permissive, to enable file sharing:
#   `setenforce 0`

# Run docker image, and share current directory.
image=alpha-unknown-linux-gnu
git clone https://github.com/Alexhuszagh/cpp-helloworld.git
docker run -it --volume "$(pwd)/cpp-helloworld:/hello" \
    "ahuszagh/cross:$image" \
    /bin/bash

# Enter the repository, and make a platform-specific build.
cd hello
mkdir build-alpha && cd build-alpha
cmake .. -DCMAKE_TOOLCHAIN_FILE=/toolchains/static.cmake \
    -DCMAKE_BUILD_TYPE=Release
make

# Exit, and check we have our proper image.
exit
file cpp-helloworld/build-alpha/hello
# cpp-helloworld/build-alpha/hello: ELF 64-bit LSB executable, 
# Alpha (unofficial), version 1 (GNU/Linux), statically linked, 
# BuildID[sha1]=252707718fb090ed987a9eb9ab3a8c3f6ae93482, for 
# GNU/Linux 3.2.0, not stripped

# Please note the generated images will be owned by `root`.
ls -lh cpp-helloworld/build-alpha/hello
# -rwxr-xr-x. 1 root root 2.4M May 30 20:50 cpp-helloworld/build-alpha/hello

Building/Running Dockerfiles

To build all Docker images, run docker/build.sh. To build and run a single docker image, use

image=ppcle-unknown-linux-gnu
docker build -t "ahuszagh/cross:$image" . --file "docker/Dockerfile.$image"
docker run -it "ahuszagh/cross:$image" /bin/bash

Images

For a list of pre-built images, see DockerHub. To remove local, installed images from the pre-built, cross toolchains, run:

# On a POSIX shell.
docker rmi $(docker images | grep 'ahuszagh/cross')

Image Types

There are two types of images:

• Images with an OS layer, such as ppcle-unknown-linux-gnu.
• Bare metal images, such as ppcle-unknown-elf.

The bare metal images use the newlib C-runtime, and are useful for compiling for resource-constrained embedded systems, and do not link to a memory allocator. Please note that not all bare-metal images provide complete startup routines (crt0), and therefore might need to be linked against standlone flags (-nostartfiles, -nostdlib, -nodefaultlibs, or -ffreestanding) with a custom _start or equivalent routine or a custom crt0 must be provided.

The other images use a C-runtime that depends on a POSIX-like OS (such as Linux, FreeBSD, or MinGW for Windows), and can be used with:

• musl (*-musl)
• glibc (*-gnu)
• uClibc-ng (*-uclibc)
• android (*-android, only available on some architectures)

If you would like to test if the code compiles (and optionally, runs) for a target architecture, you should generally use a linux-gnu image.

Triples

All images are named as ahuszagh/cross:$triple, where $triple is the target triple. The target triple consists of:

• arch, the CPU architecture (mandatory).
• vendor, the CPU vendor.
• os, the OS the image is built on.
• system, the system type, which can comprise both the C-runtime and ABI.

For example, the following image names decompose to the following triples:

• avr, or (avr, unknown, -, -)
• mips-unknown-o32, (mips, unknown, -, o32)
• mips-unknown-linux-gnu, (mips, unknown, linux, gnu)

If an $arch-unknown-linux-gnu is available, then $arch is an alias for $arch-unknown-linux-gnu.

OS/Architecture Support

In general, the focus of these images is to provide support for a wide variety of architectures, not operating systems. We will gladly accept Dockerfiles/scripts to support more operating systems, like FreeBSD.

We do not support Darwin/iOS for licensing reasons, since reproduction of the macOS SDK is expressly forbidden. If you would like to build a Darwin cross-compiler, see osxcross.

We also do not support certain cross-compilers for popular architectures, like Hexagon, due to proprietary linkers (which would be needed for LLVM support).

Versioning

Image names may optionally contain a trailing version, which will always use the same host OS, GCC, and C-runtime version.

• No Version: Alias for the latest version listed.
• 0.1: GCC 10.2.0, glibc 2.31, and Ubuntu 20.04.

Dependencies

In order to use xcross, you must have:

• python (3.6+)

In order to build the toolchains, you must have:

• docker
• bash

In order to add new toolchains, you must have:

• crosstool-NG
• cut

Everything else runs in the container.

Toolchain Files

In order to simplify using the cross-compiler with CMake, we provide 3 CMake toolchain files:

• /toolchains/toolchain.cmake, which contains the necessary configurations to cross-compile.
• /toolchains/shared.cmake, for building dynamically-linked binaries.
• /toolchains/static.cmake, for building statically-linked binaries.

Likewise, to simplify using the cross-compiler with Makefiles, we provide 2 Bash config files:

• /env/shared, for building dynamically-linked binaries.
• /env/static, for building statically-linked binaries.

Developing New Toolchains

To add your own toolchain, the general workflow is as follows:

1. List toolchain samples.
2. Configure your toolchain.
3. Move the config file to ct-ng.
4. Patch the config file.
5. Create a source environment file.
6. Create a CMake toolchain file.
7. Create a Dockerfile.

After the toolchain is created, the source environment file, CMake toolchain file, and Dockerfile may be created via:

BITNESS=32 OS=Linux TARGET=arm-unknown-linux-gnueabi docker/new-image.sh

Configure Toolchain

ct-ng list-samples
image=arm-unknown-linux-gnueabi
ct-ng "$image"
ct-ng menuconfig
mv .config ct-ng/"$image".config
ct-ng/patch.sh ct-ng/"$image".config
touch Dockerfile."$image"

Source Environment File - Linux

#!/bin/bash

scriptdir=`realpath $(dirname "$BASH_SOURCE")`
source "$scriptdir/shortcut.sh"

export PREFIX=arm-unknown-linux-gnueabi
export DIR=/home/crosstoolng/x-tools/"$PREFIX"/

shortcut_gcc
shortcut_util

Source Environment File - Bare Metal

#!/bin/bash

scriptdir=`realpath $(dirname "$BASH_SOURCE")`
source "$scriptdir/shortcut.sh"

export PREFIX=arm-unknown-eabi
export DIR=/home/crosstoolng/x-tools/"$PREFIX"/

shortcut_gcc
shortcut_util

CMake Toolchain File - Linux

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR arm)

set(CMAKE_FIND_ROOT_PATH "home/crosstoolng/x-tools/arm-unknown-linux-gnueabi/")
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

CMake Toolchain File - Bare Metal

# Need to override the system name to allow CMake to configure,
# otherwise, we get errors on bare-metal systems.
set(CMAKE_SYSTEM_NAME Generic)
set(CMAKE_SYSTEM_PROCESSOR arm)
cmake_policy(SET CMP0065 NEW)

set(CMAKE_FIND_ROOT_PATH "/home/crosstoolng/x-tools/arm-unknown-eabi/")
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

Dockerfile

# Base image
FROM ahuszagh/cross:base

# Copy our config files and build GCC.
# This is done in a single step, so the docker image is much more
# compact, to avoid storing any layers with intermediate files.
COPY ct-ng/arm-unknown-linux-gnueabi.config /ct-ng/
COPY docker/gcc.sh /ct-ng/
RUN ARCH=arm-unknown-linux-gnueabi /ct-ng/gcc.sh

# Remove GCC build scripts and config.
RUN rm -rf /ct-ng/

 Add symlinks
COPY symlink/shortcut.sh /
COPY symlink/arm-unknown-linux-gnueabi.sh /
RUN /arm-unknown-linux-gnueabi.sh
RUN rm /shortcut.sh /arm-unknown-linux-gnueabi.sh

# Add toolchains
COPY cmake/arm-unknown-linux-gnueabi.cmake /toolchains/toolchain.cmake
COPY cmake/shared.cmake /toolchains/
COPY cmake/static.cmake /toolchains/
COPY env/shared /env/
COPY env/static /env/

For a bare-metal example, see docker/Dockerfile.ppcle-unknown-elf. For a Linux example, see docker/Dockerfile.ppcle-unknown-linux-gnu. Be sure to add your new toolchain to images.sh, and run the test suite with the new toolchain image.

Platform Support

Currently, we only create images that are supported by:

• crosstool-NG with official sources
• Debian packages
• Android NDK's
• RISCV GNU utils.

We therefore support:

• ARM32 + Thumb (Linux, Android, Bare-Metal)
• ARM32-BE + Thumb (Linux, Android, Bare-Metal)
• ARM64 (Linux, Android, Bare-Metal)
• ARM64-BE (Linux, Android, Bare-Metal)
• alpha (Linux)
• ARC (Linux, Bare-Metal)
• AVR (Bare-Metal)
• HPPA (Linux)
• i386-i686 (Bare-Metal)
• i686 (Linux, MinGW, Android)
• m68k (Linux)
• MicroBlaze (Linux)
• MicroBlaze-LE (Linux)
• MIPS (Linux, Bare-Metal)
• MIPS-LE (Linux, Bare-Metal)
• MIPS64 (Linux, Bare-Metal)
• MIPS64-LE (Linux, Bare-Metal)
• Moxie (Bare-Metal: Moxiebox and ELF)
• Moxie-BE (Bare-Metal: ELF-only)
• NIOS2 (Linux, Bare-Metal)
• PowerPC (Linux, Bare-Metal)
• PowerPC-LE (Linux, Bare-Metal)
• PowerPC64 (Linux, Bare-Metal)
• PowerPC64-LE (Linux, Bare-Metal)
• RISCV32 (Linux, Bare-Metal)
• RISCV64 (Linux, Bare-Metal)
• s390 (Linux)
• s390x (Linux)
• SH1-4 (Linux, Bare-Metal)
• Sparc64 (Linux)
• x86_64 (Linux, MinGW, Android, Bare-Metal)
• xtensa (Linux)

Platform-specific details:

• Xtensa does not support newlib, glibc, or musl.

License

This is free and unencumbered software released into the public domain. This project, however, does derive off of projects that are not necessarily public domain software, such as crosstool-NG, the Android NDK, as well as build off of GCC, the Linux kernel headers, and the relevant C-runtime (glibc, musl, uClibc-ng). Therefore, distributing any images will be subject to the GPLv3 or later (for GCC), and GPLv2 for the Linux headers.

These licenses are only relevant if you distribute a toolchain as part of a proprietary system: for merely compiling and linking code as part of a standard toolchain, the usual linking exceptions apply.

Contributing

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in xcross by you, will be unlicensed (free and unencumbered software released into the public domain).