delocate 0.12.0

Move macOS dynamic libraries into package

Delocate

macOS utilities to:

• find dynamic libraries imported from python extensions

• copy needed dynamic libraries to directory within package

• update macOS install_names and rpath to cause code to load from copies of libraries

Provides scripts:

• delocate-listdeps – show libraries a tree depends on

• delocate-path – copy libraries a tree depends on into the tree and relink

• delocate-wheel – rewrite wheel having copied and relinked library dependencies into the wheel tree.

• delocate-merge – combine two wheels with different architectures into one wheel with dual architecture binaries.

Auditwheel is a similar tool for Linux. Auditwheel started life as a partial fork of Delocate.

The problem

Let’s say you have built a wheel somewhere, but it’s linking to dynamic libraries elsewhere on the machine, so you can’t distribute it, because others may not have these same libraries. Here we analyze the dependencies for a Scipy wheel:

$ delocate-listdeps scipy-0.14.0b1-cp34-cp34m-macosx_10_6_intel.whl
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

By default, this does not include libraries in /usr/lib and /System. See those too with:

$ delocate-listdeps --all scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
/usr/lib/libSystem.B.dylib
/usr/lib/libstdc++.6.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

The output tells me that Scipy has picked up dynamic libraries from my Homebrew installation of gfortran (as well as the system libs).

You can get a listing of the files depending on each of the libraries, using the --depending flag:

$ delocate-listdeps --depending scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib:
    scipy/interpolate/dfitpack.so
    scipy/special/specfun.so
    scipy/interpolate/_fitpack.so
    ...

A solution

We can fix like this:

$ delocate-wheel -w fixed_wheels -v scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
Fixing: scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
Copied to package .dylibs directory:
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

The -w flag tells delocate-wheel to output to a new wheel directory instead of overwriting the old wheel. -v (verbose) tells you what delocate-wheel is doing. In this case it has made a new directory in the wheel zipfile, named scipy/.dylibs. It has copied all the library dependencies that are outside the macOS system trees into this directory, and patched the python .so extensions in the wheel to use these copies instead of looking in /usr/local/Cellar/gfortran/4.8.2/gfortran/lib.

Check the links again to confirm:

$ delocate-listdeps --all fixed_wheels/scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
/usr/lib/libSystem.B.dylib
/usr/lib/libstdc++.6.0.9.dylib
@loader_path/../../../../.dylibs/libgcc_s.1.dylib
@loader_path/../../../../.dylibs/libgfortran.3.dylib
@loader_path/../../../../.dylibs/libquadmath.0.dylib
@loader_path/../../../.dylibs/libgcc_s.1.dylib
@loader_path/../../../.dylibs/libgfortran.3.dylib
@loader_path/../../../.dylibs/libquadmath.0.dylib
@loader_path/../../.dylibs/libgcc_s.1.dylib
@loader_path/../../.dylibs/libgfortran.3.dylib
@loader_path/../../.dylibs/libquadmath.0.dylib
@loader_path/../.dylibs/libgcc_s.1.dylib
@loader_path/../.dylibs/libgfortran.3.dylib
@loader_path/../.dylibs/libquadmath.0.dylib
@loader_path/libgcc_s.1.dylib
@loader_path/libquadmath.0.dylib

So - system dylibs the same, but the others moved into the wheel tree.

This makes the wheel more likely to work on another machine which does not have the same version of Gfortran installed - in this example.

Checking required architectures

Current Python.org Python and the macOS system Python (/usr/bin/python) are both dual architecture binaries. For example:

$ lipo -info /usr/bin/python
Architectures in the fat file: /usr/bin/python are: x86_64 arm64e

Note: you can compile ARM binaries for basic ARM (arm64), or to use some extended ARM capabilities (arm64e) - see this SO post. Both types of binaries work on Mac M1 and M2 machines, so we will use arm64 to refer to either arm64 or arm64e.

The Big Sur macOS Python above has both x86_64 and arm64 (M1) versions fused into one file. Earlier versions of macOS had dual architectures that were 32-bit (i386) and 64-bit (x86_64).

For full compatibility with system and Python.org Python, wheels built for Python.org Python or system Python should have the corresponding architectures — e.g. x86_64 and arm64 versions of the Python extensions and their libraries. It is easy to link Python extensions against single architecture libraries by mistake, and therefore get single architecture extensions and / or libraries. In fact my Scipy wheel is one such example, because I inadvertently linked against the Homebrew libraries, which were x86_64 only. To check this you can use the --require-archs flag:

$ delocate-wheel --require-archs=intel scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
Traceback (most recent call last):
File "/Users/mb312/.virtualenvs/delocate/bin/delocate-wheel", line 77, in <module>
    main()
File "/Users/mb312/.virtualenvs/delocate/bin/delocate-wheel", line 69, in main
    check_verbose=opts.verbose)
File "/Users/mb312/.virtualenvs/delocate/lib/python2.7/site-packages/delocate/delocating.py", line 477, in delocate_wheel
    "Some missing architectures in wheel")
delocate.delocating.DelocationError: Some missing architectures in wheel

Notice that this command was using an earlier version of Delocate that supported Python 2; we now support Python 3 only.

The intel argument to --require-archs above requires dual 32- and 64- bit architecture extensions and libraries. You can see which extensions are at fault by adding the -v (verbose) flag:

$ delocate-wheel -w fixed_wheels --require-archs=intel -v scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
Fixing: scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
Required arch i386 missing from /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
Required arch i386 missing from /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
Required arch i386 missing from scipy/fftpack/_fftpack.so
Required arch i386 missing from scipy/fftpack/convolve.so
Required arch i386 missing from scipy/integrate/_dop.so
...

I need to rebuild this wheel to link with dual-architecture libraries.

Making dual-architecture binaries

Modern Mac wheels can be either arm64 (M1/M2 ARM), x86_64 (64-bit Intel) or both (universal2).

Building an entire Python wheel as dual-architecture can be difficult, perhaps because you need to link different libraries in the two cases, or you need different compiler flags, or because you build for arm64 on one continuous integration platform (such as - at time of writing - Cirrus CI), and x86_64 on another.

One solution to this problem is to do an entire arm64 wheel build, and then an entire x86_64 wheel build, and fuse the two wheels into a universal wheel.

That is what the delocate-merge command does.

Let’s say you have built an ARM and Intel wheel, called, respectively:

• scipy-1.9.3-cp311-cp311-macosx_12_0_arm64.whl

• scipy-1.9.3-cp311-cp311-macosx_10_9_x86_64.whl

Then you could create a new fused (universal2) wheel in the tmp subdirectory with:

delocate-merge scipy-1.9.3-cp311-cp311-macosx_12_0_arm64.whl scipy-1.9.3-cp311-cp311-macosx_10_9_x86_64.whl -w tmp

The output wheel in that case would be:

• tmp/scipy-1.9.3-cp311-cp311-macosx_12_0_universal2.whl

In the new wheel, you will find, using lipo -archs - that all binaries with the same name in each wheel are now universal (x86_64 and arm64).

:warning: Note: In previous versions (<0.12.0) making dual architecture binaries was performed with the delocate-fuse command. This commannd would overwrite the first wheel passed in by default. This led to the user needing to rename the wheel to correctly describe what platforms it supported. For this and other reasons, wheels created with this were often incorrect. From version 0.12.0 and on, the delocate-fuse command has been removed and replaced with delocate-merge. The delocate-merge command will create a new wheel with an automatically generated name based on the wheels that were merged together. There is no need to perform any further changes to the merged wheel’s name. If the old behavior is needed (not recommended), pin the version to delocate==0.11.0.

Troubleshooting

DelocationError: “library does not exist”

When running delocate-wheel or its sister command delocate-path, you may get errors like this:

delocate.delocating.DelocationError: library "<long temporary path>/wheel/libme.dylib" does not exist

This happens when one of your libraries gives a library dependency with a relative path. For example, let’s say that some file in my wheel has this for the output of otool -L myext.so:

myext.so:
    libme.dylib (compatibility version 10.0.0, current version 10.0.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 60.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

The first line means that myext.so expects to find libme.dylib at exactly the path ./libme.dylib - the current working directory from which you ran the executable. The output should be something like:

myext.so:
    /path/to/libme.dylib (compatibility version 10.0.0, current version 10.0.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 60.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

To set the path to the library, the linker is using the install name id of the linked library. In this bad case, then otool -L libme.dylib will give something like:

libme.dylib (compatibility version 10.0.0, current version 10.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

where the first line is the install name id that the linker picked up when linking myext.so to libme.dylib. Your job is to fix the build process so that libme.dylib has install name id /path/to/libme.dylib. This is not a problem specific to Delocate; you will need to do this to make sure that myext.so can load libme.dylib without libme.dylib being in the current working directory. For CMAKE builds you may want to check out CMAKE_INSTALL_NAME_DIR.

Code

See https://github.com/matthew-brett/delocate

Released under the BSD two-clause license - see the file LICENSE in the source distribution.

GitHub Actions kindly tests the code automatically.

The latest released version is at https://pypi.python.org/pypi/delocate

Support

Please put up issues on the Delocate issue tracker.