Skip to main content
https://travis-ci.org/matthew-brett/delocate.svg?branch=master

Delocate

OSX utilities to:

  • find dynamic libraries imported from python extensions

  • copy needed dynamic libraries to directory within package

  • update OSX install_names and rpath to cause code to load from copies of libraries

Provides scripts:

  • delocate-listdeps – show libraries a tree depends on

  • delocate-path – copy libraries a tree depends on into the tree and relink

  • delocate-wheel – rewrite wheel having copied and relinked library dependencies into the wheel tree.

The problem

Let’s say you have built a wheel somewhere, but it’s linking to dynamic libraries elsewhere on the machine, so you can’t distribute it, because others may not have these same libraries. Here we analyze the dependencies for a scipy wheel:

$ delocate-listdeps scipy-0.14.0b1-cp34-cp34m-macosx_10_6_intel.whl
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

By default, this does not include libraries in /usr/lib and /System. See those too with:

$ delocate-listdeps --all scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
/usr/lib/libSystem.B.dylib
/usr/lib/libstdc++.6.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

The output tells me that scipy has picked up dynamic libraries from my homebrew installation of gfortran (as well as the system libs).

You can get a listing of the files depending on each of the libraries, using the --depending flag:

$ delocate-listdeps --depending scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib:
    scipy/interpolate/dfitpack.so
    scipy/special/specfun.so
    scipy/interpolate/_fitpack.so
    ...

A solution

We can fix like this:

$ delocate-wheel -w fixed_wheels -v scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
Fixing: scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
Copied to package .dylibs directory:
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

The -w flag tells delocate-wheel to output to a new wheel directory instead of overwriting the old wheel. -v (verbose) tells you what delocate-wheel is doing. In this case it has made a new directory in the wheel zipfile, named scipy/.dylibs. It has copied all the library dependencies that are outside the OSX system trees into this directory, and patched the python .so extensions in the wheel to use these copies instead of looking in /usr/local/Cellar/gfortran/4.8.2/gfortran/lib.

Check the links again to confirm:

$ delocate-listdeps --all fixed_wheels/scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
/usr/lib/libSystem.B.dylib
/usr/lib/libstdc++.6.0.9.dylib
@loader_path/../../../../.dylibs/libgcc_s.1.dylib
@loader_path/../../../../.dylibs/libgfortran.3.dylib
@loader_path/../../../../.dylibs/libquadmath.0.dylib
@loader_path/../../../.dylibs/libgcc_s.1.dylib
@loader_path/../../../.dylibs/libgfortran.3.dylib
@loader_path/../../../.dylibs/libquadmath.0.dylib
@loader_path/../../.dylibs/libgcc_s.1.dylib
@loader_path/../../.dylibs/libgfortran.3.dylib
@loader_path/../../.dylibs/libquadmath.0.dylib
@loader_path/../.dylibs/libgcc_s.1.dylib
@loader_path/../.dylibs/libgfortran.3.dylib
@loader_path/../.dylibs/libquadmath.0.dylib
@loader_path/libgcc_s.1.dylib
@loader_path/libquadmath.0.dylib

So - system dylibs the same, but the others moved into the wheel tree.

This makes the wheel more likely to work on another machine which does not have the same version of gfortran installed - in this example.

Checking required architectures

Current Python.org Python and the OSX system Python (/usr/bin/python) are both dual Intel architecture binaries. For example:

$ lipo -info /usr/bin/python
Architectures in the fat file: /usr/bin/python are: x86_64 i386

This means that wheels built for Python.org Python or system Python should also have i386 and x86_64 versions of the Python extensions and their libraries. It is easy to link Python extensions against single architecture libraries by mistake, and therefore get single architecture extensions and / or libraries. In fact my scipy wheel is one such example, because I inadvertently linked against the homebrew libraries, which are x86_64 only. To check this you can use the --require-archs flag:

$ delocate-wheel --require-archs=intel scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
Traceback (most recent call last):
File "/Users/mb312/.virtualenvs/delocate/bin/delocate-wheel", line 77, in <module>
    main()
File "/Users/mb312/.virtualenvs/delocate/bin/delocate-wheel", line 69, in main
    check_verbose=opts.verbose)
File "/Users/mb312/.virtualenvs/delocate/lib/python2.7/site-packages/delocate/delocating.py", line 477, in delocate_wheel
    "Some missing architectures in wheel")
delocate.delocating.DelocationError: Some missing architectures in wheel

The “intel” argument requires dual architecture extensions and libraries. You can see which extensions are at fault by adding the -v (verbose) flag:

$ delocate-wheel -w fixed_wheels --require-archs=intel -v scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
Fixing: scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
Required arch i386 missing from /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
Required arch i386 missing from /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
Required arch i386 missing from scipy/fftpack/_fftpack.so
Required arch i386 missing from scipy/fftpack/convolve.so
Required arch i386 missing from scipy/integrate/_dop.so
...

I need to rebuild this wheel to link with dual-architecture libraries.

Troubleshooting

When running delocate-wheel or its sister command delocate-path, you may get errors like this:

delocate.delocating.DelocationError: library "<long temporary path>/wheel/libme.dylib" does not exist

This happens when one of your libraries gives a library dependency with a relative path. For example, let’s say that some file in my wheel has this for the output of otool -L myext.so:

myext.so:
    libme.dylib (compatibility version 10.0.0, current version 10.0.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 60.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

The first line means that myext.so expects to find libme.dylib at exactly the path ./libme.dylib - the current working directory from which you ran the executable. The output should be something like:

myext.so:
    /path/to/libme.dylib (compatibility version 10.0.0, current version 10.0.0)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 60.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

To set the path to the library, the linker is using the install name id of the linked library. In this bad case, then otool -L libme.dylib will give something like:

libme.dylib (compatibility version 10.0.0, current version 10.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

where the first line is the install name id that the linker picked up when linking myext.so to libme.dylib. You job is to fix the build process so that libme.dylib has install name id /path/to/libme.dylib. This is not a problem specific to delocate; you will need to do this to make sure that myext.so can load libme.dylib without libme.dylib being in the current working directory. For CMAKE builds you may want to check out CMAKE_INSTALL_NAME_DIR.

Code

See https://github.com/matthew-brett/delocate

Released under the BSD two-clause license - see the file LICENSE in the source distribution.

travis-ci kindly tests the code automatically under Python 2.7, 3.3 and 3.4.

The latest released version is at https://pypi.python.org/pypi/delocate

Support

Please put up issues on the delocate issue tracker.

Metadata

Release files for delocate 0.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distributions (sdists)

Source distribution for delocate 0.6.1
File Size Uploaded
delocate-0.6.1.zip 92.5 kB Details
delocate-0.6.1.tar.gz 68.1 kB Details

Built distributions (wheels)

Table of built distributions (wheels) for delocate 0.6.1
File Interpreter ABI Platform
delocate-0.6.1-py33-none-any.whl Python 3.3 none any Details
delocate-0.6.1-py3-none-any.whl Python 3 none any Details
delocate-0.6.1-py2-none-any.whl Python 2 none any Details

Total release size: 410.6 kB

Release files / delocate-0.6.1.zip

Download URL delocate-0.6.1.zip
Size 92.5 kB
Tags Source
SHA-256 checksum
How to use checksums
a3ab8a6c903f51eb38f760bfdbb22aec70426e92d73531a0397d33b92ed96e16
BLAKE2b-256 checksum
How to use checksums
02ebde3dca441ca89680c9ad27cd6ccea2addb60c501853b2105c717e9845ab4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / delocate-0.6.1.tar.gz

Download URL delocate-0.6.1.tar.gz
Size 68.1 kB
Tags Source
SHA-256 checksum
How to use checksums
f04b0d66e2e5e39f69669691609fb2120cd2daf229733aec57ad07d24e5ce099
BLAKE2b-256 checksum
How to use checksums
93f1293f71b24ab7b1a2c7159dc169d1368c7dff2a66b66f7ee29fdab288a3ff
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / delocate-0.6.1-py33-none-any.whl

Download URL delocate-0.6.1-py33-none-any.whl
Size 83.3 kB
Tags Python 3.3
SHA-256 checksum
How to use checksums
813be41d1fb6b7cf44d9c751fdc03879372916aabd720ad7d0ed6f2a712fd61b
BLAKE2b-256 checksum
How to use checksums
75a94322977d23d9c01f891e2967488cc92492a952444d0fc65e1d0dc3648aae
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / delocate-0.6.1-py3-none-any.whl

Download URL delocate-0.6.1-py3-none-any.whl
Size 83.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4084f9c386f7f518e33237b4beb73b27e2c821193ca68877df798d29aeda55ee
BLAKE2b-256 checksum
How to use checksums
90c1c8a9f439d5971f11147f2329249cbc773f8c87b286a3aba6d5da1ccb64a7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / delocate-0.6.1-py2-none-any.whl

Download URL delocate-0.6.1-py2-none-any.whl
Size 83.3 kB
Tags Python 2
SHA-256 checksum
How to use checksums
17059b000952ae1e4ba49f5166868abc08457e57da9264f4f32537f40aca81e0
BLAKE2b-256 checksum
How to use checksums
f45ecb663da5fce487d249c7d86222736aae9d8058261191868b1ec8a5f167d0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release history Release notifications | RSS feed

0.13.0

2 release files

0.12.0

2 release files

0.11.0

2 release files

0.10.7

2 release files

0.10.6

2 release files

0.10.5

2 release files

0.10.4

2 release files

0.10.1

2 release files

0.10.0

2 release files

0.9.1

2 release files

0.9.0

1 release file

0.8.2

1 release file

0.8.1

1 release file

0.8.0

1 release file

0.7.4

1 release file

0.7.3

1 release file

0.7.2

1 release file

0.7.1

1 release file

0.7.0

1 release file

0.6.5

1 release file

0.6.4

3 release files

0.6.3

5 release files

0.6.2

5 release files

This release

0.6.1 This release

5 release files

0.6.0

4 release files

0.5.0

5 release files

0.4.0

5 release files

0.3.0

4 release files

0.2.1

4 release files

0.2.0

5 release files

0.1

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page