pex is a library for generating .pex (Python EXecutable) files which are
executable Python environments in the spirit of virtualenvs.
pex is an expansion upon the ideas outlined in
and makes the deployment of Python applications as simple as cp. pex files may even
include multiple platform-specific Python distributions, meaning that a single pex file
can be portable across Linux and OS X.
pex files can be built using the pex tool. Build systems such as Pants and Buck also
support building .pex files directly.
Still unsure about what pex does or how it works? Watch this quick lightning
talk: WTF is PEX?.
pex is licensed under the Apache2 license.
To install pex, simply
$ pip install pex
You can also build pex in a git clone using tox:
$ tox -e py27-package
$ cp dist/pex ~/bin
This builds a pex binary in dist/pex that can be copied onto your $PATH.
The advantage to this approach is that it keeps your Python environment as empty as
possible and is more in-line with what pex does philosophically.
Launch an interpreter with requests, flask and psutil in the environment:
$ pex requests flask 'psutil>2,<3'
Or instead freeze your current virtualenv via requirements.txt and execute it anywhere:
$ pex -r <(pip freeze) -o my_virtualenv.pex
Run webserver.py in an environment containing flask as a quick way to experiment:
$ pex flask -- webserver.py
Launch Sphinx in an ephemeral pex environment using the Sphinx entry point sphinx:main:
$ pex sphinx -e sphinx:main -- --help
Build a standalone pex binary into pex.pex using the pex console_scripts entry point:
$ pex pex -c pex -o pex.pex
You can also build pex files that use a specific interpreter type:
$ pex pex -c pex --python=pypy -o pypy-pex.pex
Most pex options compose well with one another, so the above commands can be
mixed and matched. For a full list of options, just type pex --help.
Integrating pex into your workflow
If you use tox (and you should!), a simple way to integrate pex into your
workflow is to add a packaging test environment to your tox.ini:
deps = pex
commands = pex . -o dist/app.pex
Then tox -e package will produce a relocateable copy of your application
that you can copy to staging or production environments.
More documentation about pex, building .pex files, and how .pex files work
is available at https://pex.readthedocs.io.
pex uses tox for test and development automation. To run
the test suite, just invoke tox:
If you don’t have tox, you can generate a pex of tox:
$ pex tox -c tox -o ~/bin/tox
- Repair Executor error handling for other classes of IOError/OSError. (#292)
- Fix bdist_pex –pex-args. (#285)
- Inherit user site with –inherit-path. (#284)
- Repair passing of stdio kwargs to PEX.run(). (#288)
- Fix bdist_pex interpreter cache directory. (#286)
- Normalize and edify subprocess execution. (#255)
- Don’t ignore exit codes when using setuptools entry points. (#280)
- Update cache dir when bdist_pex.run is called directly.
- Improve failure modes for os.rename() as used in distribution caching.
- Add README and supported python versions to PyPI description.
- Use open with utf-8 support.
- Add --pex-root option.
This release is a quick fix for a regression that inadvertently went out in 1.1.5 (Issue #243).
- Fix PEXBuilder.clone and thus bdist_pex --pex-args for --python and --python-shebang.
- Fix old pkg_resources egg version normalization.
- Fix the inherit_path handling.
- Fix handling of bad distribution script names when used as the pex entrypoint.
This release is a quick fix for a regression that inadvertently went out in 1.1.3 (Issue #216).
- Add a test for the regression in FixedEggMetadata._zipinfo_name and revert the breaking commit.
This release includes an initial body of work towards Windows support, ABI tag support for CPython 2.x and a fix for version number normalization.
- Add python 2.x abi tag support.
- Add .idea to .gitignore.
- Don’t normalize version numbers as names.
- More fixes for windows.
- Fixes to get pex to work on windows.
- Bump setuptools & wheel version pinning.
- Unescape html in PageParser.href_match_to_url.
- Memoize calls to Crawler.crawl() for performance win in find-links based resolution.
- Fix infinite recursion when PEX_PYTHON points at a symlink.
- Add /etc/pexrc to the list of pexrc locations to check.
- Improve error messaging for platform constrained Untranslateable errors.
- Add support for .pexrc files for influencing the pex environment. See the notes here.
- Bug fix: PEX_PROFILE_FILENAME and PEX_PROFILE_SORT were not respected.
- Adds the bdist_pex command to setuptools.
- Bug fix: We did not normalize package names in ResolvableSet, so it was possible to depend on
sphinx and Sphinx-1.4a0.tar.gz and get two versions build and included into the pex.
- Adds a pex-identifying User-Agent. #101.
- Bug fix: Accommodate OSX Python python binaries. Previously the OSX python distributions shipped
with OSX, XCode and available via https://www.python.org/downloads/ could fail to be detected using
the PythonInterpreter class.
- Bug fix: PEX_SCRIPT failed when the script was from a not-zip-safe egg.
Original PR #139.
- Bug fix: sys.exit called without arguments would cause None to be printed on stderr since pex 1.0.1.
- Bug fix: PEX-INFO values were overridden by environment Variables with default values that were
not explicitly set in the environment.
- Bug fix: Since 69649c1 we have been unpatching
the side-effects of sys.modules after PEX.execute. This takes all modules imported during
the PEX lifecycle and sets all their attributes to None. Unfortunately, sys.excepthook,
atexit and __del__ may still try to operate using these tainted modules, causing exceptions
on interpreter teardown. This reverts just the sys unpatching so that the abovementioned
teardown hooks behave more predictably.
- Allow PEXBuilder to optionally copy files into the PEX environment instead of hard-linking them.
- Allow PEXBuilder to optionally skip precompilation of .py files into .pyc files.
- Bug fix: PEXBuilder did not respect the target interpreter when compiling source to bytecode.
- Bug fix: Fix complex resolutions when using a cache.
The 1.0.0 release of pex introduces a few breaking changes: pex -r now takes requirements.txt files
instead of requirement specs, pex -s has now been removed since source specs are accepted as arguments,
and pex -p has been removed in favor of its alias pex -o.
The pex command line interface now adheres to semver insofar as backwards incompatible CLI
changes will invoke a major version change. Any backwards incompatible changes to the PEX
environment variable semantics will also result in a major version change. The pex API adheres
to semver insofar as backwards incompatible API changes will invoke minor version changes.
For users of the PEX API, it is recommended to add minor version ranges, e.g. pex>=1.0,<1.1.
For users of the PEX CLI, major version ranges such as pex>=1,<2 should be sufficient.
- BREAKING CHANGE: Removes the -s option in favor of specifying directories directly as
arguments to the pex command line.
- BREAKING CHANGE: pex -r now takes requirements.txt filenames and not requirement
specs. Requirement specs are now passed as arguments to the pex tool. Use -- to escape
command line arguments passed to interpreters spawned by pex. Implements
- Adds a number of flag aliases to be more compatible with pip command lines: --no-index,
-f, --find-links, --index-url, --no-use-wheel. Removes -p in favor of
- Adds --python-shebang option to the pex tool in order to set the #! shebang to an exact
- Adds support for PEX_PYTHON environment variable which will cause the pex file to reinvoke
itself using the interpreter specified, e.g. PEX_PYTHON=python3.4 or
- Adds support for PEX_PATH environment variable which allows merging of PEX environments at
runtime. This can be used to inject plugins or entry_points or modules from one PEX into
another without explicitly building them together. #30.
- Consolidates documentation of PEX_ environment variables and adds the --help-variables option
to the pex client. Partially addresses #13.
- Adds helper method to dump a package subdirectory onto disk from within a zipped PEX file. This
can be useful for applications that know they’re running within a PEX and would prefer some
static assets dumped to disk instead of running as an unzipped PEX file.
- Now supports extras for static URLs and installable directories.
- Adds -m and --entry-point alias to the existing -e option for entry points in
the pex tool to evoke the similarity to python -m.
- Adds console script support via -c/--script/--console-script and PEX_SCRIPT. This allows
you to reference the named entry point instead of the exact module:name pair. Also supports
scripts defined in the scripts section of setup.py.
- Adds more debugging information when encountering unresolvable requirements.
- Bug fix: PEX_COVERAGE and PEX_PROFILE did not function correctly when SystemExit was raised.
- Bug fix: Fixes caching in the PEX tool since we don’t cache the source distributions of installable
This is the last release before the 1.0.0 development branch is started.
- Change the setuptools range to >=2.2,<16 by handling EntryPoint changes as well as
being flexible on whether pkg_resources is a package or a module. Fixes
- Adds option groups to the pex tool to make the help output slightly more readable.
- Bug fix: Make pip install pex work better by removing extras_requires on the
console_script entry point. Fixes #48
- New feature: Adds an interpreter cache to the pex tool. If the user does not explicitly
disable the wheel feature and attempts to build a pex with wheels but does not have the wheel
package installed, pex will download it in order to make the feature work.
Implements #47 in order to
- Bug fix: Honor installed sys.excepthook in pex teardown.
- Bug fix: UrllibContext used replace as a keyword argument for bytes.decode
but this only works on Python 3. Pull Request #46
- Bug fix: Fixup string formatting in pex/bin/pex.py to support Python 2.6
Pull Request #40
- Performance improvement: Speed up the best-case scenario of dependency resolution.
- Bug fix: Change from uuid4().get_hex() to uuid4().hex to maintain Python3
compatibility of pex.common.
Pull Request #39
- Bug fix: Actually cache the results of translation. Previously bdist translations
would be created in a temporary directory even if a cache location was specified.
- Bug fix: Support all potential abi tag permutations when determining platform
Pull Request #33
- Performance improvement: Don’t always write packages to disk if they’ve already been
cached. This can significantly speed up launching PEX files with a large
number of non-zip-safe dependencies.
- Bug fix: Allow pex 0.8.x to parse pex files produced by earlier versions of
pex and twitter.common.python.
- Pin pex to setuptools prior to 9.x until we have a chance to make changes
related to PEP440 and the change of pkg_resources.py to a package.
- Bug fix: Fix issue where it’d be possible to os.path.getmtime on a remote Link object
- API change: Decouple translation from package iteration. This removes
the Obtainer construct entirely, which likely means if you’re using PEX as
a library, you will need to change your code if you were doing anything
nontrivial. This adds a couple new options to resolve but simplifies
the story around how to cache packages.
- Refactor http handling in pex to allow for alternate http implementations. Adds support
improving both performance and security. For more information, read the commit notes at
- Improvements to API documentation throughout.
- Renamed Tracer to TraceLogger to prevent nondeterministic isort ordering.
- Refactor tox.ini to increase the number of environment combinations and improve coverage.
- Adds HTTP retry support for the RequestsContext.
- Make pex –version correct.
- Bug fix: Fix over-aggressive sys.modules scrubbing for namespace packages. Under
certain circumstances, namespace packages in site-packages could conflict with packages
within a PEX, causing them to fail importing.
- Bug fix: Replace uses of os.unsetenv(...) with del os.environ[...]
Pull Request #11
- Bug fix: Scrub sys.path and sys.modules based upon both supplied path and
realpath of files and directories. Newer versions of virtualenv on Linux symlink site-packages
which caused those packages to not be removed from sys.path correctly.
- Bug fix: The pex -s option was not correctly pulling in transitive dependencies.
- Bug fix: Adds content method to HTTP contexts that does HTML content decoding, fixing
an encoding issue only experienced when using Python 3.
- Rename twitter.common.python to pex and split out from the
- Change the interpretation of -i (and of PyPIFetcher’s pypi_base)
to match pip’s -i. This is useful for compatibility with devpi.
Ensures that .egg/.whl distributions on disk have their mtime updated
even though we no longer overwrite them. This gives them a new time
lease against their ttl.
Without this change, once a distribution aged past the ttl it would
never be used again, and builds would re-create the same distributions
in tmpdirs over and over again.
Fixes an issue where SourceTranslator would overwrite .egg/.whl
distributions already on disk. Instead it should always check to see if
a copy already exists and reuse if there.
This ordinarily should not be a problem but the zipimporter caches
metadata by filename instead of stat/sha, so if the underlying contents
changed a runtime error would be thrown due to seemingly corrupt zip file
offsets. RB #684
- Adds -i/--index option to the pex tool.
- Adds twitter.common.python.pex_bootstrap bootstrap_pex_env function in
order to initialize a PEX environment from within a python interpreter.
(Patch contributed by @kwlzn)
- Adds stdin=,stdout=,stderr= keyword parameters to the PEX.run function.
(Patch from @benjy)
- The crawler now defaults to not follow links for security reasons.
(Before the default behavior was to implicitly --follow-links for all
requirements.) RB #293
- Improves scrubbing of site-packages from PEX environments.
0.5.1 - 0.5.4
- Silences exceptions reported during interpreter teardown (the exceptions
resulting from incorrect atexit handler behavior) introduced by 0.4.3
- Adds __hash__ to Link so that Packages are hashed correctly in
- Adds twitter.common.python.finders which are additional finders for
- find eggs within a .zip
- find wheels within a directory
- find wheels within a .zip
- Adds a new Package abstraction by refactoring Link into Link and Package.
- Adds support for PEP425 tagging necessary for wheel support.
- Improves python environment isolation by correctly scrubbing namespace
packages injected into module __path__ attributes by nspkg pth files.
- Adds twitter.common.python.resolver resolve method that handles
transitive dependency resolution better. This means that if the
requirement futures==2.1.2 and an unqualified futures>=2 is pulled in
transitively, our resolver will correctly resolve futures 2.1.2 instead
of reporting a VersionConflict if any version newer than 2.1.2 is
available. RB #129
- Factors all twitter.common.python test helpers into
- Bug fix: Fix OrderedSet atexit exceptions
- Bug fix: Fix cross-device symlinking (patch from @benjy)
- Bug fix: Raise a RuntimeError if we fail to write pkg_resources into a .pex
- twitter.common.python is no longer a namespace package
- Kill the egg distiller. We now delegate .egg generation to bdist_egg.
- Short-circuit resolving a distribution if a local exact match is found.
- Correctly patch the global pkg_resources WorkingSet for the lifetime
of the Python interpreter. RB #52
- Fixes a performance regression in setuptools build_zipmanifest
Setuptools Issue #154
- Plumb through the --zip-safe, --always-write-cache, --ignore-errors
and --inherit-path flags to the pex tool.
- Delete the unused PythonDirWrapper code.
- Split PEXEnvironment resolution into twitter.common.python.environment
and deconflate WorkingSet/Environment state.
- Removes the monkeypatched zipimporter in favor of keeping all eggs
unzipped within PEX files. Refactors the PEX dependency cache in
- Adds interpreter detection for Jython and PyPy.
- Dependency translation errors should be made uniform.
(Patch from @johnsirois)
- Adds PEX_PROFILE_ENTRIES to limit the number of entries reported when
PEX_PROFILE is enabled. (Patch from @rgs_)
- Bug fix: Several fixes to error handling in twitter.common.python.http
(From Marc Abramowitz)
- Bug fix: PEX should not always assume that $PATH was available.
(Patch from @jamesbroadhead)
- Bug fix: Filename should be part of the .pex cache key or else multiple
identical versions will incorrectly resolve (Patch from @tc)
- Bug fix: Executed entry points shouldn’t be forced to run in an
environment with __future__ imports enabled. (Patch from
- Bug fix: Detect versionless egg links and fail fast. (Patch from
- Bug fix: Handle setuptools>=2.1 correctly in the zipimport monkeypatch
(Patch from @johnsirois.)
- Bug fix: Fix handling of Fetchers with file:// urls.
- Adds the pex tool as a standalone tool.
- Bug fix: Bootstrapped twitter.common.python should declare twitter.common
as a namespace package.
- Make twitter.common.python fully standalone by consolidating
external dependencies within twitter.common.python.common.
- Initial published version of twitter.common.python.