Build and publish crates with pyo3, rust-cpython and cffi bindings as well as rust binaries as python packages
Project description
Pyo3-pack
Build and publish crates with pyo3, rust-cpython and cffi bindings as well as rust binaries as python packages.
This project is meant as a zero configuration replacement for setuptools-rust and milksnake. It supports building wheels for python 3.5+ on windows, linux, mac and freebsd, can upload them to pypi and has basic pypy support.
Usage
You can either download binaries from the latest release or install it with pip:
pip install pyo3-pack
There are three main commands:
pyo3-pack publish
builds the crate into python packages and publishes them to pypi.pyo3-pack build
builds the wheels and stores them in a folder (target/wheels
by default), but doesn't upload them.pyo3-pack develop
builds the crate and install it's as a python module directly in the current virtualenv.
pyo3
and rust-cpython
bindings are automatically detected, for cffi or binaries you need to pass -b cffi
or -b bin
. pyo3-pack doesn't needs extra configuration files and doesn't clash with an existing setuptools-rust or milksnake configuration. You can even integrate it with testing tools such as tox. There are examples for the different bindings in the test-crates
folder.
The name of the package will be the name of the cargo project, i.e. the name field in the [package]
section of Cargo.toml. The name of the module, which you are using when importing, will be the name
value in the [lib]
section (which defaults to the name of the package). For binaries it's simply the name of the binary generated by cargo.
pyo3 and rust-cpython
For pyo3 and rust-cpython, pyo3-pack can only build packages for installed python versions. On linux and mac, all python versions in PATH
are used. If you don't set your own interpreters with -i
, a heuristic is used to search for python installations. On windows all versions from the python launcher (which is installed by default by the python.org installer) and all conda environments except base are used. You can check which versions are picked up with the list-python
subcommand.
pyo3 will set the used python interpreter in the environment variable PYTHON_SYS_EXECUTABLE
, which can be used from custom build scripts.
Cffi
Cffi wheels are compatible with all python versions, but they need to have cffi
installed for the python used for building (pip install cffi
).
pyo3-pack will utilize cbindgen to generate a header file. To customize the header file you can either configure cbindgen through a cbindgen.toml file inside your project root or write a setup a build script which writes a header file to $PROJECT_ROOT/target/header.h
.
Based on these header file pyo3-pack is going to generated a module which exports a ffi
and a lib
object.
Example of a custom build script
use cbindgen; // Use `extern crate cbindgen` in rust 2015
use std::env;
use std::path::Path;
fn main() {
let crate_dir = env::var("CARGO_MANIFEST_DIR").unwrap();
let bindings = cbindgen::Builder::new()
.with_no_includes()
.with_language(cbindgen::Language::C)
.with_crate(crate_dir)
.generate()
.unwrap();
bindings.write_to_file(Path::new("target").join("header.h"));
}
Mixed rust/python projects
To create a mixed rust/python project, create a folder with your module name (i.e. lib.name
in Cargo.toml) next to your Cargo.toml and add your python sources there:
my-project
├── Cargo.toml
├── my_project
│ ├── __init__.py
│ └── bar.py
├── Readme.md
└── src
└── lib.rs
pyo3-pack will add the native extension as a module in your python folder. When using develop, pyo3-pack will copy the native library and for cffi also the glue code to your python folder. You should add those files to your gitignore.
With cffi you can do from .my_project import lib
and then use lib.my_native_function
, with pyo3/rust-cpython you can directly from .my_project import my_native_function
.
Example layout with pyo3 after pyo3-pack develop
:
my-project
├── Cargo.toml
├── my_project
│ ├── __init__.py
│ ├── bar.py
│ └── my_project.cpython-36m-x86_64-linux-gnu.so
├── Readme.md
└── src
└── lib.rs
Python metadata
To specifiy python dependecies, add a list requires-dist
in a [package.metadata.pyo3-pack]
section in the Cargo.toml. This list is equivalent to install_requires
in setuptools:
[package.metadata.pyo3-pack]
requires-dist = ["flask~=1.1.0", "toml==0.10.0"]
Pip allows adding so called console scripts, which are shell commands that execute some function in you program. You can add console scripts in a section [package.metadata.pyo3-pack.scripts]
. The keys are the script names while the values are the path to the function in the format some.module.path:class.function
, where the class
part is optional. The function is called with no arguments. Example:
[package.metadata.pyo3-pack.scripts]
get_42 = "my_project:DummyClass.get_42"
You can also specify trove classifiers in your Cargo.toml under package.metadata.pyo3-pack.classifier
:
[package.metadata.pyo3-pack]
classifier = ["Programming Language :: Python"]
You can use other fields from the python core metadata in the [package.metadata.pyo3-pack]
section, specifically maintainer
, maintainer-email
and requires-python
(string fields), as well as requires-external
, project-url
and provides-extra
(lists of strings).
pyproject.toml
pyo3-pack supports building through pyproject.toml. To use it, create a pyproject.toml
next to your Cargo.toml
with the following content:
[build-system]
requires = ["pyo3-pack"]
build-backend = "pyo3_pack"
If a pyproject.toml
with a [build-system]
entry is present, pyo3-pack will build a source distribution (sdist) of your package, unless --no-sdist
is specified. The source distribution will contain the same files as cargo package
.
You can then e.g. install your package with pip install .
. With pip install . -v
you can see the output of cargo and pyo3-pack.
You can use the options manylinux
, skip-auditwheel
, bindings
, strip
, cargo-extra-args
and rustc-extra-args
under [tool.pyo3-pack]
the same way you would when running pyo3-pack directly. The bindings
key is required for cffi and bin projects as those can't be automatically detected. Currently, all build are in release mode (see this thread for details).
For a non-manylinux build you could with cffi bindings you could use the following:
[build-system]
requires = ["pyo3-pack"]
build-backend = "pyo3_pack"
[tool.pyo3-pack]
bindings = "cffi"
manylinux = "off"
Using tox with build isolation is currently blocked by a tox bug (tox-dev/tox#1344). There's a cargo sdist
command for only building a source distribution as workaround for pypa/pip#6041.
Manylinux and auditwheel
For portability reasons, native python modules on linux must only dynamically link a set of very few libraries which are installed basically everywhere, hence the name manylinux. The pypa offers a special docker container and a tool called auditwheel to ensure compliance with the manylinux rules.
pyo3-pack contains a reimplementation of a major part of auditwheel automatically checking the generated library. If you want to disable those checks or build for native linux target, use the --manylinux
flag.
For full manylinux compliance you need to compile in a cent os 5 docker container. The konstin2/pyo3-pack image is based on the official manylinux image. You can use it like this:
docker run --rm -v $(pwd):/io konstin2/pyo3-pack build
pyo3-pack itself is manylinux compliant when compiled for the musl target. The binaries on the release pages have additional keyring integration (through the password-storage
feature), which is not manylinux compliant.
PyPy
pyo3-pack can build wheels for pypy with pyo3. Note that pypy is not compatible with manylinux1 and you can't publish pypy wheel to pypi pypy has been only tested manually and on linux. See #115 for more details.
Build
FLAGS:
-h, --help
Prints help information
--no-sdist
Don't build a source distribution
--release
Pass --release to cargo
--skip-auditwheel
[deprecated, use --manylinux instead] Don't check for manylinux compliance
--strip
Strip the library for minimum file size
-V, --version
Prints version information
OPTIONS:
-m, --manifest-path <PATH>
The path to the Cargo.toml [default: Cargo.toml]
--target <TRIPLE>
The --target option for cargo
-b, --bindings <bindings>
Which kind of bindings to use. Possible values are pyo3, rust-cpython, cffi and bin
--cargo-extra-args <cargo_extra_args>...
Extra arguments that will be passed to cargo as `cargo rustc [...] [arg1] [arg2] --`
Use as `--cargo-extra-args="--my-arg"`
-i, --interpreter <interpreter>...
The python versions to build wheels for, given as the names of the interpreters. Uses autodiscovery if not
explicitly set.
--manylinux <manylinux>
Control the platform tag on linux.
- `1`: Use the manylinux1 tag and check for compliance
- `1-unchecked`: Use the manylinux1 tag without checking for compliance
- `2010`: Use the manylinux2010 tag and check for compliance
- `2010-unchecked`: Use the manylinux1 tag without checking for compliance
- `off`: Use the native linux tag (off)
This option is ignored on all non-linux platforms [default: 1] [possible values: 1, 1-unchecked, 2010,
2010-unchecked, off]
-o, --out <out>
The directory to store the built wheels in. Defaults to a new "wheels" directory in the project's target
directory
--rustc-extra-args <rustc_extra_args>...
Extra arguments that will be passed to rustc as `cargo rustc [...] -- [arg1] [arg2]`
Use as `--rustc-extra-args="--my-arg"`
Publish
FLAGS:
--debug
Do not pass --release to cargo
-h, --help
Prints help information
--no-sdist
Don't build a source distribution
--no-strip
Strip the library for minimum file size
--skip-auditwheel
[deprecated, use --manylinux instead] Don't check for manylinux compliance
-V, --version
Prints version information
OPTIONS:
-m, --manifest-path <PATH>
The path to the Cargo.toml [default: Cargo.toml]
--target <TRIPLE>
The --target option for cargo
-b, --bindings <bindings>
Which kind of bindings to use. Possible values are pyo3, rust-cpython, cffi and bin
--cargo-extra-args <cargo_extra_args>...
Extra arguments that will be passed to cargo as `cargo rustc [...] [arg1] [arg2] --`
Use as `--cargo-extra-args="--my-arg"`
-i, --interpreter <interpreter>...
The python versions to build wheels for, given as the names of the interpreters. Uses autodiscovery if not
explicitly set.
--manylinux <manylinux>
Control the platform tag on linux.
- `1`: Use the manylinux1 tag and check for compliance
- `1-unchecked`: Use the manylinux1 tag without checking for compliance
- `2010`: Use the manylinux2010 tag and check for compliance
- `2010-unchecked`: Use the manylinux1 tag without checking for compliance
- `off`: Use the native linux tag (off)
This option is ignored on all non-linux platforms [default: 1] [possible values: 1, 1-unchecked, 2010,
2010-unchecked, off]
-o, --out <out>
The directory to store the built wheels in. Defaults to a new "wheels" directory in the project's target
directory
-p, --password <password>
Password for pypi or your custom registry. Note that you can also pass the password through
PYO3_PACK_PASSWORD
-r, --repository-url <registry>
The url of registry where the wheels are uploaded to [default: https://upload.pypi.org/legacy/]
--rustc-extra-args <rustc_extra_args>...
Extra arguments that will be passed to rustc as `cargo rustc [...] -- [arg1] [arg2]`
Use as `--rustc-extra-args="--my-arg"`
-u, --username <username>
Username for pypi or your custom registry
Develop
FLAGS:
-h, --help
Prints help information
--release
Pass --release to cargo
--strip
Strip the library for minimum file size
-V, --version
Prints version information
OPTIONS:
-b, --binding-crate <binding_crate>
Which kind of bindings to use. Possible values are pyo3, rust-cpython, cffi and bin
--cargo-extra-args <cargo_extra_args>...
Extra arguments that will be passed to cargo as `cargo rustc [...] [arg1] [arg2] --`
Use as `--cargo-extra-args="--my-arg"`
-m, --manifest-path <manifest_path>
The path to the Cargo.toml [default: Cargo.toml]
--rustc-extra-args <rustc_extra_args>...
Extra arguments that will be passed to rustc as `cargo rustc [...] -- [arg1] [arg2]`
Use as `--rustc-extra-args="--my-arg"`
Code
The main part is the pyo3-pack library, which is completely documented and should be well integratable. The accompanying main.rs
takes care username and password for the pypi upload and otherwise calls into the library.
The sysconfig
folder contains the output of python -m sysconfig
for different python versions and platform, which is helpful during development.
You need to install cffi
(pip install cffi
) to run the tests.
There are two optional hacks that can speed up the tests (over 80s to 17s on my machine). By running cargo build --release --manifest-path test-crates/cargo-mock/Cargo.toml
you can activate a cargo cache avoiding to rebuild the pyo3 test crates with every python vesion. Delete target/test-cache
to clear the cache (e.g. after changing a test crate) or remove test-crates/cargo-mock/target/release/cargo
to deactive it. By running the tests with the faster-tests
feature, binaries are stripped and wheels are only stored and not compressed.
You might want to have look into my blog post which explains the intricacies of building native python packages.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyo3_pack-0.7.0_beta.11-py2.py3-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e8a3386f4d3a24d453f01f11f246a2f6398afedfb0bd38d390890328cafbe247 |
|
MD5 | a6680c28d250d85c2e2b268b90640c6d |
|
BLAKE2b-256 | a0ed8b9bbd341e7d43e81bd5ac233d4c295f9564292fbe59c24799169b687dfd |
Hashes for pyo3_pack-0.7.0_beta.11-py2.py3-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf23ba596d73f6c6ecb731e58bfa637ac2237abbe32da1fb4695edfa361b182c |
|
MD5 | 5e6b72b613213d14893b271955354eef |
|
BLAKE2b-256 | 96fec2a4f762cd2c8b801522b491432ba19a13714664b8a85bd2ffb8361ce419 |
Hashes for pyo3_pack-0.7.0_beta.11-py2.py3-none-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1fc44a2709d01a5b11c00ad1f9226b2bd3a86dea5c9802d85c4cab51ca04a265 |
|
MD5 | 664036fe72ae2a49fff2fd82d1a7f55e |
|
BLAKE2b-256 | 1571687257543578d4e60866ad163a1b9aaba9dc935e57707c6dee693b6d39cf |
Hashes for pyo3_pack-0.7.0_beta.11-py2.py3-none-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5fce6f61143980ab7f5a0f16762c477673fa502fa7b1d4b0e1051b90089605db |
|
MD5 | ed364366bca09f2a5c763e41d7c903b7 |
|
BLAKE2b-256 | 830e48b801a5c38acb7a8840256c61de9ff73d7709dc17605f16a8ae2bd26408 |
Hashes for pyo3_pack-0.7.0_beta.11-py2.py3-none-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b2fd3468fba07b820427d66d8592e2d0974230d50882702530f43ec3e4d78a5 |
|
MD5 | f7b1a735850feca83ad783b30559fdc7 |
|
BLAKE2b-256 | 9359c4d06160aedf08a5b98c546c4ed09e970b08ed66437c33829ac465ff1851 |