rustimport 1.3.0

Import Rust files directly from Python!

rustimport - Import Rust directly from Python!

rustimport was heavily inspired by and is partly based upon cppimport. Check it out if you're interested in the same functionality for C and C++!

Installation

Install with pip install rustimport.

A quick example

Save the Rust code below as somecode.rs.

// rustimport:pyo3

use pyo3::prelude::*;

#[pyfunction]
fn square(x: i32) -> i32 {
    x * x
}

Then open a Python interpreter and import the Rust extension:

>>> import rustimport.import_hook
>>> import somecode  # This will pause for a moment to compile the module
>>> somecode.square(9)
81

Hurray, you've called some Rust code from Python using a combination of rustimport and pyo3

This workflow enables you to edit both Rust files and Python and recompilation happens automatically and transparently! It's also handy for quickly whipping together an optimized version of a slow Python function.

To easily create a new single-file extension (like above), or a complete crate, use the provided tool:

$ python3 -m rustimport new my_single_file_extension.rs
# or create a full rust crate:
$ python3 -m rustimport new my_crate

And import it from Python:

>>> import rustimport.import_hook
>>> import my_single_file_extension, my_crate
>>> my_single_file_extension.say_hello()
Hello from my_single_file_extension, implemented in Rust!
>>> my_crate.say_hello()
Hello from my_crate, implemented in Rust!

Smooth!

An explanation

Okay, now that I've hopefully convinced you on how exciting this is, let's get into the details of how to do this yourself. First, the comment at top is essential to opt in to rustimport. Don't forget this! (See below for an explanation of why this is necessary.)

// rustimport:pyo3

The bulk of the file is a generic, simple pyo3 extension. We use the pyo3 crate, then define a simple function that squares x, and rustimport takes care of exporting that function as part of a Python extension called somecode.

Templating & Pre-Processing

rustimport offers several layers of customization. This is archieved through a simple pre-processor and templates (well, the only existing template at the moment is pyo3 - pull requests welcome :D).

What rustimport did for you in the background

The first example in this Readme is the simplest possible form of using rustimport. You just tell rustimport to use the pyo3 template by writing rustimport:pyo3 in the first line, and define a function annotated with pyo3's #[pyfunction] macro. In the background, rustimport handled a lot of stuff for you:

1. It set up a minimalistic folder structure for a rust crate with your source code in a temporary location.
2. It generated a Cargo.toml file with the basic configuration for pyo3 and your extension:

[package]
name = "somecode"
version = "0.1.0"
edition = "2021"

[lib]
name = "somecode"
crate-type = [ "cdylib",]

[dependencies.pyo3]
version = "0.16.2"
features = [ "extension-module",]

3. It generated a code block exporting your method and appended it to the end of your file:

#[pymodule]
fn somecode(_py: Python, m: &PyModule) -> PyResult<()> {
  m.add_function(wrap_pyfunction!(square, m)?)?;
  Ok(())
}

Customizing an extension

You can do all the above yourself. rustimport will detect that and only fill in the missing parts to make your extension work.

1. Extending Cargo.toml

For example, to add additional contents to the generated Cargo.toml file, use the special //: comment syntax at the top of your .rs file:

// rustimport:pyo3

// Set the library's version and add a dependency:
//: [package]
//: version = "1.2.3"
//:
//: [dependencies]
//: rand = "0.8.5"

use rand::Rng;

#[pyfunction]
fn myfunc() {
    println!("{}", rand::thread_rng().gen_range(0..10))
}

2.Tracking additional source files

To track additional files for changes, use the special //d: comment syntax:

//d: ../other-folder/somefile.rs
//d: ../*.rs
//d: ./my-directory/**/*.json

// --snip--

rustimport will now track files matching these patterns too and re-compiles your extension if any of them changes.

3. Full customization for more control

If you write a more complex extension, it's preferrable to just create a normal Rust crate:

$ python3 -m rustimport new my_crate
$ tree my_crate
my_crate
├── Cargo.toml
├── .rustimport
└── src
    └── lib.rs

The crate contains all necessary configuration to be directly be imported by rustimport and also some additional explanations on how to configure manually.

Using environment variables

rustimport exposed most of it's settings via environment variables. This can be pretty handy in development but also in production environments.

For example, to force recompilation, just run your script like this:

RUSTIMPORT_FORCE_REBUILD=true python my_script.py

Or to instruct the compiler to optimize binaries (for example to examine the real performance boost rust gives you) run it like this:

RUSTIMPORT_RELEASE_BINARIES=true python my_script.py

Take a look at settings.py for all available environment variables.

Usage in production

1. Building release binaries

In production deployments you usually don't want to include the Rust toolchain, all the sources and compile at runtime. Therefore, a simple cli utility for pre-compiling all source files is provided. This utility may, for example, be used in CI/CD pipelines.

Usage is as simple as

python -m rustimport build --release

This will build a release-optimized binary for all *.rs files and Rust crates in the current directory (and it's subdirectories) if they are eligible to be imported (i.e. contain the // rustimport comment in the first line or a .rustimport file in case of a crate).

Alternatively, you may specifiy one or more root directories or source files to be built:

python -m rustimport build --release ./my/root/folder/ ./my/single/file.rs ./my/crate/

Note: When specifying a path to a file, the header check (// rustimport) is skipped for that file.

2. Toggling release mode on

To further improve startup performance for production builds, you can opt-in to skip the checksum and compiled binary existence checks during importing by either setting the environment variable RUSTIMPORT_RELEASE_MODE to true or setting the configuration from within Python:

rustimport.settings.release_mode = True

This essentially just disables the import hook and uses the standard python utilities to import the pre-compiled binary.

Warning: Make sure to have all binaries pre-compiled with when in release mode, as importing any missing ones will cause exceptions.

In case you would, for whatever reason, like the binaries to be checked and built in production too, set rustimport.settings.compile_release_binaries to True to use release-optimized binaries.

Frequently asked questions

What's actually going on?

Sometimes Python just isn't fast enough. Or you have existing code in a Rust crate. So, you write a Python extension module, a library of compiled code. I recommend pyo3 for Rust to Python bindings.

I discovered that my productivity is slower when my development process goes from Edit -> Test in just Python to Edit -> Compile -> Test in Python plus Rust. So, rustimport combines the process of compiling and importing an extension in Python so that you can just run import foobar and not have to worry about multiple steps. Internally, rustimport looks for a file foobar.rs or a Rust crate (discovered through foobar/Cargo.toml). Assuming one is found, the comments at it's beginning are parsed for either a template (rustimport:pyo3) or a cargo manifest, then it's compiled and loaded as an extension module.

Does rustimport recompile every time a module is imported?

No! Compilation should only happen the first time the module is imported. The Rust source is compared with a checksum on each import to determine if any relevant file has changed. Additional dependencies can be tracked by adding to the header comments:

//d: ../myothercrate/**/*.rs
//d: ../myothercrate/Cargo.toml

By default, rustimport tracks all *.rs files as well as Cargo.toml and Cargo.lock for crates and no additional dependencies for single-file Rust extensions.

rustimport isn't doing what I want, can I get more verbose output?

rustimport uses the standard Python logging tools. Thus, you can enable logging like this:

import logging
logging.basicConfig(level=logging.DEBUG)  # or logging.INFO for a bit less verbosity
# ... do some rustimport stuff here

It's fast, but can it get even faster?

To create release-optimized binaries, set

rustimport.settings.compile_release_binaries = True

Or set the environment variable RUSTIMPORT_RELEASE_BINARIES to true

Compilation might be a little bit slower now due to rust's optimization mechanisms, but at runtime the extension is significantly faster in most cases.

How can I force a rebuild even when the checksum matches?

Set:

rustimport.settings.force_rebuild = True

Or set the environment variable RUSTIMPORT_FORCE_REBUILD to true

And if this is a common occurrence, I would love to hear your use case and why the normal dependency tracking is insufficient!

Can I use something else than pyo3?

Sure! Though I recommend using pyo3 due to it's simplicity, you're completely free to use any other library, for example rust-cpython.

There is an example using rust-cpython in examples/doublecount.rs

How can I make compilation faster?

Compilation happens incrementally by default. That is, the first compilation might take a bit, but subsequent ones are usually much faster.

rustimport uses a temporary directory for caching, which is deleted after a reboot on most systems. Therefore it might be beneficial for you to set a custom cache directory to have a more permanent cache:

rustimport.settings.cache_dir = os.path.realpath("./.rust-cache")

Or - you guessed it - use the RUSTIMPORT_CACHE_DIR environment variable.

If this directory doesn't exist, it will be created automatically by rustimport.

Apple, huh?

Yes, macOS is supported. No additional config should be necessary for pyo3 as the required linker args are set automatically by rustimport.

Why does the import hook need "rustimport" on the first line of the .rs file?

Modifying the Python import system is a global modification and thus affects all imports from any other package. As a result, when cppimport was first implemented, other packages (e.g. scipy) suddenly started breaking because import statements internal to those packages were importing C or C++ files instead of the modules they were intended to import. To avoid this failure mode, the import hook uses an "opt in" system where C and C++ files can specify they are meant to be used with cppimport by having a comment on the first line that includes the text "cppimport".

rustimport has adopted from this and follows the same pattern. Since rustimport also supports importing whole crates, an additional mechanism was necessary to make that work in the same fashion: You can either create a .rustimport file in the crate's root folder (next to Cargo.toml) or, alternatively, add a # rustimport comment to Cargo.tomls first line.

As an alternative to the import hook, you can use imp or imp_from_path. The rustimport.imp and rustimport.imp_from_path performs exactly the same operation as the import hook but in a slightly more explicit way:

foobar = rustimport.imp("foobar")
foobar = rustimport.imp_from_path("./some/path/foobar.rs")
mycrate = rustimport.imp_from_path("./mycrate/")

By default, these explicit function do not require the "rustimport" keyword on the first line of the .rs source file or the according marker in the crate.

Contributing and architecture

See CONTRIBUTING.md for details on the internals of rustimport and how to get involved in development.

License

rustimport uses the MIT License.