py2dist 0.1.3

Compile Python code to binary for distribution

py2dist

中文文档

py2dist is a tool that uses Cython to compile Python source code into binary extension modules (.so/.pyd). It is designed to simply protect source code from modification and is suitable for scenarios such as releasing Python projects or building Docker service images.

Features

• Compile single .py files or entire directories into binary files.
• Preserve directory structure, automatically copy other files to the output directory.
• Support excluding specific files or directories.
• Automatically detect and use ccache to accelerate compilation.
• Get a small performance boost from Cython compilation.
• Provide both CLI and Python API.

Installation

pip install py2dist

Important Note: Python Version Consistency

The compiled binary extension modules (.so/.pyd) are bound to a specific Python version. You must ensure that the Python version used for compilation is exactly the same as the Python version used at runtime (including minor version numbers; for example, 3.10 and 3.11 are incompatible).

If the versions do not match, you may encounter errors like the following when importing the module: ImportError: ... undefined symbol: _PyThreadState_UncheckedGet or ModuleNotFoundError: No module named ...

Usage

Command Line Interface (CLI)

The default output directory is dist / {directory name specified by -d}, but you can also specify the output directory using the -o parameter.

Compile a single file:

py2dist -f myscript.py

The output file location will be dist/myscript.so.

Compile an entire directory:

py2dist -d myproject

The output location will be dist/myproject, and non-.py files will be automatically copied to the output directory.

Example Usage

For example, if I have a Python FastAPI project and I want to package it as a Docker image while protecting the source code from modification, I can use py2dist to compile the core code directory of the project into binary extension modules, and then copy them into the Docker image. Direct release is also possible, the principle is similar.

In actual projects, I am more accustomed to using files like uv, pyproject.toml, .python-version to control project dependencies and Python versions. Docker image builds can also install tools like ccache, uv to optimize the workflow. This demonstration simplifies the process and will not expand on this; you can research improvements on your own.

Example Project Environment

• ccache: Recommended installation to accelerate compilation speed for subsequent project changes. py2dist will automatically identify and use it.
• python3.12: The Python version used during project compilation. Therefore, the Docker image must also use this Python version, otherwise an error will occur.

Project Example Structure:

myproject/
├── Makefile (Project build file)
├── run.py (Server startup file, cannot be compiled)
├── requirements.txt
├── Dockerfile
├── server/ (Project code directory, the compilation target)
│   ├── __init__.py (Must exist for every module, content can be empty)
│   ├── main.py (FastAPI main entry file)
│   ├── utils.py
│   ├── router/ (Module router directory)
│   │   ├── __init__.py (Must exist for every module, content can be empty)
│   │   └── user.py
│   ├── static/ (Other files, will be copied as is)
│   │   └── image.png
│   └── templates/ (Other files, will be copied as is)
│       ├── index.html
│       └── about.html
├── tests/
│   └── test_main.py
├── models/
└   └── ...

Sample run.py:

import uvicorn

if __name__ == "__main__":
    uvicorn.run("server.main:app", host="0.0.0.0", port=3000)

1. The __init__.py file must exist and its content can be empty. This allows it to be recognized as a module after compilation.

2. An uncompiled run.py file is needed to start the server.

3. According to general standards, it is not recommended to place resource files in the source code directory; they are usually placed in a separate directory for reference. However, for demonstration purposes, some resource files are placed here to demonstrate the automatic copy function.

So you can write the Makefile like this:

.PHONY: install compile build

install:
    pip3 install py2dist

compile:
    python3 -m py2dist -d server

build: compile
    docker build -t myproject .

Write the Dockerfile like this:

FROM python:3.12-slim

WORKDIR /app

COPY models /app/models

RUN --mount=type=bind,source=requirements.txt,target=requirements.txt \
    pip3 install -r requirements.txt

COPY dist/server /app/server
COPY run.py /app/run.py

EXPOSE 3000
CMD ["python3", "/app/run.py"]

Then run the command:

cd myproject
make build

This will build the release version of the image. At this point, if we check the file structure inside the image, it looks like this:

/app/
├── run.py
├── server/
│   ├── __init__.py
│   ├── main.so
│   ├── utils.so
│   ├── router/
│   │   ├── __init__.py
│   │   └── user.so
│   ├── static/
│   │   └── image.png
│   └── templates/
│       ├── index.html
│       └── about.html
├── models/
└   └── ...

This achieves the goal of simply protecting source code from modification.

If you don't want to use a Docker image, you can directly package and release the project, or use a similar process; the principle is the same.

First modify the run.py file and add the following code at the beginning of the file:

import sys
import os

# ================= Import lib =================

current_dir = os.path.dirname(os.path.abspath(__file__))

lib_path = os.path.join(current_dir, "lib")

if lib_path not in sys.path:
    sys.path.insert(0, lib_path)

# ================= End of Import lib =================

And modify the server/main.py file, adding the following code at the beginning of the file:

import sys
import os

# ================= Import lib =================

current_dir = os.path.dirname(os.path.abspath(__file__))

lib_path = os.path.join(current_dir, "../lib")

if lib_path not in sys.path:
    sys.path.insert(0, lib_path)

# ================= End of Import lib =================

The purpose is to allow the Python interpreter to recognize third-party library files. Next, package and release using a command similar to the following:

cd myproject
make compile
mkdir -p build/lib
cp -r dist/server build/server
cp run.py build/run.py
pip install -r requirements.txt --target "./build/lib" --python-version "3.12" --only-binary=":all:"
tar -czvf myproject.tar.gz build

You can release the project to any server. Of course, this method also requires a Python 3.12 environment with the same version number in the runtime environment. You can also place a portable Python 3.12 environment yourself, or use tools like uv to control project dependencies and Python versions, which will not be expanded here.

Advanced

Arguments:

• -f, --file: Specify a single .py file to compile.
• -d, --directory: Specify the directory to compile.
• -o, --output: Output directory (default is dist).
• -m, --maintain: Files or directories to exclude (comma-separated).
• -x, --nthread: Number of compilation threads (default is 1).
• -q, --quiet: Quiet mode.
• -r, --release: Release mode (cleans up temporary build files).
• -c, --ccache: Use ccache (auto-detect by default, or specify path).

Python API

from py2dist import compile_file, compile_dir

# Compile a single file
compile_file("myscript.py", output_dir="dist")

# Compile a directory
compile_dir(
    "myproject",
    output_dir="dist",
    exclude=["tests", "setup.py"],
    nthread=4
)