cffi-mkstub 0.0.3

Type stub generator for cffi modules

cffi-mkstub

Type stub generator for Python modules generated by cffi. This allows statically checking that the C APIs are being used correctly and provides autocompletion of:

• function names
• global variable names
• struct or union member names
• enum members
• C type expressions passed to ffi.new, ffi.cast and friends

To get an idea of what generated stubs look like, see example (source C code). They contain:

• a few generic helper definitions that should ideally be incorporated to typeshed
• a types namespace with definitions for all discovered ctypes
• a Lib class specialization with definitions for all globals (functions, variables, constants)
• an FFI class specialization where methods like new, cast and such are given precise overloads for every C type expression, member name, etc.

The type stubs would ideally be bundled into the resulting cffi package, but can also be used directly by end users. Can also be used with in-line mode, see programmatic generation below.

Philosophy

It's best to start using the generated stubs from the start. In particular, you should not expect existing code to typecheck out-of-the-box when applying them. The intent of this project isn't to rule out wrong usages of the API and 'stand down' on usages that could be correct, because due to cffi's design, most of your code would fall in the latter category. Instead we aim to expose a reasonable subset of cffi's API that can be reliably checked.

A major consequence of this can be seen in ffi methods that accept C type expressions, like ffi.new('MyStruct_t *'). cffi will happily parse and normalize your C type expression, so the string MyStruct_t* would also work, but this can't possibly be done from a type stub. Therefore, rather than falling back to returning a generic CData object if an unrecognized string is passed, the stubs require you to enter the exact (normalized) C type expression. Any difference, like a missing space, triggers a typechecker error. This was deemed a reasonable tradeoff because when using an appropriate language server, you will usually get autocompletion of the accepted strings when writing your code:

Another consequence is type conversions. cffi is very lenient with what it accepts, e.g. on a parameter of C type int you can pass a Python int or an integer CData or an enum CData. The stubs currently only allow int, because this is what cffi returns when converting in the other direction.

Status

Despite the above, the aim is still to eventually expose all functionality of cffi (even if through a restricted set of API usages). Even though this project is in the proof of concept stage (and I'm depressed so manage your expectations about my ability to improve and maintain it) it's already mature enough to cover most of the functionality, and I'm using it on my projects. Some notable limitations:

• Pylance (or maybe Pyright itself, or the VSCode extension) seems to be very bad at handling methods with lots of overloads. In particular it freezes when the cursor is inside of a call like ffi.new() and it attempts to display the parameters, to the point where I have to restart the language server. Short of cffi adding new per-ctype APIs (which would greatly reduce the number of overloads in a single function) there isn't much we can do to work around it.

• When calling ffi.addressof() with a field name, Pylance will fail to filter valid names according to the type being passed (it will show all member names of all types). Typechecking still works correctly, i.e. an error will result if the member name of another struct is passed.

• When using ffi.def_extern, the name needs to be passed as an argument:

  @ffi.def_extern('my_callback')
  def my_callback():
      ...
  

• Included FFI objects (ffi/ffibuilder.include(other_ffi)) are currently ignored.

One of the priorities right now is to make it (both the generated stubs, and with less priority, the generator itself) more compatible with older typecheckers / Python runtimes. It has only been tested with Python 3.13 so far.

Works in both ABI and API modes, but most of the testing has been done on the former, so some wrinkles may still be present when targetting API mode. It has only been tested with Pylance/PyRight so far.

Getting started

Generating the stubs

cffi-mkstub works by interrogating your specific ffi object. Because of this, cffi-mkstub doesn't actually pull cffi as a dependency -- but because ffi objects need to import _cffi_backend to function, it presumes that module to be available for import.

ctypes backend support

The above is not entirely true; although this is currently not very well supported by cffi, when in in-line ABI mode, cffi can be told to use its ctypes backend (removing any dependency on custom native extensions at all). To support this case, cffi-mkstub will tolerate `_cffi_backend` failing to import when `cffi` could be imported, and if an in-line ABI FFI object is passed, it will use its undocumented `_backend` property to know which backend was requested.

[!IMPORTANT] Although we try to use documented cffi APIs whenever possible, their current introspection APIs are simply not enough. We have an open PR that implements more APIs. This project needs cffi to have that patch applied to run. Note that the patch is only needed to run the type stub generator; your project can then use the generated type stubs with a vanilla version of cffi, as the (non-introspection) APIs remain unchanged. There's also no need for the cffi module to have been built with a patched cffi.

A simple CLI interface is provided which should be enough for most users; run it passing -m and the name of the module to import:

cffi-mkstub -m my_lib._foo_cffi

This should be the module that ffibuilder.compile() produced, e.g. what you passed in the first argument to ffibuilder.set_source(). cffi-mkstub will import it and fetch its ffi attribute. Alternatively you can pass a filesystem path with -p:

cffi-mkstub -p my_lib/_foo_cffi.py

By default, the stubs will be written to a .pyi file next to the imported module file. This should cause tools to find them automatically. If this is not desired (for example, when the original module is in an unwritable system directory) pass -o <output path>.

Using the stubs

Note that the stub declares type aliases, classes and namespaces that do not really exist at runtime in the real module. Never attempt to use those at runtime (with e.g. instanceof). If you need to reference types in an annotation...

import _foo_cffi
lib = _foo_cffi.ffi.dlopen('my_foo.so')

def pat(animal: _foo_cffi.types.Animal_t) -> _foo_cffi.Pointer[int]:
    return lib.foo_pat(animal)

...the code will only run on Python 3.14+, where annotations are evaluated lazily. To support older versions, put the expression inside a string literal:

def pat(animal: '_foo_cffi.types.Animal_t') -> '_foo_cffi.Pointer[int]':
    return lib.do_pat(animal)

You can also do from __future__ import __annotations__ (supported since 3.7+) which will cause the interpreter to do that for you, but in the future it will get deprecated and eventually removed.

If you want to import the types namespace to reduce verbosity, or even some of the types themselves, you can use typing.TYPE_CHECKING to skip doing so at runtime (where it would fail). You can even bring specific types into scope (but you can't use import for that, since types isn't a submodule):

from typing import TYPE_CHECKING
from _foo_cffi import ffi

if TYPE_CHECKING:
    from _foo_cffi import types, Pointer
    Animal_t = types.Animal_t

lib = ffi.dlopen('my_foo.so')

animals: list['Pointer[Animal_t]'] = []
...

Programmatic generation

cffi-mkstub can also be invoked programmatically through the Python API, which is ideal in e.g. build scripts after compiling with ffibuilder. The simple API (used by the CLI) is write_type_stub which takes either a module object or a name to import, and allows overriding the output path through output_path:

from cffi_mkstub import write_type_stub
write_type_stub('my_lib._foo_cffi')

import external_lib._cffi
write_type_stub(external_lib._cffi, output_path='external_cffi.pyi')

If more control is desired, the format_type_hints API can be used directly. This takes the ffi object (not the module) and returns Python code as a string, and accepts a number of arguments to customize the output (but not a lot yet). Keep in mind the returned Python code defines the FFI and Lib classes, but does not assume instances of those to be present in the ffi and/or lib attributes of your module -- when using this API, you need to declare these yourself:

from cffi_mkstub import format_type_hints
from my_lib._foo_cffi import ffi

hints = format_type_hints(ffi, ffi_cls_name='FooFFI', lib_cls_name='FooLib')
hints += '\n\n' 'ffi: FooFFI' '\n' 'lib: FooLib'
with open('src/my_lib/_foo_cffi.pyi', 'w') as f:
    f.write(hints)

This API can be used to support in-line usage, e.g. with ABI mode:

from typing import TYPE_CHECKING, cast
from cffi import FFI
from cffi_mkstub import format_type_hints

ffi = FFI()
ffi.cdef("""
    int printf(const char *format, ...);   // copy-pasted from the man page
""")

# on the first run (see below) the type stub file will be populated and the typechecker will be able to load it
with open('build/_gen_ffi_stubs.pyi', 'w') as f:
    f.write(format_type_hints(ffi))
if TYPE_CHECKING:
    from build._gen_ffi_stubs import FFI
    ffi = cast(FFI, ffi)

lib = ffi.dlopen(None)
arg = ffi.new("char[]", b"world")
lib.printf(b"hi there, %s.\n", arg)

This could ideally be used as a compact way to generate type stubs (in-line ABI mode doesn't need a working toolchain, a library to link against nor a temporary output directory) to distribute in a stub package and use them later with the compiled module. However keep in mind there are some subtle differences between the four mode combinations in cffi, most notably that in ABI mode no overloads for def_extern are generated. In the future we could provide flags to 'simulate' other modes to enable this use case.

Wishlist for cffi

PEP-484 did not exist when cffi was started, so it's understandable that its design doesn't lend very well to static typechecking. While I'd say we can get a very decent result already, some changes on their side would allow us to improve the developer experience (sometimes even outside cffi-mkstub as well) even more:

• An option to retain documentation comments in the produced module. It would then be exposed via __doc__ or a dedicated ctype API. This would also include function argument names.

• Expose the base type of an enum, i.e. _CTypeEnum.base_type.

• Per-type addressof, offsetof, new, cast APIs (see above for motivation).

• Retain typedefs as their own CType kind. Being able to tell which particular typedef, if any, was used at a specific spot (say, a struct member, or a function argument) to reference a given type.

These are changes that could ideally be implemented in a backwards compatible way.

License

cffi-mkstub itself is GPL licensed for now, but you are of course free to use the stubs it generates for whatever purpose.