metac: C metaprogramming & code generation package for Python

metac is a general-purpose template engine targeting plain ANSI C — simply put, it is to C what templates are to C++. Generic types are written once in Python against type parameters; the engine specializes each definition for the concrete type arguments a module actually uses and emits ordinary C source files. Using the same monomorphization technique a C++ compiler performs, every (definition, type arguments) pair is stamped out at generation time as a distinct, fully typed implementation. Nothing generic survives into compilation: the C compiler only ever sees plain, C++-safe ANSI C code.

Alongside the template engine, metac provides a comprehensive standard library of containers and utility types — a functional analog of the C++ STL. It spans direct-access and linked sequences, strings and string buffers, hashed, intrusive, and ordered sets and maps, bit sets, records, tagged variants, and raw or reference-counted references. Custom types are first-class citizens: implement the value protocol in Python for complex custom types to work seamlessly as elements and fields anywhere in the catalog, intrusive containers included.

What holds everything together is uniform value semantics with the uniform treatment of primitive and composite types. Every type — from int to a nested map of records — carries structured lifetime and maintenance operations, with capabilities derived automatically from its constituent parts and only valid operations emitted. Primitives realize the protocol as inline expressions while composites use generated extern or inline functions, keeping call sites uniform. Owing to free composition of build-in and custom types a list of vectors of records is just another type definition. Ever dreamt of having std::map<std::string, std::list<int>> in plain C? So here it is.

Why metac?

C is notorious for its complete lack of complex data types in the standard library, and the usual workarounds each hurt in a familiar way:

• Macro-based generics (typeof tricks, token pasting) are opaque, hard to debug, and explode into incomprehensible errors.

• void*-based generic libraries abandon type safety, incur pointer-chasing overhead, and obscure ownership semantics (who copies? who destroys?).

• Hand-writing one container per type works, but is tedious, error-prone, and never quite finished.

metac takes a fourth road: containers are written once against a strict value-semantics protocol, and the generator stamps out the exact C code that would have otherwise written manually (arguably better), with none of the macro or void* compromises.

• Zero dependencies: generated code targets strict ANSI C and requires nothing beyond the C standard library — just #include the header and compile the generated sources along with consuming code.

• Build-system native: provides CMake, SCons, and Meson integrations that automatically re-run the generator whenever definitions change, so specialized code never goes stale.

• Battle-tested output: the project validates its generated test suites across MSVC, GCC, Clang, along with a bunch of second tier C compilers.

• Bolierplate control: the generator is designed to deal with large amount of generated C code with its ability to distribute the code over multiple translation units to sport parallel compilation and to perform the smart generation to avoid triggering unneccessary recompilations of unaltered code.

This project is the successor to the AutoC package for Ruby.

Quick start

Requires Python ≥ 3.10. metac itself has zero dependencies.

1. Define a module — e.g. mymodule.py:

from metac.module import Module
from metac.vector import Vector
from metac.chained_hash_map import Map
import metac.std as std

with Module("mymodule") as m:
  m.add(Vector("int_vector", "int"))
  m.add(Map("str2double", "const char*", "double"))

2. Generate — run the script; metac writes mymodule_auto.h / mymodule_auto.c (splitting into several sources when a module grows too large for one translation unit).

3. Use — from C:

#include "mymodule_auto.h"

int_vector v;
int_vector_create(&v);
int_vector_set(&v, 0, 42);
assert(int_vector_get(&v, 0) == 42);

/* iterate */
int_vector_range r;
for(r = int_vector_range_new(&v); !int_vector_range_empty(&r);
    int_vector_range_move_front(&r)) {
  printf("%d\n", int_vector_range_front(&r));
}

int_vector_destroy(&v);

Every generated type speaks the same vocabulary of value semantics:

void  T_create (T* target);              /* default construction        */
void  T_destroy(T* target);              /* release resources           */
void  T_copy   (T* target, const T* src);
int   T_equal  (const T* left, const T* right);
int   T_compare(const T* left, const T* right);   /* when orderable    */
size_t T_hash  (const T* target);                 /* when hashable     */

plus the container-specific operations (set/get/view/contains for sequences and sets, put/remove for sets, get/set/view for maps, empty/size, forward ranges for iteration, …).

Tutorial: a complete project, end to end

Let's build a small but real program — a sensor log that accumulates readings into a list of ints, aggregates them and drains them — from an empty directory to a running binary.

1. Install the generator

From a checkout of this repository (Python ≥ 3.10 required):

pip install .

or as a package from remote PyPi repository:

pip install metac

2. Scaffold the project

metac provides project scaffolders for CMake, SCons, and Meson:

python -m metac.cmake readings   # CMake
# or: python -m metac.scons readings   # SCons
# or: python -m metac.meson readings   # Meson
# or: python -m metac.sample readings  # generic, defaults to CMake (-b meson / -b scons)
cd readings

This sets up a working project with the generator already wired in. For CMake:

readings/
├── CMakeLists.txt            # build + regeneration rules
├── CMakePresets.json         # debug/release presets
├── cmake/
│   └── MetaC.cmake           # add_metac_module() helper
├── readings.py               # type definitions — the generator script
├── readings.c                # your application
├── readings.code-workspace
├── .vscode/launch.json
└── .gitignore

For SCons, the scaffolder generates SConstruct, site_scons/site_tools/metac.py, and .metac/scons.site. For Meson, it generates meson.build, meson.options, and .metac/meson.site.

The project name doubles as the module name: readings.py will generate readings_auto.h / readings_auto.c, and CMake will build them as the readings-auto library your executable links against.

3. Define the container

Replace readings.py with:

import metac.core
import metac.module
import metac.cmake
from metac.list import List

# CamelCase naming (IntListPushFront) is the default; switch to the
# snake_case style used throughout the test suite and this document:
metac.core.decorator = metac.core.snake_decorator

with metac.module.Module("readings") as m:
  m.add(List("int_list", "int"))

metac.cmake.CMake(m)   # emit the CMake fragment wiring the generated sources

4. Use it from C

Replace readings.c with:

#include <stdio.h>
#include "readings_auto.h"

int main(void) {
  int_list readings;
  int_list_create(&readings);

  /* readings arrive one at a time, newest first */
  int samples[] = {23, 21, 25, 24, 27, 22, 26};
  for(size_t i = 0; i < sizeof(samples) / sizeof(samples[0]); ++i) {
    int_list_push_front(&readings, samples[i]);
  }

  /* aggregate via the range interface */
  size_t count = int_list_size(&readings);
  long sum = 0;
  int max = 0;
  for(int_list_range r = int_list_range_new(&readings);
      !int_list_range_empty(&r);
      int_list_range_move_front(&r)) {
    int v = int_list_range_front(&r);
    sum += v;
    if(v > max) max = v;
  }
  printf("count=%zu sum=%ld avg=%.1f max=%d\n",
         count, sum, (double)sum / (double)count, max);

  /* membership test */
  printf("seen 25: %s\n", int_list_contains(&readings, 25) ? "yes" : "no");

  /* drain, consuming the stored values one by one */
  while(!int_list_empty(&readings)) {
    printf("consumed %d\n", int_list_pop_front(&readings));
  }

  int_list_destroy(&readings);
  return 0;
}

5. Build and run

With CMake:

cmake --preset debug
cmake --build --preset debug
./build/debug/readings

With SCons:

scons
./readings

With Meson:

meson setup build
meson compile -C build
./build/readings

Output:

count=7 sum=168 avg=24.0 max=27
seen 25: yes
consumed 26
consumed 22
consumed 27
consumed 24
consumed 25
consumed 21
consumed 23

(The values come back in reverse order of insertion — a singly-linked list is pushed and popped at the front.)

What just happened

• Running readings.py rendered readings_auto.h / readings_auto.c — plain C, no runtime — plus readings.cmake and readings.state (digests of what was generated).
• add_metac_module() in CMakeLists.txt registered the generator as a proper build step: it re-runs readings.py whenever the definitions change, so generated code never goes stale.
• readings-auto is an object library; your executable just links it and includes the header.
• The workflow from here: add more types to the module (a Map for calibration constants, an intrusive hash set for deduplication, a Record for readings with timestamps, …), rerun the build, and use the generated functions — nothing else in your C code changes.

Principles

Plain C, zero runtime. The deliverable is source code you can read, grep, step through in a debugger and ship. No library to link, no ABI to track, no preprocessor machinery to fight. Regenerating is cheap; the code is meant to be regenerated, not edited.

Type-safe by monomorphization. Each container is specialized to its element type at generation time. No void*, no casts, no element-size parameters — int_vector_set takes an int, str2double_set takes a const char* and a double. Passing conventions are chosen per element kind: primitives cross API boundaries by value, composites by pointer-to-const.

One protocol for all types. Primitives and composites are not two worlds. Every type — from int to a nested Map of records — carries the same operations (create/destroy/copy/equal/compare/hash), so containers compose freely: a List of Vectors of records just works. Primitives realize the protocol as inline expressions (target = source, left == right, a cast for hashing); composites realize it as generated functions. The difference never leaks to the call sites.

Capabilities are derived, dead code is elided. Types carry traits — constructible, destructible, copyable, comparable, orderable, hashable. Composites derive them structurally from their fields (a record is destructible if any field is; copyable only if all are), and the generator emits only what the traits permit: a Vector("int_vector", "int") contains no element-destruction code at all, and an unordered container never grows a comparison operator.

Custom types are first-class citizens. Your own composites join the system by implementing the same protocol in Python — then they work as element or field types everywhere, including inside intrusive containers. Your code is injected as C expressions/functions at well-defined hook points, wrapped so it cannot break operator precedence or naming conventions.

Intrusive variants trade value space for layout. The intrusive hash set/map store elements in one flat allocation of exactly the element type — no wrapper entries, no tag arrays — with slot states (empty/deleted) encoded as sentinel values you choose. If your element type can reserve two distinguishable states (e.g. INT_MIN/INT_MAX for an int, or low pointer values for handles), you get the most cache-friendly layout possible; if it cannot, use the regular non-intrusive containers.

Deterministic builds. Generation is integrated with CMake via add_metac_module(), SCons via env.MetaCModule(), and Meson via custom_target: output digests are tracked, code is regenerated only when definitions actually change, and every generated artifact is declared as a proper build dependency.

Reference-counted memory as an option, not a mandate. Counted references provide automatic reference counting for shared objects — including when used as container elements — while Raw references give you plain unmanaged pointers. Both speak the same protocol as every other type.

Container catalog

Build system integration

CMake

include(MetaC)   # from the generated cmake/ directory

add_metac_module(
  mymodule
  DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
  MAIN_DEPENDENCY ${CMAKE_CURRENT_SOURCE_DIR}/mymodule.py
  COMMAND ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/mymodule.py mymodule
)

target_link_libraries(myapp mymodule-auto)   # header + generated sources

The module bootstraps itself on first configure and re-runs the generator only when mymodule.py (or its stated dependencies) change. See test/CMakeLists.txt for a working example.

SCons

env = Environment(tools=["default", "metac"], toolpath=["site_scons/site_tools"])

metac_mod = env.MetaCModule("mymodule", directory=".")
env.Program(target="myapp", source=["myapp.c"] + metac_mod.sources)

The SCons tool checks file timestamps against .metac/mymodule.state to trigger regeneration only when definitions change.

Meson

# In meson.build
module_gen = custom_target('mymodule-generate',
  input: 'mymodule.py',
  output: 'mymodule.stamp',
  command: [py, '@INPUT@', 'mymodule', '@OUTPUT@'],
  build_by_default: true,
)
metac_dep = declare_dependency(
  sources: [metac_sources, module_gen],
  include_directories: include_directories('.')
)
executable('myapp', sources: ['myapp.c'], dependencies: [metac_dep])

How it tests itself

The repository's test suite is written the way users write code: Python scripts in src/metac/test/ define container instances and their unit tests, and metac generates a C test module (test_auto.c, ~2,200 lines) that is compiled and executed as part of the build. The generator is only as good as the C it produces, so that is what gets tested.

License

BSD 3-Clause — see LICENSE.

The end

Happy hacking time and have fun!

Oleg A. Khlybov fougas@mail.ru