codegen-cpp

codegen-cpp is a C++ code generation utility written in Python.

It can be used to genereate C++ code from TOML specification files. Presently it supports generation Struct-of-Array data structures and code to read those tables to and from CSV and Parquet files using Apache Arrow.

Requirements

codegen-cpp itself needs Python 3.12 or later. The code it generates needs a C++23 compiler and Apache Arrow built with the CSV and Parquet support.

Installation

python -m venv .venv
.venv/bin/pip install .

Usage

Write a specification, for example spec.toml:

[[table]]
name = "Measurement"
columns = [
    { name = "station_id", type = "i64" },
    { name = "temperature", type = "f64" },
    { name = "note", type = "str" },
]

[[csv_reader]]
name = "MeasurementCsvReader"
table = "Measurement"
default_values = { note = "" }

[[parquet_writer]]
name = "MeasurementParquetWriter"
table = "Measurement"

Generate the headers:

codegen-cpp generate spec.toml --output-dir include

This writes one header per section, named after the section: include/Measurement.hpp, include/MeasurementCsvReader.hpp, and include/MeasurementParquetWriter.hpp.

Use them to convert a CSV file into a Parquet file:

#include "MeasurementCsvReader.hpp"
#include "MeasurementParquetWriter.hpp"

int main() {
    MeasurementCsvReader reader("measurements.csv.gz", 100000);
    MeasurementParquetWriter writer("measurements.parquet");

    while (reader.has_more_batches()) {
        const Measurement batch = reader.read_batch();
        writer.write_batch(batch);
    }

    writer.close();
    return 0;
}

examples/example1.toml is an exemplar specification that shows features of the specification format.. To see how a specification is parsed, without generating anything:

codegen-cpp debug parse-spec examples/example1.toml

The specification

A specification is a TOML document holding any number of sections of five kinds. Every section is an array of tables, written [[table]], [[csv_reader]], and so on.

Every section has a name, which is used verbatim as the name of the generated class and of the header file holding it. The names of all sections share one namespace and have to be unique. Every reader and writer names the table it reads into or writes out.

A table declares its columns, each with a name used verbatim as a C++ member name, and one of the scalar types:

Readers may declare default_values, a mapping of column names to the value stored when that column is null in the input file. A null in any other column is an error. The value has to fit the type of its column.

The generated code

For a table called Measurement, Measurement.hpp defines two structs. MeasurementRow holds a single row by value, and Measurement holds the rows column by column, one std::vector per column:

struct Measurement {
    std::vector<std::int64_t> station_id;
    std::vector<double> temperature;
    std::vector<std::string> note;

    std::size_t size() const noexcept;
    void clear() noexcept;
    void reserve(std::size_t n);
    void push_back(const MeasurementRow& row);
    void push_back(const std::int64_t& station_id_,
                   const double& temperature_,
                   const std::string& note_);
    MeasurementRow operator[](std::size_t i) const;
};

All the columns of a table have the same length, which is what size() reports. operator[] returns a copy of a row, because the rows are not stored as rows.

Readers hand out one batch of rows at a time, and writers take one batch at a time:

bool has_more_batches();       // readers
Table read_batch();            // readers, at most batch_size rows

void write_batch(const Table& table);  // writers
void close();                          // writers

A writer replaces the file it opens if it already exists. close() writes out what is left and releases the file; calling it twice is allowed. The destructor closes the file as well, but only close() reports a failure to write.

The constructors take the path of the file, and readers also take the number of rows per batch. The remaining arguments are optional:

For the CSV classes, the compression is guessed from the suffix of the file name, so .gz, .zst, .bz2 and .lz4 are compressed and everything else is plain text. Passing a codec explicitly overrides the guess. Parquet files carry their compression inside them, so the Parquet reader needs no such argument.

Development

Install the package and its development dependencies in a virtual environment, and run the tests:

python -m venv .venv
.venv/bin/pip install -e ".[dev]"
.venv/bin/pytest

The project is formatted with black, checked with pycodestyle and pyright.

Testing the generated C++ code

The tests under tests/cpp generate headers, write CSV and Parquet files with Arrow, and read them back with the generated readers. Arrow is installed with Conan; see conanfile.txt for the features it is built with. Note that Arrow needs C++20 or later, which the default Conan profile does not ask for.

conan install . --build=missing -of build -s compiler.cppstd=20
cmake -S tests/cpp -B build/cpp \
    -DCMAKE_TOOLCHAIN_FILE="$PWD/build/build/Release/generators/conan_toolchain.cmake" \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build/cpp
ctest --test-dir build/cpp --output-on-failure

License

MIT, see LICENSE.