kconfiglib 5.0.0

A flexible Python Kconfig parser

Table of contents

• Overview

• Documentation

• Installation

  • Installation with pip

  • Installation for the Linux kernel

  • Manual installation

  • Configuration interfaces

• Examples

  • Example scripts

  • Real-world examples

  • Sample make iscriptconfig session

• Test suite

• Notes

• Thanks

• License

Overview

This is version 2 of Kconfiglib, which is not backwards-compatible with Kconfiglib 1. For a summary of changes between Kconfiglib 1 and Kconfiglib 2, see kconfiglib-2-changes.txt.

Kconfiglib 2 now has two interactive configuration interfaces available. See the Configuration Interfaces section below.

Kconfiglib is a Python 2/3 library for working with Kconfig configuration systems. It also functions well as a standalone Kconfig implementation, and has menuconfig implementations available.

Kconfiglib can do the following, among other things:

• Programmatically get and set symbol values

  allnoconfig.py and allyesconfig.py examples are provided, automatically verified to produce identical output to the standard make allnoconfig and make allyesconfig.

• Read and write .config and defconfig files

  The generated .config and defconfig (minimal configuration) files are character-for-character identical to what the C implementation would generate (except for the header comment). The test suite relies on this, as it compares the generated files.

• Write C headers

  The generated headers use the same format as include/generated/autoconf.h from the Linux kernel.

• Implement incremental builds

  This uses the same scheme as the include/config directory in the kernel: Symbols are translated into files that are touched when the symbol’s value changes between builds, which can be used to avoid having to do a full rebuild whenever the configuration is changed.

  See the sync_deps() function for more information.

• Inspect symbols

  Printing a symbol gives output which could be fed back into a Kconfig parser to redefine it***. The printing function (__str__()) is implemented with public APIs, meaning you can fetch just whatever information you need as well.

  A helpful __repr__() is implemented on all objects too, also implemented with public APIs.

  ***Choice symbols get their parent choice as a dependency, which shows up as e.g. prompt "choice symbol" if <choice> when printing the symbol. This could easily be worked around if 100% reparsable output is needed.

• Inspect expressions

  Expressions use a simple tuple-based format that can be processed manually if needed. Expression printing and evaluation functions are provided, implemented with public APIs.

• Inspect the menu tree

  The underlying menu tree is exposed, including submenus created implicitly from symbols depending on preceding symbols. This can be used e.g. to implement menuconfig-like functionality.

  See menuconfig.py and the minimalistic menuconfig_example.py example.

Here are some other features:

• Single-file implementation

  The entire library is contained in kconfiglib.py.

• Runs unmodified under both Python 2 and Python 3

  The code mostly uses basic Python features and has no third-party dependencies. The most advanced things used are probably @property and __slots__.

• Robust and highly compatible with the standard Kconfig C tools

  The test suite automatically compares output from Kconfiglib and the C tools by diffing the generated .config files for the real kernel Kconfig and defconfig files, for all ARCHes.

  This currently involves comparing the output for 36 ARCHes and 498 defconfig files (or over 18000 ARCH/defconfig combinations in “obsessive” test suite mode). All tests are expected to pass.

  A comprehensive suite of selftests is included as well.

• Not horribly slow despite being a pure Python implementation

  The allyesconfig.py example currently runs in about 1.3 seconds on a Core i7 2600K (with a warm file cache), including the make overhead from make scriptconfig.

  Kconfiglib is especially speedy in cases where multiple .config files need to be processed, because the Kconfig files will only need to be parsed once.

  For long-running jobs, PyPy gives a big performance boost. CPython is faster for short-running jobs as PyPy needs some time to warm up.

  Kconfiglib also works well with the multiprocessing module. No global state is kept.

• Warning parity with the C implementation

  Generates the same warnings as the C implementation, plus a few extra ones.

  This is less important if the input is assumed to be well-formed, but makes Kconfiglib a viable replacement for the C tools if e.g. a menuconfig interface is added.

  All warnings point out the location(s) in the Kconfig files where a symbol is defined, where applicable.

• Kconfig extensions

  So far, these include a relative source statement (rsource) where Kconfig file paths are specified relative to the directory of the current Kconfig file, and a globbing source (gsource) that doubles as an include-if-exists function.

• Unicode support

  Unicode characters in string literals in Kconfig and .config files are correctly handled. This support mostly comes for free from Python.

• Windows support

  Nothing Linux-specific is used. Universal newlines mode is used for both Python 2 and Python 3.

  The Zephyr project uses Kconfiglib to generate .config files and C headers on Linux as well as Windows.

• Internals that (mostly) mirror the C implementation

  While being simpler to understand and tweak.

Documentation

Kconfiglib comes with extensive documentation in the form of docstrings. To view it, run e.g. the following command:

$ pydoc kconfiglib

For HTML output, add -w:

$ pydoc -w kconfiglib

A good starting point is to read the module docstring (which you could also just read directly at the beginning of kconfiglib.py). It gives an introduction to symbol values, the menu tree, and expressions.

After reading the module docstring, a good next step is to read the Kconfig class documentation, and then the documentation for the Symbol, Choice, and MenuNode classes.

Please tell me if something is unclear or can be explained better.

Installation

Installation with pip

Kconfiglib is available on PyPI and can be installed with e.g.

$ pip(3) install kconfiglib --user

All releases have a corresponding tag in the git repository, e.g. v5.0.0 (the latest version).

Semantic versioning is used. There’s been three small changes (1, 2, 3) to the behavior of the API (which I don’t think will affect anyone), which is why the major version is at 5 rather than 2.

Installation for the Linux kernel

See the module docstring at the top of kconfiglib.py.

Manual installation

The entire library is contained in kconfiglib.py. Just drop it somewhere.

Configuration interfaces

Two configuration interfaces are currently available:

• menuconfig.py is a terminal-based configuration interface implemented using the standard Python curses module.

  Some screenshots below:

  

  (Sorry about that yellow color. See the styling code at the top of menuconfig.py if you want to try to make it prettier. :))

  menuconfig.py only supports Python 3, mostly due to curses.get_wch() being used, which is needed for Unicode support. curses.get_wch() isn’t available in the Python 2 version of the curses module.

  menuconfig.py has no third-party dependencies on *nix.

  On Windows, the curses modules is not available by default, but support can be added by installing the windows-curses package:

  $ pip install windows-curses

  This uses wheels built from this repository, which is in turn based on Christoph Gohlke’s Python Extension Packages for Windows.

  See the docstring at the top of menuconfig.py for more information about the terminal menuconfig implementation.

• RomaVis has built a fully portable Python 2/3 TkInter menuconfig implementation. It is still a work-in-progress, but is already functional.

  See the pymenuconfig project for more information.

  Screenshot below:

  

Examples

Example scripts

The examples/ directory contains some simple example scripts. Among these are the following ones. Make sure you run them with the latest version of Kconfiglib, as they might make use of newly added features.

• allnoconfig.py, allnoconfig_simpler.py, and allyesconfig.py implement make allnoconfig and make allyesconfig in various ways. Demonstrates menu tree walking and value setting.

• defconfig.py has the same effect as going into make menuconfig and immediately saving and exiting.

• eval_expr.py evaluates an expression in the context of a configuration.

• find_symbol.py searches through expressions to find references to a symbol, also printing a “backtrace” with parents for each reference found.

• help_grep.py searches for a string in all help texts.

• print_tree.py prints a tree of all configuration items.

• print_config_tree.py is similar to print_tree.py, but dumps the tree as it would appear in menuconfig, including values. This can be handy for visually diffing between .config files and different versions of Kconfig files.

• list_undefined.py finds references to symbols that are not defined by any architecture in the Linux kernel.

• merge_config.py merges configuration fragments to produce a complete .config, similarly to scripts/kconfig/merge_config.sh from the kernel.

• oldconfig.py provides make oldconfig functionality, prompting the user for the values of new symbols to update an old .config file.

• menuconfig_example.py implements a configuration interface that uses notation similar to make menuconfig. It’s deliberately kept as simple as possible to demonstrate just the core concepts.

Real-world examples

• kconfig.py from the Zephyr project handles .config and header file generation, also doing configuration fragment merging.

• genrest.py generates a Kconfig symbol cross-reference, which can be viewed here.

These use the older Kconfiglib 1 API, which was clunkier and not as general (functions instead of properties, no direct access to the menu structure or properties, uglier __str__() output):

• genboardscfg.py from Das U-Boot generates some sort of legacy board database by pulling information from a newly added Kconfig-based configuration system (as far as I understand it :).

• gen-manual-lists.py generated listings for an appendix in the Buildroot manual. (The listing has since been removed.)

• gen_kconfig_doc.py from the esp-idf project generates documentation from Kconfig files.

• SConf builds an interactive configuration interface (like menuconfig) on top of Kconfiglib, for use e.g. with SCons.

• kconfig-diff.py – a script by dubiousjim that compares kernel configurations.

• Originally, Kconfiglib was used in chapter 4 of my master’s thesis to automatically generate a “minimal” kernel for a given system. Parts of it bother me a bit now, but that’s how it goes with old work.

Sample make iscriptconfig session

The following log should give some idea of the functionality available in the API:

$ make iscriptconfig
A Kconfig instance 'kconf' for the architecture x86 has been created.
>>> kconf  # Calls Kconfig.__repr__()
<configuration with 13711 symbols, main menu prompt "Linux/x86 4.14.0-rc7 Kernel Configuration", srctree ".", config symbol prefix "CONFIG_", warnings enabled, undef. symbol assignment warnings disabled>
>>> kconf.mainmenu_text  # Expanded main menu text
'Linux/x86 4.14.0-rc7 Kernel Configuration'
>>> kconf.top_node  # The implicit top-level menu
<menu node for menu, prompt "Linux/$ARCH $KERNELVERSION Kernel Configuration" (visibility y), deps y, 'visible if' deps y, has child, Kconfig:5>
>>> kconf.top_node.list  # First child menu node
<menu node for symbol SRCARCH, deps y, has next, Kconfig:7>
>>> print(kconf.top_node.list)  # Calls MenuNode.__str__()
config SRCARCH
    string
    option env="SRCARCH"
    default "x86"

>>> sym = kconf.top_node.list.next.item  # Item contained in next menu node
>>> print(sym)  # Calls Symbol.__str__()
config 64BIT
    bool
    prompt "64-bit kernel" if ARCH = "x86"
    default ARCH != "i386"
    help
      Say yes to build a 64-bit kernel - formerly known as x86_64
      Say no to build a 32-bit kernel - formerly known as i386

>>> sym  # Calls Symbol.__repr__()
<symbol 64BIT, bool, "64-bit kernel", value y, visibility y, direct deps y, arch/x86/Kconfig:2>
>>> sym.assignable  # Currently assignable values (0, 1, 2 = n, m, y)
(0, 2)
>>> sym.set_value(0)  # Set it to n
True
>>> sym.tri_value  # Check the new value
0
>>> sym = kconf.syms["X86_MPPARSE"]  # Look up symbol by name
>>> print(sym)
config X86_MPPARSE
    bool
    prompt "Enable MPS table" if (ACPI || SFI) && X86_LOCAL_APIC
    default "y" if X86_LOCAL_APIC
    help
      For old smp systems that do not have proper acpi support. Newer systems
      (esp with 64bit cpus) with acpi support, MADT and DSDT will override it

>>> default = sym.defaults[0]  # Fetch its first default
>>> sym = default[1]  # Fetch the default's condition (just a Symbol here)
>>> print(sym)  # Print it. Dependencies are propagated to properties, like in the C implementation.
config X86_LOCAL_APIC
    bool
    default "y" if X86_64 || SMP || X86_32_NON_STANDARD || X86_UP_APIC || PCI_MSI
    select IRQ_DOMAIN_HIERARCHY if X86_64 || SMP || X86_32_NON_STANDARD || X86_UP_APIC || PCI_MSI
    select PCI_MSI_IRQ_DOMAIN if PCI_MSI && (X86_64 || SMP || X86_32_NON_STANDARD || X86_UP_APIC || PCI_MSI)

>>> sym.nodes  # Show the MenuNode(s) associated with it
[<menu node for symbol X86_LOCAL_APIC, deps n, has next, arch/x86/Kconfig:1015>]
>>> kconfiglib.expr_str(sym.defaults[0][1])  # Print the default's condition
'X86_64 || SMP || X86_32_NON_STANDARD || X86_UP_APIC || PCI_MSI'
>>> kconfiglib.expr_value(sym.defaults[0][1])  # Evaluate it (0 = n)
0
>>> kconf.syms["64BIT"].set_value(2)
True
>>> kconfiglib.expr_value(sym.defaults[0][1])  # Evaluate it again (2 = y)
2
>>> kconf.write_config("myconfig")  # Save a .config
>>> ^D
$ cat myconfig
# Generated by Kconfiglib (https://github.com/ulfalizer/Kconfiglib)
CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_MMU=y
...

Test suite

The test suite is run with

$ python(3) Kconfiglib/testsuite.py

pypy works too, and is much speedier for everything except allnoconfig.py/allnoconfig_simpler.py/allyesconfig.py, where it doesn’t have time to warm up since the scripts are run via make scriptconfig.

The test suite must be run from the top-level kernel directory. It requires that the Kconfiglib git repository has been cloned into it and that the makefile patch has been applied.

To get rid of warnings generated for the kernel Kconfig files, add 2>/dev/null to the command to discard stderr.

NOTE: Forgetting to apply the Makefile patch will cause some tests that compare generated configurations to fail

NOTE: The test suite overwrites .config in the kernel root, so make sure to back it up.

The test suite consists of a set of selftests and a set of compatibility tests that compare configurations generated by Kconfiglib with configurations generated by the C tools, for a number of cases. See testsuite.py for the available options. You might want to use the “speedy” option to speed things up a bit.

The tests/reltest script runs the test suite and all the example scripts for both Python 2 and Python 3, verifying that everything works.

Rarely, the output from the C tools is changed slightly (most recently due to a change I added). If you get test suite failures, try running the test suite again against the linux-next tree, which has all the latest changes. I will make it clear if any non-backwards-compatible changes appear.

A lot of time is spent waiting around for make and the C utilities (which need to reparse all the Kconfig files for each defconfig test). Adding some multiprocessing to the test suite would make sense too.

Notes

• I sometimes see people add custom output formats, which is pretty straightforward to do (see the implementations of write_autoconf() and write_config() for a template). If you come up with something you think might be useful to other people, I’m happy to take it in upstream. Batteries included and all that.

• Kconfiglib assumes the modules symbol is MODULES, which is backwards-compatible. A warning is printed by default if option modules is set on some other symbol.

  Let me know if you need proper option modules support. It wouldn’t be that hard to add.

• fpemud has put together Python bindings to internal functions in the C implementation. This is an alternative to Kconfiglib’s all-Python approach.

• The test suite failures (should be the only ones) for the following Blackfin defconfigs on e.g. Linux 3.7.0-rc8 are due to a bug in the C implementation:

  • arch/blackfin/configs/CM-BF537U_defconfig

  • arch/blackfin/configs/BF548-EZKIT_defconfig

  • arch/blackfin/configs/BF527-EZKIT_defconfig

  • arch/blackfin/configs/BF527-EZKIT-V2_defconfig

  • arch/blackfin/configs/TCM-BF537_defconfig

Thanks

Thanks to Philip Craig for adding support for the allnoconfig_y option and fixing an obscure issue with comments inside choices (that didn’t affect correctness but made outputs differ). allnoconfig_y is used to force certain symbols to y during make allnoconfig to improve coverage.

License

See LICENSE.txt. SPDX license identifiers are used in the source code.