Skip to main content

C++ Documentation generator.

Project description

wurfapi

https://ci.appveyor.com/api/projects/status/l41u9e7y50r685ep?svg=true&branch=master https://travis-ci.org/steinwurf/wurfapi.svg?branch=master

We wanted to have a configurable and easy to use Sphinx API documentation generator for our C++ projects. To achieve this we leaned on others for inspiration:

So what is wurfapi:

  • Essentially we picked up where Gasp let go. We have borrowed the idea of templates to make it highly configurable.
  • We made it easy to use by automatically running Doxygen to generate the initial API documentation.
  • We parse the Doxygen XML into an easy to use Python dictionary. Which can be consumed in the templates.
  • We prepared the extension for other backends (replacing Doxygen) e.g. https://github.com/foonathan/standardese once they become ready.

Status

We are still very much in the initial development phase - all things are subject to change.

  • Parsing Doxygen XML: We do not support everything yet (and probably never will). We still are missing some crucial elements like proper parsing of the text elements in comments, parameter descriptions etc.

Usage

We recommend that you install wurfapi and sphinx in a virtual environment. To use the extension, the following steps are needed:

  1. Create a virtual environment:

    Follow the https://docs.python.org/3/tutorial/venv.html
    
  2. Install the extension:

    pip install sphinx
    pip install wurfapi
    
  3. Generate the initial Sphinx documentation by running:

    mkdir docs
    cd docs
    python sphinx-quickstart
    

    You will need to enter some basic information about your project such as the project name etc.

  4. Open the conf.py generated by sphinx-quickstart and add the the following:

    # Append or insert 'wurfapi' in the extensions list
    extensions = ['wurfapi']
    
    # wurfapi options - relative to your docs dir
    wurfapi = {
      'source_paths': ['../src'],
      'recursive': True,
      'parser': {'type': 'doxygen', 'download': True,  'warnings_as_error': True}
    }
    

    Note

    source_path

    If you separate source and build dir in sphinx your ‘source_path’ should be something like ‘../../src’.

    recursive

    Set recursive True if you want recursively scan the source_paths

    download

    If you do not want to automatically download Doxygen, set download to False. In that case wurfapi will try to invoke plain doxygen without specifying any path or similar. This means it doxygen must be available in the path.

    warnings_as_error

    If Doxygen emits many warnings you might want to set warnings_as_error to False until they have been fixed.

  5. To generate the API documentation for a class open a .rst file e.g. index.rst if you ran sphinx-quickstart. Say we want to generate docs for a class called test in the namespace project.

    To do this we add the following directive to the rst file:

    .. wurfapi:: class_synopsis.rst
      :selector: project::coffee::machine
    

    Such that index.rst becomes something like:

      Welcome to Coffee's documentation!
      ===================================
    
      .. toctree::
        :maxdepth: 2
        :caption: Contents:
    
      .. wurfapi:: class_synopsis.rst
          :selector: project::coffee::machine
    
      .. wurfapi:: class_synopsis.rst
          :selector: project::coffee::recipe
    
    
      Indices and tables
      ==================
    
      * :ref:`genindex`
      * :ref:`modindex`
      * :ref:`search`
    
    
    To do this we use the ``class_synopsis.rst`` template.
    
  6. Generate the Documentation

    make html

Running on readthedocs.org

To use this on readthedocs.org you need to have the wurfapi Sphinx extension installed. This can be done by adding a requirements.txt in the documentation folder. readthedocs.org can be configured to use the requirements.txt when building a project. Simply put wurfapi in to the requirements.txt.

Doxygen issues

Nothing is perfect, neither is Doxygen. Sometimes Doxygen gets it wrong e.g. in the following example:

class foo
{
private:
    class bar;
};

Doxygen incorrectly reports that bar has public scope (also reported here https://bit.ly/2BWPllZ). To deal with such issues, until a fix lands in Doxygen, you can do the following:

Add a list of patches to the API to your conf.py file. Extending the example from before, we can add the following fix:

wurfapi = {
  'source_paths': ['../src'],
  'recursive': True,
  'parser': {
    'type': 'doxygen', 'download': True,  'warnings_as_error': True,
     'patch_api': [
      {'selector': 'foo::bar', 'key': 'access', 'value': 'private'}
    ]
  }
}

The patch_api allows you to reach in to the parsed API information and update certain values. The selector is the unique-name of the entity you want to update. Check the “Dictionary layout” section further down for more information.

Collapse inline namespaces

For symbol versioning you may use inline namespaces, however typically you don’t want these to show up in the docs, as these are mostly invisible for your users.

With wurfapi you can collapse the inline namespace such that it is removed form the scopes etc.

Example:

namespace foo { inline namespace v1_2_3 { struct bar{}; } }

The scope to bar is foo::v1_2_3. If you collapse the inline namespace it will just be foo.

First issue you have to deal with is that Doxygen currently does not support inline namespaces. So we need to patch the API first:

wurfapi = {
  'source_paths': ['../src'],
  'recursive': True,
  'parser': {
    'type': 'doxygen', 'download': True,  'warnings_as_error': True,
     'patch_api': [
      {'selector': 'foo::v1_2_3', 'key': 'inline', 'value': True}
    ]
  }
}

After this we can collapse the namespace:

wurfapi = {
  'source_paths': ['../src'],
  'recursive': True,
  'parser': {
    'type': 'doxygen', 'download': True,  'warnings_as_error': True,
     'patch_api': [
      {'selector': 'foo::v1_2_3', 'key': 'inline', 'value': True}
    ],
    'collapse_inline_namespaces': [
      "foo::v1_2_3"
    ]
  }
}

Now you will be able to refer to bar as foo::bar. Note, that collapsing the namespace will affect the selectors you write when generating the documentation.

Custom templates

You can write you own custom templates for generating the rst output. To to this you simply write a Jinja2 compatible rst template and place it in some folder. Adding the user_templates key to the wurfapi configuration dictionary in the conf.py file will make it available.

For example:

wurfapi = {
    'source_paths': ['../src', '../examples/header/header.h'],
    'recursive': True,
    'user_templates': 'rst_templates',
    'parser': {
        'type': 'doxygen', 'download': True, 'warnings_as_error': True
    }
}

exclude_patterns = ['rst_templates/*.rst']

Now we can use *.rst files inside the rst_templates folder e.g. if we had a class_list.rst template we could use it like this:

.. wurfapi:: class_list.rst
    :selector: project::coffee

Release new version

  1. Edit NEWS.rst, wscript and src/wurfapi/wurfapi.py (set correct VERSION)

  2. Run

    ./waf upload
    

Source code

Tests

The tests will run automatically by passing --run_tests to waf:

./waf --run_tests

This follows what seems to be “best practice” advise, namely to install the package in editable mode in a virtualenv.

Recordings

A bunch of the tests use a class called Record, defined in (test/record.py). The Record class is used to store output as files from different parsing and rendering operations.

E.g. say we want to make sure that a parser function returns a certain dict object. Then we can record that dict:

recorder = record.Record(filename='test.json',
                         recording_path='/tmp/recording',
                         mismatch_path='/tmp/mismatch')

recorder.record(data={'foo': 2, 'bar': 3})

If data changes compared to a previous recording a mismatch will be detected. To update a recording simply delete the recording file.

Test directories

You will also notice that a bunch of the tests take a parameter called testdirectory. The testdirectory is a pytest fixture, which represents a temporary directory on the filesystem. When running the tests you will notice these temporary test directories pop up under the pytest_temp directory in the project root.

You can read more about that here:

Developer Notes

The sphinx documentation on creating extensions: http://www.sphinx-doc.org/en/stable/extdev/index.html#dev-extensions

Dictionary layout

We want to support different “backends” like Doxygen to parse the source code. To make this possible we define an internal source code description format. We then translate e.g. Doxygen XML to this and use that to render the API documentation.

This way a different “backend” e.g. Doxygen2 could be use used as the source code parser and the API documentation could be generated.

unique-name

In order to be able to reference the different entities in the API we need to assign them a name.

We use a similar approach here as described in standardese.

This means that the unique-name of an entity is the name with all scopes e.g. foo::bar::baz.

  • For functions you need to specify the signature (parameter types and for member functions cv-qualifier and ref-qualifier) e.g. foo::bar::baz::func() or foo::bar::baz::func(int a, char*) const. See cppreference for more information.

  • For class template specializations the unique name includes the specialization arguments. For example:

    // Here the unique-name is just 'foo'
    template<class T>
    class foo {};
    
    // Here the unique name is foo<int>
    template<>
    class foo<int> {};
    
  • In addition to types, we also have entries for the parsed files. For files the unique name will be the relative path from the project root.

The API dictionary

The internal structure is a dicts with the different API entities. The unique-name of the entity is the key and the entity type also a Python dictionary is the value e.g:

api = {
  'unique-name': { ... },
  'unique-name': { ... },
  ...
}

To make this a bit more concrete consider the following code:

namespace ns1
{
  class shape
  {
    void print(int a) const;
  };

  namespace ns2
  {
    struct box
    {
      void hello();
    };

    void print();
  }
}

Parsing the above code would produce the following API dictionary:

api = {
  'ns1': { 'kind': 'namespace', ...},
  'ns1::shape': { 'kind': 'class', ... },
  'ns1::shape::print(int) const': { kind': function' ... },
  'ns1::ns2': { 'kind': 'namespace', ... },
  'ns1::ns2::box': { 'kind': 'struct', ... },
  'ns1::ns2::box::hello()': { kind': function' ... },
  'ns1::ns2::print()': { 'kind': 'function', ...},
  'ns1.hpp': { 'kind': 'file', ...}
}

The different entity kinds expose different information about the API. We will document the different kinds in the following.

We make some keys optional this is marked in the following way:

api = {
  'unique-name': {
    'some_key': ...
    Optional('an_optional_key'): ...
  },
  ...
}

namespace Kind

Python dictionary representing a C++ namespace:

info = {
  'kind': 'namespace',
  'name': 'unqualified-name',
  'scope': 'unique-name' | None,
  'members: [ 'unique-name', 'unique-name' ],
  'briefdescription': paragraphs,
  'detaileddescription': paragraphs,
  'inline': True | False
}

Note: Currently Doxygen does not support parsing inline namespaces. So you need to use the patch API to change the value from False to True manually. Maybe at some point https://github.com/doxygen/doxygen/issues/6741 it will be supported.

class | struct Kind

Python dictionary representing a C++ class or struct:

info = {
  'kind': 'class' | 'struct',
  'name': 'unqualified-name',
  'location': location,
  'scope': 'unique-name' | None,
  'access': 'public' | 'protected' | 'private',
  Optional('template_parameters'): template_parameters,
  'members: [ 'unique-name', 'unique-name' ],
  'briefdescription': paragraphs,
  'detaileddescription': paragraphs
}

enum | enum class Kind

Python dictionary representing a C++ enum or enum class:

info = {
  'kind': 'enum',
  'name': 'unqualified-name',
  'location': location,
  'scope': 'unique-name' | None,
  'access': 'public' | 'protected' | 'private',
  'values: [
    {
      'name': 'somename',
      'briefdescription': paragraphs,
      'detaileddescription': paragraphs,
      Optional('value'): 'some value'
    }
   ],
  'briefdescription': paragraphs,
  'detaileddescription': paragraphs
}

typedef | using Kind

Python dictionary representing a C++ using or typedef statement:

info = {
  'kind': 'typedef' | 'using',
  'name': 'unqualified-name',
  'location': location,
  'scope': 'unique-name' | None,
  'access': 'public' | 'protected' | 'private',
  'type': type,
  'briefdescription': paragraphs,
  'detaileddescription': paragraphs
}

file Kind

Python dictionary representing a file in the project:

info = {
  'kind': 'file',
  'name': 'somefile.hpp',
  'path': 'relative/path/to/somefile.hpp',
}

function Kind

Python dictionary representing a C++ function:

  info = {
    'kind': 'function',
    'name': 'unqualified-name',
    'location': location,
    'scope': 'unique-name' | None,
    Optional('return'): {
      'type': type,
      'description': paragraphs
    }
    Optional('template_parameters'): template_parameters,
    'is_const': True | False,
    'is_static': True | False,
    'is_virtual': True | False,
    'is_explicit': True | False,
    'is_inline': True | False,
    'is_constructor': True | False,
    'is_destructor': True | False,
    'trailing_return': True | False,
    'access': 'public' | 'protected' | 'private',
    'briefdescription: paragraphs,
    'detaileddescription: paragraphs,
    'parameters': [
      { 'type': type, Optional('name'): 'somename', 'description': paragraphs },
      ...
    ]
}

The return key is optional if the function is either a constructor or destructor.

variable Kind

Python dictionary representing a C++ variable:

info = {
  'kind': 'variable',
  'name': 'unqualified-name',
  Optional('value'): 'some value',
  'type': type,
  'location': location,
  'is_static': True | False,
  'is_mutable': True | False,
  'is_volatile': True | False,
  'is_const': True | False,
  'is_constexpr': True | False,
  'scope': 'unique-name' | None,
  'access': 'public' | 'protected' | 'private',
  'briefdescription: paragraphs,
  'detaileddescription: paragraphs,
}

location item

Python dictionary representing a location:

location = {
  Optional('include'): 'some/header.h',
  'path': 'src/project/header.h',
  'line-start': 10,
  'line-end': 12 | None
  }
  • The include will be relative to any include_paths specified in the wurfapi dictionary in your Sphinx conf.py.
  • The path will be relative to the project root folder.

type item

Python list representing a C++ type:

type = [
  {
    'value': 'sometext',
    Optional('link'): link
  }, ...
]

Having the type as a list of items we can create links to nested types e.g. say we have a std::unique_ptr<impl> and we would like to make impl a link. This could look like:

"type": [
  {
    "value": "std::unique_ptr<"
  },
  {
    "link": {"url": False, "value": "project::impl"},
    "value": "impl"
  },
  {
    "value": ">"
  }
]

Any spaces in the type list should be preserved all the way from the Doxygen output and into the type list. In the rst it should be sufficient to simply output the values of the type. No spaces or other stuff should be injected.

parameter item

Dictionary representing a function parameter:

parameter = {
  'type': type,
  Optional('name'): 'somestring',
  Optional('description'): paragraphs
}

For the parameter the name is also included into the type list. The reason is that some parameters can be pretty complex, with the name embedded inside the type e.g.:

void function(int (*(*foo)())[3]);

This is a function which takes one parameter foo which is pointer function returning pointer to array 3 of int - nice right? Anyway, in such cases the parameter name is embedded inside the type of the parameter. We therefore took the easy out and wurfapi will always include the parameter name in the type.

As an example the parameter dictionary for a function void test(int b) could be:

{
   'type': [{'value': 'int '}, {'value': 'b'}],
   'name': 'b'
}

template_parameters item

Python list of dictionaries representing template parameters:

template_parameters = [{
  'type': type,
  'name': 'somestring',
  Optional('default'): type,
  Optional('description'): paragraphs
}]

Text information

Text information is stored in a of list paragraphs:

paragraphs = [paragraph]

A paragraph consists of a list of paragraph elements:

paragraph = [
      {
        "kind": "text" | "code" | "list",
        ...
      },
    ]

Paragraph elements can be one of three kinds, “text”, “code” or “list”:

text = {
  'kind': 'text',
  'content': 'hello',
  Optional('link'): link
  }

code = {
  'kind': 'code',
  'content': 'void print();',
  'is_block': true | false
}

list = {
  'kind': 'list',
  'ordered': true | false,
  'items': [paragraphs] # Each item is a list of paragraphs
}

Problem with unique-name for functions

Issue equivalent C++ function signatures can be written in a number of different ways:

void hello(const int *x); // x is a pointer to const int
void hello(int const *x); // x is a pointer to const int

We can also move the asterisk (*) to the left:

void hello(const int* x); // x is a pointer to const int
void hello(int const* x); // x is a pointer to const int

So we need some way to normalize the function signature when transforming it to unique-name. We cannot simply rely on sting comparisons.

According to the numerous google searches it is hard to write a regex for this. Instead we will try to use a parser:

We only need to parse the function parameter list denoted as the http://www.externsoft.ch/media/swf/cpp11-iso.html#parameters_and_qualifiers.

Generated output

Since we are going to be using Doxygen’s XML output as input to the extension we need a place to store it. We store it system temporary folder e.g. if the project name is “foobar” on Linux this would be /tmp/wurfapi-foobar-123456 where 123456 is a hash of the source directory paths. In addition to Doxygen’s XML we also store the generated rst for the different directives there. This is nice for debugging to see whether we generate broken rst.

The API in json format can be found in the _build/.doctree/wurfapi_api.json.

Paths and directories

  • Source directory: In Sphinx the source directory is where our .rst files are located. This is what you pass to sphinx-build when building your documentation. We will use this in our extension to find the C++ source code and output customization templates.

Notes

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for wurfapi, version 6.0.1
Filename, size File type Python version Upload date Hashes
Filename, size wurfapi-6.0.1-py2.py3-none-any.whl (43.1 kB) File type Wheel Python version py2.py3 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page