dwarfgen 0.3

IDL/Code generation utility

Dwarfgen

Description

Use dwarfgen to

• Generate datastructures that promote easy domain specific analysis of code
• Generate code in other languages
• Generate IDL's (Interactive Data Language)

Dwarfgen introduces JIDL (JSON IDL) to the already crowded IDL space. JIDL is less about being an IDL and more about being an intermediate representation of your code for downstream processing.

// sample.cpp, compiled with '-g -fPIC -gdwarf-2' and default alignment
typedef struct MyStruct
{
    char a;
    int b;
} MyStruct;

// output of 'dwarfgen --file sample.o --to-idl jidl --to-idl-dest ~/jidl'
{
    "namespaces": {},
    "enumerations": {},
    "unions": {},
    "structures": {
        "MyStruct": {
            "byteSize": 8,
            "members": {
                "a": {
                    "byteOffset": 0,
                    "byteSize": 1,
                    "type": "char",
                    "accessibility": "public"
                },
                "b": {
                    "byteOffset": 4,
                    "byteSize": 4,
                    "type": "int",
                    "accessibility": "public"
                }
            }
        }
    }
}

# Some simple domain-specific static analysis
import json

with open('/path/to/jidl.json', 'r') as f:
    jidl = json.load(f)

# list all public integers
for name, data in jidl['structures'].items():
    for member_name, member_data in data['members'].items():
        if member_data['accessibility'] == 'public' and member_data['type'] == 'int':
            print ("{}.{}".format(name, member_name) + " is a public int!")

The above example isn't extremely complicated or even useful. It's meant to illustrate the simplicity of JIDL analysis where source code analysis would be rather difficult to implement correctly. The amount of effort required for source level analysis only increases when multiple languages are involved.

A more useful example would be to ensure types are identical in size, within and across shared objects.

import json

with open('/path/to/jidl_1.json', 'r') as f:
    jidl_1 = json.load(f)

with open('/path/to/jid_2.json', 'r') as f:
    jidl_2 = json.load(f)

# get structure 1 size
structure_1_size = jidl_1['structures']['Structure1']['byteSize']

# get structure 2 size, that resides in a different namespace and is of a different name
structure_2_size = jidl_2['namespaces']['InnerNamespace']['structures']['Structure2']['byteSize']

# ensure they are equal
if structure_1_size != structure_2_size:
    print ("Structure1 != InnerNamespace::Structure2 !!")

One thing that can be easily missed when analyzing source code directly are packing rules of languages as well as compiler options that can adjust alignment of structures.

For example

typedef struct StructA {
    char a;
    int b;
} StructA;

#pragma pack(1)
typedef struct StructB {
    char a;
    int b;
} StructB;

StructA bytesize is 8 while StructB bytesize is 5. The #pragma pack(1) statement tells the compiler to not add padding to the structure. This is easily handled by dwarfgen but can be difficult to implement correctly otherwise.

In general, analyzing the output of the compiler hides almost all complicating factors versus analyzing the source code directly. The simple reason is that you would have to re-implement all compiler specific rules in your analysis (including platform specific differences)

Install

pip install dwarfgen

DWARF Version Support

• :heavy_check_mark: - Language specific features implemented
• :warning: - Core DWARF features implemented, language specific features are not implemented
• :x: - Not implemented

	Ada	C	C++	Cobol	Fortran
DWARF v2	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:warning:	:warning:
DWARF v3	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:warning:	:warning:
DWARF v4	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:warning:	:warning:
DWARF v5	:x:	:x:	:x:	:x:	:x:

Compiler Support

• :heavy_check_mark: - Tested
• :warning: - Untested
• :x: - Not Implemented

	gcc	llvm
DWARF v2	:heavy_check_mark:	:warning:
DWARF v3	:heavy_check_mark:	:warning:
DWARF v4	:heavy_check_mark:	:warning:
DWARF v5	:x:	:x:

Language-Feature Support

• :heavy_check_mark: - Implemented
• :warning: - Results may vary
• :x: - Not implemented

Common	Accessibility	Arrays	Custom Types
DWARF v2	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
DWARF v3	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
DWARF v4	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
DWARF v5	:x:	:x:	:x:

Ada	Constrained Types	Records	Repspec	Tagged Types
DWARF v2	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
DWARF v3	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
DWARF v4	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
DWARF v5	:x:	:x:	:x:	:x:

C	Enumerations	Structures	Unions	Bitfields
DWARF v2	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
DWARF v3	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
DWARF v4	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
DWARF v5	:x:	:x:	:x:	:x:

C++	Enumerations	Structures	Classes	Unions	Bitfields	Inheritance	STL	Templates
DWARF v2	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:warning:	:warning:
DWARF v3	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:warning:	:warning:
DWARF v4	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:warning:	:warning:
DWARF v5	:x:	:x:	:x:	:x:	:x:	:x:	:x:	:x:

Examples

# Help command
python -m dwarfgen -h

# Generate JIDL from a shared object
python -m dwarfgen --file /path/to/shared_object.so --to-idl jidl --to-idl-dest ~/jidl

# Generate cpp code from a shared object
python -m dwarfgen --file /path/to/shared_object.so --to-lang cpp --to-lang-dest ~/autogen/cpp

# Register a custom language generator module and generate in that language
python -m dwarfgen --file /path/to/shared_object.so \
                   --lang-generator python /path/to/module \
                   --to-lang python \
                   --to-lang-dest ~/autogen/python