caterpillar-py 2.7.0

Library to pack and unpack structurized binary data.

Caterpillar - 🐛

Caterpillar is a Python 3.12+ library to pack and unpack structurized binary data (with support for 3.10+). It enhances the capabilities of Python Struct by enabling direct class declaration. More information about the different configuration options will be added in the future. Documentation is here >.

Caterpillar is able to:

• Pack and unpack data just from processing Python class definitions (including support for powerful bitfields, c++-like templates and c-like unions!),
• apply a wide range of data types (with endianess and architecture configuration),
• dynamically adapt structs based on their inheritance layout,
• reduce the used memory space using __slots__,
• allowing you to place conditional statements into class definitions,
• insert proper types into the class definition to support documentation and
• it helps you to create cleaner and more compact code.
• There is also a feature that lets you dynamically change the endian within a struct!
• You can even extend Caterpillar and write your parsing logic in C or C++!!

[!NOTE] Python 3.14 breaks with statements in class definitions since __annotations__ are added at the end of a class definition. Therefore, Digest and conditional statements ARE NOT SUPPORTED using the with syntax in Python 3.14+. As of version 2.4.5 the Digest class has a counterpart (DigestField), which can be used to manually specify a digest without the need of a ẁith statement.

Give me some code!

from caterpillar.py import *

@bitfield(order=LittleEndian)
class Header:
    version : 4                # 4bit integer
    valid   : 1                # 1bit flag (boolean)
    ident   : (8, CharFactory) # 8bit char
    # automatic alignment to 16bits

@struct(order=LittleEndian)
class Format:
    magic  : b"ITS MAGIC"        # Supports string and byte constants directly
    header : Header
    a      : uint8               # Primitive data types
    b      : Dynamic + int32     # dynamic endian based on global config
    length : uint8               # String fields with computed lengths
    name   : String(this.length) #  -> you can also use Prefixed(uint8)

    _hash_begin : DigestField.begin("hash", Md5_Algo)   # custom actions, e.g. for hashes
    names       : CString[uint8::]                      # Sequences with prefixed, computed lengths
    hash        : Md5_Field("hash", verify=True) = None # automatic hash creation and verification + default value

# Instantiation (keyword-only arguments, magic is auto-inferred):
obj = Format(
    header=Header(version=2, valid=True, ident="F"),
    a=1,
    b=2,
    length=3,
    name="foo",
    names=["a", "b"]
)
# Packing the object, reads as 'PACK obj FROM Format'
# objects of struct classes can be packed right away
blob = pack(obj, Format)
# results in: b'ITS MAGIC0*\x01\x02\x00\x00\x00\x03foo\x02a\x00b\x00)\x9a...'

# Unpacking the binary data, reads as 'UNPACK Format FROM blob'
obj2 = unpack(Format, blob)

# to pack with a different endian for field 'b', use _order
data = pack(obj, Format, _order=BigEndian)

This library offers extensive functionality beyond basic struct handling. For further details on its powerful features, explore the official documentation, examples, and test cases.

Installation

[!NOTE] As of Caterpillar v2.1.2 it is possible to install the library without the need of compiling the C extension.

PIP installation (Python-only)

pip install caterpillar-py

Python-only installation

pip install "caterpillar[all]@git+https://github.com/MatrixEditor/caterpillar"

Installation + C-extension

pip install "caterpillar[all]@git+https://github.com/MatrixEditor/caterpillar/#subdirectory=src/ccaterpillar"

Starting Point

Please visit the Documentation, it contains a complete tutorial on how to use this library.

Other Approaches

A list of similar approaches to parsing structured binary data with Python can be taken from below:

• construct
• kaitai_struct
• hachoir
• mrcrowbar

The documentation also provides a Comparison to these approaches.

License

Distributed under the GNU General Public License (V3). See License for more information.