cs.binary 20240316

Facilities associated with binary data parsing and transcription. The classes in this module support easy parsing of binary data structures, returning instances with the binary data decoded into attributes and capable of transcribing themselves in binary form (trivially via `bytes(instance)` and also otherwise).

Facilities associated with binary data parsing and transcription. The classes in this module support easy parsing of binary data structures, returning instances with the binary data decoded into attributes and capable of transcribing themselves in binary form (trivially via bytes(instance) and also otherwise).

Latest release 20240316: Fixed release upload artifacts.

Note: this module requires Python 3.6+ because various default behaviours rely on dicts preserving their insert order.

See cs.iso14496 for an ISO 14496 (eg MPEG4) parser built using this module.

Terminology used below:

• buffer: an instance of cs.buffer.CornuCopyBuffer, which presents an iterable of bytes-like values via various useful methods; it also has a few factory methods to make one from a variety of sources such as bytes, iterables, binary files, mmapped files, TCP data streams, etc.
• chunk: a piece of binary data obeying the buffer protocol, almost always a bytes instance or a memoryview, but in principle also things like bytearray.

There are 5 main classes on which an implementor should base their data structures:

• BinarySingleStruct: a factory for classes based on a struct.struct format string with a single value; this builds a namedtuple subclass
• BinaryMultiStruct: a factory for classes based on a struct.struct format string with multiple values; this also builds a namedtuple subclass
• BinarySingleValue: a base class for subclasses parsing and transcribing a single value
• BinaryMultiValue: a base class for subclasses parsing and transcribing multiple values with no variation
• SimpleBinary: a base class for subclasses with custom .parse and .transcribe methods, for structures with variable fields

All the classes derived from the above inherit all the methods of BinaryMixin. Amongst other things, this means that the binary transcription can be had simply from bytes(instance), although there are more transcription methods provided for when greater flexibility is desired. It also means that all classes have parse* and scan* methods for parsing binary data streams.

You can also instantiate objects directly; there's no requirement for the source information to be binary.

There are several presupplied subclasses for common basic types such as UInt32BE (an unsigned 32 bit big endian integer).

Class AbstractBinary(BinaryMixin)

Abstract class for all Binary* implementations, specifying the parse and transcribe methods and providing the methods from BinaryMixin.

Class BinaryByteses(AbstractBinary, BinaryMixin)

A list of bytes parsed directly from the native iteration of the buffer.

Function BinaryFixedBytes(class_name, length: int)

Factory for an AbstractBinary subclass matching length bytes of data. The bytes are saved as the attribute .data.

Class BinaryListValues(AbstractBinary, BinaryMixin)

A list of values with a common parse specification, such as sample or Boxes in an ISO14496 Box structure.

Class BinaryMixin

Presupplied helper methods for binary objects.

Naming conventions:

• parse* methods parse a single instance from a buffer
• scan* methods are generators yielding successive instances from a buffer

Function BinaryMultiStruct(class_name: str, struct_format: str, field_names: Union[str, List[str]])

A class factory for AbstractBinary namedtuple subclasses built around complex struct formats.

Parameters:

• class_name: name for the generated class
• struct_format: the struct format string
• field_names: field name list, a space separated string or an interable of strings

Example:

# an "access point" record from the .ap file
Enigma2APInfo = BinaryMultiStruct('Enigma2APInfo', '>QQ', 'pts offset')

# a "cut" record from the .cuts file
Enigma2Cut = BinaryMultiStruct('Enigma2Cut', '>QL', 'pts type')

Function BinaryMultiValue(class_name, field_map, field_order=None)

Construct a SimpleBinary subclass named class_name whose fields are specified by the mapping field_map.

The field_map is a mapping of field name to buffer parsers and transcribers.

Note: if field_order is not specified it is constructed by iterating over field_map. Prior to Python 3.6, dicts do not provide a reliable order and should be accompanied by an explicit field_order. From 3.6 onward a dict is enough and its insertion order will dictate the default field_order.

For a fixed record structure the default .parse and .transcribe methods will suffice; they parse or transcribe each field in turn. Subclasses with variable records should override the .parse and .transcribe methods accordingly.

The field_map is a mapping of field name to a class returned by the pt_spec() function.

If the class has both parse_value and transcribe_value methods then the value itself will be directly stored. Otherwise the class it presumed to be more complex subclass of AbstractBinary and the instance is stored.

Here is an example exhibiting various ways of defining each field:

• n1: defined with the *_value methods of UInt8, which return or transcribe the int from an unsigned 8 bit value; this stores a BinarySingleValue whose .value is an int

• n2: defined from the UInt8 class, which parses an unsigned 8 bit value; this stores an UInt8 instance (also a BinarySingleValue whole .value is an int)

• n3: like n2

• data1: defined with the *_value methods of BSData, which return or transcribe the data bytes from a run length encoded data chunk; this stores a BinarySingleValue whose .value is a bytes

• data2: defined from the BSData class which parses a run length encoded data chunk; this is a BinarySingleValue so we store its bytes value directly.

  >>> class BMV(BinaryMultiValue("BMV", {
  ...         'n1': (UInt8.parse_value, UInt8.transcribe_value),
  ...         'n2': UInt8,
  ...         'n3': UInt8,
  ...         'nd': ('>H4s', 'short bs'),
  ...         'data1': (
  ...             BSData.parse_value,
  ...             BSData.transcribe_value,
  ...         ),
  ...         'data2': BSData,
  ... })):
  ...     pass
  >>> BMV.FIELD_ORDER
  ['n1', 'n2', 'n3', 'nd', 'data1', 'data2']
  >>> bmv = BMV.from_bytes(b'\x11\x22\x77\x81\x82zyxw\x02AB\x04DEFG')
  >>> bmv.n1  #doctest: +ELLIPSIS
  17
  >>> bmv.n2
  34
  >>> bmv  #doctest: +ELLIPSIS
  BMV(n1=17, n2=34, n3=119, nd=nd_1_short__bs(short=33154, bs=b'zyxw'), data1=b'AB', data2=b'DEFG')
  >>> bmv.nd  #doctest: +ELLIPSIS
  nd_1_short__bs(short=33154, bs=b'zyxw')
  >>> bmv.nd.bs
  b'zyxw'
  >>> bytes(bmv.nd)
  b'‚zyxw'
  >>> bmv.data1
  b'AB'
  >>> bmv.data2
  b'DEFG'
  >>> bytes(bmv)
  b'\x11"w\x81\x82zyxw\x02AB\x04DEFG'
  >>> list(bmv.transcribe_flat())
  [b'\x11', b'"', b'w', b'\x81\x82zyxw', b'\x02', b'AB', b'\x04', b'DEFG']
  

Function BinarySingleStruct(class_name, struct_format, field_name=None)

A convenience wrapper for BinaryMultiStruct for struct_formats with a single field.

Parameters:

• class_name: the class name for the generated class
• struct_format: the struct format string, specifying a single struct field
• field_name: optional field name for the value, default 'value'

Example:

>>> UInt16BE = BinarySingleStruct('UInt16BE', '>H')
>>> UInt16BE.__name__
'UInt16BE'
>>> UInt16BE.format
'>H'
>>> UInt16BE.struct   #doctest: +ELLIPSIS
<_struct.Struct object at ...>
>>> field = UInt16BE.from_bytes(bytes((2,3)))
>>> field
UInt16BE(value=515)
>>> field.value
515

Class BinarySingleValue(AbstractBinary, BinaryMixin)

A representation of a single value as the attribute .value.

Subclasses must implement:

• parse or parse_value
• transcribe or transcribe_value

Class BinaryUTF16NUL(BinarySingleValue, AbstractBinary, BinaryMixin)

A NUL terminated UTF-16 string.

Method BinaryUTF16NUL.__init__(self, value, *, encoding): pylint: disable=super-init-not-called

Class BinaryUTF8NUL(BinarySingleValue, AbstractBinary, BinaryMixin)

A NUL terminated UTF-8 string.

Class BSData(BinarySingleValue, AbstractBinary, BinaryMixin)

A run length encoded data chunk, with the length encoded as a BSUInt.

Class BSSFloat(BinarySingleValue, AbstractBinary, BinaryMixin)

A float transcribed as a BSString of str(float).

Class BSString(BinarySingleValue, AbstractBinary, BinaryMixin)

A run length encoded string, with the length encoded as a BSUInt.

Class BSUInt(BinarySingleValue, AbstractBinary, BinaryMixin)

A binary serialised unsigned int.

This uses a big endian byte encoding where continuation octets have their high bit set. The bits contributing to the value are in the low order 7 bits.

Function flatten(chunks)

Flatten chunks into an iterable of bytes instances.

This exists to allow subclass methods to easily return transcribeable things (having a .transcribe method), ASCII strings or bytes or iterables or even None, in turn allowing them simply to return their superclass' chunks iterators directly instead of having to unpack them.

An example from the cs.iso14496.METABoxBody class:

def transcribe(self):
    yield super().transcribe()
    yield self.theHandler
    yield self.boxes

The binary classes flatten the result of the .transcribe method to obtain bytes insteances for the object's bnary transcription.

Class Float64BE(Float64BE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '>d' and presents the attributes ('value',).

Class Float64LE(Float64LE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '<d' and presents the attributes ('value',).

Class Int16BE(Int16BE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '>h' and presents the attributes ('value',).

Class Int16LE(Int16LE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '<h' and presents the attributes ('value',).

Class Int32BE(Int32BE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '>l' and presents the attributes ('value',).

Class Int32LE(Int32LE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '<l' and presents the attributes ('value',).

Function pt_spec(pt, name=None)

Convert a parse/transcribe specification pt into an AbstractBinary subclass.

This is largely used to provide flexibility in the specifications for the BinaryMultiValue factory but can be used as a factory for other simple classes.

If the specification pt is a subclass of AbstractBinary this is returned directly.

If pt is a 2-tuple of str the values are presumed to be a format string for struct.struct and filed names separated by spaces; a new BinaryMultiStruct class is created from these and returned.

Otherwise two functions f_parse_value(bfr) and f_transcribe_value(value) are obtained and used to construct a new BinarySingleValue class as follows:

If pt has .parse_value and .transcribe_value callable attributes, use those for f_parse_value and f_transcribe_value respectively.

Otherwise, if pt is an int define f_parse_value to obtain exactly that many bytes from a buffer and f_transcribe_value to return those bytes directly.

Otherwise presume pt is a 2-tuple of (f_parse_value,f_transcribe_value).

Class SimpleBinary(types.SimpleNamespace, AbstractBinary, BinaryMixin)

Abstract binary class based on a SimpleNamespace, thus providing a nice __str__ and a keyword based __init__. Implementors must still define .parse and .transcribe.

To constrain the arguments passed to __init__, define an __init__ which accepts specific keyword arguments and pass through to super().__init__(). Example:

def __init__(self, *, field1=None, field2):
    """ Accept only `field1` (optional)
        and `field2` (mandatory).
    """
    super().__init__(field1=field1, field2=field2)

Class UInt16BE(UInt16BE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '>H' and presents the attributes ('value',).

Class UInt16LE(UInt16LE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '<H' and presents the attributes ('value',).

Class UInt32BE(UInt32BE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '>L' and presents the attributes ('value',).

Class UInt32LE(UInt32LE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '<L' and presents the attributes ('value',).

Class UInt64BE(UInt64BE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '>Q' and presents the attributes ('value',).

Class UInt64LE(UInt64LE, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format '<Q' and presents the attributes ('value',).

Class UInt8(UInt8, builtins.tuple, AbstractBinary, BinaryMixin)

An AbstractBinary namedtuple which parses and transcribes the struct format 'B' and presents the attributes ('value',).

Release Log

Release 20240316: Fixed release upload artifacts.

Release 20240201: BREAKING CHANGE: drop the long deprecated PacketField related classes.

Release 20231129: BinaryMultiStruct.parse: promote the buffer arguments to a CornuCopyBuffer.

Release 20230401:

• BinaryMixin.scan: bfr parameter may be any object acceptable to CornuCopyBuffer.promote.
• BinaryMixin.scan: accept new optional with_offsets parameter; deprecate scan_with_offsets and scan_fspathi in favour of scan.

Release 20230212:

• BinaryMixin: new load(file) and save(file) methods.
• BinaryMixin.scan: promote the bfr argument.

Release 20221206: Documentation fix.

Release 20220605: BinaryMixin: replace scan_file with scan_fspath, as the former left uncertainty about the amount of the file consumed.

Release 20210316:

• BSUInt: rename parse_bytes to decode_bytes, the former name conflicted with BinaryMixin.parse_bytes and broken the semantics.
• Minor refactors.

Release 20210306: MAJOR RELEASE: The PacketField classes and friends were hard to use; this release supplied a suite of easier to use and more consistent Binary* classes, and ports most of those things based on the old scheme to the new scheme.

Release 20200229:

• ListField: replace transcribe method with transcribe_value method, aids external use.
• Add .length attribute to struct based packet classes providing the data length of the structure (struct.Struct.size).
• Packet: new add_deferred_field method to consume the raw data for a field for parsing later (done automatically if the attribute is accessed).
• New @deferred_field decorator for the parser for that stashed data.

Release 20191230.3: Docstring tweak.

Release 20191230.2: Documentation updates.

Release 20191230.1: Docstring updates. Semantic changes were in the previous release.

Release 20191230:

• ListField: new iter method.
• Packet: str: accept optional skip_fields parameter to omit some field names.
• Packet: new .add_from_value method to add a named field with a presupplied value.
• Packet: new remove_field(field_name) and pop_field() methods to remove fields.
• BytesesField: iter yields the bytes values, transcribe=iter.
• PacketField: propagate keyword arguments through various methods, required for parameterised PacketFields.
• New UTF16NULField, a NUL terminated UTF16 string.
• PacketField: provide a default .transcribe_value method which makes a new instance and calls its .transcribe method.
• Documentation update and several minor changes.

Release 20190220:

• Packet.self_check: fields without a sanity check cause a warning, not a ValueError.
• New Float64BE, Float64LE and BSSFloat classes for IEEE floats and floats-as-strings.
• Additional module docstringage on subclassing Packet and PacketField.
• BSString: drop redundant from_buffer class method.
• PacketField.init: default to value=None if omitted.

Release 20181231: flatten: do not yield zero length bytelike objects, can be misread as EOF on some streams.

Release 20181108:

• New PacketField.transcribe_value_flat convenience method to return a flat iterable of bytes-like objects.
• New PacketField.parse_buffer generator method to parse instances of the PacketField from a buffer until end of input.
• New PacketField.parse_buffer_values generator method to parse instances of the PacketField from a buffer and yield the .value attribute until end of input.

Release 20180823:

• Some bugfixes.
• Define PacketField.eq.
• BSUInt, BSData and BSString classes implementing the serialisations from cs.serialise.
• New PacketField.value_from_bytes class method.
• New PacketField.value_from_buffer method.

Release 20180810.2: Documentation improvements.

Release 20180810.1: Improve module description.

Release 20180810: BytesesField.from_buffer: make use of the buffer's skipto method if discard_data is true.

Release 20180805:

• Packet: now an abstract class, new self_check method initially checking the
• PACKET_FIELDS class attribute against the instance, new methods get_field
• and set_field to fetch or replace existing fields, allow keyword arguments
• to initialise the Packet fields and document the dependency on keyword
• argument ordering.
• PacketField: len computed directory from a transcribe, drop other len
• methods.
• EmptyField singleton to use as a placeholder for missing optional fields.
• BytesField: implement value_s and from_buffer.
• multi_struct_field: implement len for generated class.
• flatten: treat memoryviews like bytes.
• Assorted docstrings and fixes.

Release 20180801: Initial PyPI release.