pynetstring 0.5

A module for encoding and decoding netstrings.

A module for encoding and decoding netstrings. See the definition of netstrings at https://cr.yp.to/proto/netstrings.txt.

Requirements

Pynetstring is written for Python 3.

Usage

Encoding

# Will give b'5:Hello,'
print(pynetstring.encode('Hello'))

It is also possible to encode lists of data.

# Will give [b'5:Hello,', b'5:World,']
print(pynetstring.encode(['Hello', 'World']))

Decoding

In the simplest case, when we know for sure that the data we are trying to decode ends on a netstring boundary we can simply do:

# Will give [b'Hello', b'World!']
print(pynetstring.decode('5:Hello,6:World!,'))

In many cases however, such when netstring are transmitted over a network, the chunks of data that arrive may not necessarily align to netstring boundaries. For example a chunk of data may contain a netstring and then parts of the next. To handle this, call Decoder.feed(), which will buffer and parse the data and emit decoded data as soon as one or more netstrings have been fully received.

decoder = pynetstring.Decoder()
# Will give []
print(decoder.feed('5:He'))
# Will give ['Hello', 'World!']
print(decoder.feed('llo,6:World!,5:ag'))
# Will give ['again']
print(decoder.feed('ain,'))

It is possible at any moment to check if there is partial data which is still pending to be returned from feed().

decoder.feed('3:ab')
# Will return True
print(decoder.pending())
decoder.feed('c,')
# Will return False
print(decoder.pending())

The receiving side could look something like this:

decoder = pynetstring.Decoder()

def handle_network_data(data):
  decoded_list = decoder.feed(data)
  for item in decoded_list:
      print(item)

Also the Decoder class supports limiting the maximal decoded netstring length. This is required for network applications to restrict the maximal length of the input buffer in order to prevent unintentional memory bloat or intentional misuse from malicious senders. The netstring length limit is optional and specified as the first argument to the constructor:

decoder = pynetstring.Decoder(maxlen=1024)

The Decoder will raise TooLong exception as soon as it’ll discover next string can’t fit the limit.

Stream decoding

pynetstring provides a StreamingDecoder class for cases when you need to decode a stream of very large netstrings where individual netstrings may not fit in memory or data should be processed before the entire netstring has been transmitted.

StreamingDecoder has an interface similar to Decoder class, but its feed() method returns parts of the decoded netstrings as soon as they are extracted. The end of each individual string is signalized with an empty bytestring in the sequence. In order to the collect returned strings you should accumulate the fragments from the returned sequence until an empty binary string encountered. After each empty string in sequence you should start over accumulating outputs into a new string.

For example, if feed() returns the sequence [ b’ab’, b’cd’, b’’, b’!!!!’, b’’, b’’, b’12’ ] it means we have so far received three complete strings: b’abcd’, b’!!!!’ and b’’, and part of one incomplete netstring starting with b’12’ (further parts of which will appear in subsequent calls to feed()).

Data encoding

Regardless of the type of the data that is sent to encode(), it will always return binary data, i.e. python bytes. The data that is returned from decode() and Decoder.feed() will also be binary. This is because the decoder can not make any assumptions on the encoding of the original data. If you know that the data that comes in can be interpreted in a particular way or encoding, e.g. UTF-8, you have to explicitly do that conversion.

encoded = pynetstring.encode(u'Hello World!')
# This will be <class 'bytes'>
print(type(encoded))
decoded_list = pynetstring.decode(encoded)
# This will be <class 'bytes'>
print(type(decoded_list[0]))
# This will return the original unicode string u'Hello World!'
print(decoded_list[0].decode('utf-8'))

Error handling

A ParseError subclass exception will be raised if trying to decode an invalid netstring.

# IncompleteString exception due to missing trailing comma:
pynetstring.decode('3:ABC_')

# BadLength due to no length specified
pynetstring.decode(b' :X,')

decoder = Decoder(3)
# TooLong exception due to exceeded netstring limit in stream parser:
decoder.feed(b'4:ABCD,')

# BadLength due to invalid character in length declaration:
decoder.feed(b' 1:X,')

All other exceptions of this module can be expected to be subclass of NetstringException.