Skip to main content

Incrementally decode bytes into strings and lines

Project description

# An incremental decoder of bytes into characters and lines

## The DecodeAccumulator class

The `DecodeAccumulator` class implements an incremental decoder: an object
that may be fed bytes (one or several at a time) as they are e.g. read
from a network stream or a subprocess's output, and that adds to a result
string as soon as enough bytes have been accumulated to produce a character
in the specified encoding.

Note that `DecodeAccumulator` objects are immutable value objects:
the `add()` method does not modify its invocant, but returns a new
`DecodeAccumulator` object instead.

Sample usage:

while True:
bb =
if len(bb) == 0:
acc = acc.add(bb)
assert(not acc.done)
if acc.splitter.lines:
# at least one full line was produced
(acc, lines) = acc.pop_lines()

if acc.buf:
print('Leftover bytes left in the buffer!', file=sys.stderr)

if acc.splitter.buf:
print('Incomplete line: ' + acc.splitter.buf)

final = acc.add(None)
assert(final.splitter.buf == '')
if acc.splitter.buf:
assert(len(final.splitter.lines) == len(acc.splitter.lines) + 1)

## The splitter classes: UniversalNewlines, FixedEOLSplitter, NullSplitter

The `decode_acc.newlines` module provides three classes that may be used to
split a text string into lines in different ways. The `UniversalNewlines`
class does its best to simulate the "universal newlines" behavior of `file`
objects. The `FixedEOLSplitter` class uses a specified string as a line
terminator to split on. The `NullSplitter` class does not do any splitting.

Sample usage:

spl = newlines.UniversalNewlines()
for char in input_string:
spl = spl.add(char)

for (idx, line) in enumerate(spl.lines):
print('line {idx}: {line}'.format(idx=idx, line=line))

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

decode-acc-0.1.0.tar.gz (5.8 kB view hashes)

Uploaded Source

Built Distributions

decode_acc-0.1.0-py3-none-any.whl (9.5 kB view hashes)

Uploaded Python 3

decode_acc-0.1.0-py2-none-any.whl (9.5 kB view hashes)

Uploaded Python 2

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page