Skip to main content

Incrementally decode bytes into strings and lines

Project description

# An incremental decoder of bytes into characters and lines

## The DecodeAccumulator class

The `DecodeAccumulator` class implements an incremental decoder: an object
that may be fed bytes (one or several at a time) as they are e.g. read
from a network stream or a subprocess's output, and that adds to a result
string as soon as enough bytes have been accumulated to produce a character
in the specified encoding.

Note that `DecodeAccumulator` objects are immutable value objects:
the `add()` method does not modify its invocant, but returns a new
`DecodeAccumulator` object instead.

Sample usage:

while True:
bb = subprocess.stdout.read(1024)
if len(bb) == 0:
break
acc = acc.add(bb)
assert(not acc.done)
if acc.splitter.lines:
# at least one full line was produced
(acc, lines) = acc.pop_lines()
print('\n'.join(lines)

if acc.buf:
print('Leftover bytes left in the buffer!', file=sys.stderr)

if acc.splitter.buf:
print('Incomplete line: ' + acc.splitter.buf)

final = acc.add(None)
assert(final.splitter.buf == '')
assert(final.splitter.done)
assert(final.done)
if acc.splitter.buf:
assert(len(final.splitter.lines) == len(acc.splitter.lines) + 1)

## The splitter classes: UniversalNewlines, FixedEOLSplitter, NullSplitter

The `decode_acc.newlines` module provides three classes that may be used to
split a text string into lines in different ways. The `UniversalNewlines`
class does its best to simulate the "universal newlines" behavior of `file`
objects. The `FixedEOLSplitter` class uses a specified string as a line
terminator to split on. The `NullSplitter` class does not do any splitting.

Sample usage:

spl = newlines.UniversalNewlines()
for char in input_string:
spl = spl.add(char)
spl.add(None)

for (idx, line) in enumerate(spl.lines):
print('line {idx}: {line}'.format(idx=idx, line=line))


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for decode-acc, version 0.1.1
Filename, size File type Python version Upload date Hashes
Filename, size decode_acc-0.1.1-py2-none-any.whl (9.6 kB) File type Wheel Python version py2 Upload date Hashes View
Filename, size decode_acc-0.1.1-py3-none-any.whl (9.6 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size decode-acc-0.1.1.tar.gz (5.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page