Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

A library for transparent transformation of indexable containers (lists, etc.)

Project description

PyPi package Continuous integration Documentation Code quality analysis Tests coverage Citable paper

SeqTools

SeqTools facilitates the manipulation of datasets and the evaluation of a transformation pipeline. Some of the provided functionalities include: mapping element-wise operations, reordering, reindexing, concatenation, joining, slicing, minibatching, etc.

To improve ease of use, SeqTools manipulates list-like objects, otherwise known as a sequences (objects with a length supporting integer or slice based indexing).

Manipulating a dataset as a whole can be slow and resource/memory intensive. To circumvent this issue, SeqTools implements on-demand evaluation under the hood: operations and transformations on a dataset are only applied to individual items when they are actually accessed. This is particularly convenient for prototyping.

When comes the transition from prototyping to execution, the list-like container interface facilitates serial evaluation. Besides, SeqTools also provides simple helpers to dispatch work between multiple workers (threads or processes).

SeqTools originally targets data science, more precisely the data preprocessing stages. Being aware of the experimental nature of this usage, on-demand execution is made as transparent as possible to users by providing fault-tolerant functions and insightful error reporting. Moreover, internal code is kept concise and clear with comments to facilitate error tracing through a failing transformation pipeline.

Example

>>> def f1(x):
... return x + 1
...
>>> def f2(x): # slow and memory heavy transformation
... time.sleep(.01)
... return [x for _ in range(500)]
...
>>> def f3(x):
... return sum(x) / len(x)
...
>>> data = list(range(1000))

Without delayed evaluation, defining the pipeline and reading values looks like so:

>>> tmp1 = [f1(x) for x in data]
>>> tmp2 = [f2(x) for x in tmp1] # takes 10 seconds and a lot of memory
>>> res = [f3(x) for x in tmp2]
>>> print(res[2])
3.0
>>> print(max(tmp2[2])) # requires to store 499 500 useless values along
3

With seqtools:

>>> tmp1 = seqtools.smap(f1, data)
>>> tmp2 = seqtools.smap(f2, tmp1)
>>> res = seqtools.smap(f3, tmp2) # no computations so far
>>> print(res[2]) # takes 0.01 seconds
3.0
>>> print(max(tmp2[2])) # easy access to intermediate results
3

Batteries included!

The library comes with a set of functions to manipulate sequences:

concatenate
batch
gather
 
prefetch
interleaving
 

and others (suggestions are also welcome).

Installation

pip install seqtools

Documentation

The documentation is hosted at https://seqtools-doc.readthedocs.io.

Contributing and Support

Use the issue tracker to request features, propose improvements or report issues. For questions regarding usage, please send an email.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for SeqTools, version 1.0.0
Filename, size File type Python version Upload date Hashes
Filename, size SeqTools-1.0.0-cp35-cp35m-manylinux1_i686.whl (44.2 kB) File type Wheel Python version cp35 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp35-cp35m-manylinux1_x86_64.whl (40.2 kB) File type Wheel Python version cp35 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp35-cp35m-manylinux2010_i686.whl (44.2 kB) File type Wheel Python version cp35 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp35-cp35m-manylinux2010_x86_64.whl (40.2 kB) File type Wheel Python version cp35 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp36-cp36m-manylinux1_i686.whl (44.4 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (40.2 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp36-cp36m-manylinux2010_i686.whl (44.4 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp36-cp36m-manylinux2010_x86_64.whl (40.2 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp37-cp37m-manylinux1_i686.whl (45.3 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp37-cp37m-manylinux1_x86_64.whl (40.2 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp37-cp37m-manylinux2010_i686.whl (45.3 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp37-cp37m-manylinux2010_x86_64.whl (40.2 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp38-cp38-manylinux1_i686.whl (45.3 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp38-cp38-manylinux1_x86_64.whl (41.1 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp38-cp38-manylinux2010_i686.whl (45.3 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size SeqTools-1.0.0-cp38-cp38-manylinux2010_x86_64.whl (41.1 kB) File type Wheel Python version cp38 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page