Skip to main content

Only one-way pipeline (pyplyn) for data handling in Python

Project description

pyplyn: One-way only pipeline for data handling

https://pypip.in/d/pyplyn/badge.png https://pypip.in/license/pyplyn/badge.png

Pyplyn is an MIT Licensed simple flow-based data handling structure for making data handling repetitive tasks, easily without repeating yourself for every different scenario.

It is based on Python’s lovely generators, so for every data flow into the pipe is in an iterative fashion. It is currently used in a research project to handle some repetitive daily tasks. (Moving, filtering, altering the data)

Still the pyplyn module that is used in the project has some dirty but useful components like progressbar, ML based classification filter and so on, with this simple library, I think there can be a common simple ground for handling our repetitive tasks.

Installation

In order to install pyplyn, just simply:

pip install pyplyn

Or alternatively, download the package from pypi, extract and execute:

python setup.py install

Quick Start

Pyplyn aims to make data handling in a flow based fashion:

import pyplyn as p

pipe = p.Pipe()
pipe.add(p.LineReader("hello.txt"))
pipe.add(p.LambdaFilter(lambda line: len(line) < 50))
pipe.add(p.LineWriter("small_hello.txt"))
pipe.run()

You can even write your own Pyp modules as simple as this:

import pyplyn as p
import pymongo

class MongoCollection(p.InPypElement):
    def __init__(self, db, collection):
        self.collection = pymongo.MongoClient()[db][collection]
    def grasp(self):
        for document in self.collection:
            yield document

Add this new pipe element to your current flow by:

pipe = p.Pipe()
pipe.add(MongoCollection("data","raw"))
pipe.add(p.LambdaExtension(lambda document: document["text"])
pipe.add(p.LineWriter("data_text.txt"))

Documentation

Sorry, it is currently not available, but I recommend you to check the source, it is pretty straightforward for now.

Contribute

Any contribution is welcome.

Project details


Release history Release notifications

This version
History Node

0.1.5

History Node

0.1.4

History Node

0.1.3

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
pyplyn-0.1.5.macosx-10.9-intel.exe (69.5 kB) Copy SHA256 hash SHA256 Windows Installer any Jul 5, 2014
pyplyn-0.1.5.tar.gz (3.9 kB) Copy SHA256 hash SHA256 Source None Jul 5, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page