Skip to main content

Only one-way pipeline (pyplyn) for data handling in Python

Project description

pyplyn: One-way only pipeline for data handling

https://badge.fury.io/py/pyplyn.svg https://pypip.in/d/pyplyn/badge.png https://pypip.in/license/pyplyn/badge.png

Pyplyn is an MIT Licensed simple flow-based data handling structure for making data handling repetitive tasks, easily without repeating yourself for every different scenario.

It is based on Python’s lovely generators, so for every data flow into the pipe is in an iterative fashion. It is currently used in a research project to handle some repetitive daily tasks. (Moving, filtering, altering the data)

Still the pyplyn module that is used in the project has some dirty but useful components like progressbar, ML based classification filter and so on, with this simple library, I think there can be a common simple ground for handling our repetitive tasks.

Installation

In order to install pyplyn, just simply:

pip install pyplyn

Or alternatively, download the package from pypi, extract and execute:

python setup.py install

Quick Start

Pyplyn aims to make data handling in a flow based fashion:

import pyplyn as p

pipe = p.Pipe()
pipe.add(p.LineReader("hello.txt"))
pipe.add(p.LambdaFilter(lambda line: len(line) < 50))
pipe.add(p.LineWriter("small_hello.txt"))
pipe.run()

You can even write your own Pyp modules as simple as this:

import pyplyn as p
import pymongo

class MongoCollection(p.InPypElement):
    def __init__(self, db, collection):
        self.collection = pymongo.MongoClient()[db][collection]
    def grasp(self):
        for document in self.collection:
            yield document

Add this new pipe element to your current flow by:

pipe = p.Pipe()
pipe.add(MongoCollection("data","raw"))
pipe.add(p.LambdaExtension(lambda document: document["text"])
pipe.add(p.LineWriter("data_text.txt"))

Documentation

Sorry, it is currently not available, but I recommend you to check the source, it is pretty straightforward for now.

Contribute

Any contribution is welcome.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyplyn-0.1.5.tar.gz (3.9 kB view details)

Uploaded Source

Built Distribution

pyplyn-0.1.5.macosx-10.9-intel.exe (69.5 kB view details)

Uploaded Source

File details

Details for the file pyplyn-0.1.5.tar.gz.

File metadata

  • Download URL: pyplyn-0.1.5.tar.gz
  • Upload date:
  • Size: 3.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for pyplyn-0.1.5.tar.gz
Algorithm Hash digest
SHA256 e8cecc95d7210eaf6a142ef02176be6df6c8e27e461e50cd20bfb447ee6ae741
MD5 fd05d15b348b2c038677d319aa4dd59a
BLAKE2b-256 f2bf61b5db084551c108c0e775513864f75082ab899ff1a3ecf6e0a8fe5d9120

See more details on using hashes here.

File details

Details for the file pyplyn-0.1.5.macosx-10.9-intel.exe.

File metadata

File hashes

Hashes for pyplyn-0.1.5.macosx-10.9-intel.exe
Algorithm Hash digest
SHA256 7446cf5aaa44ad82e424cdb3171fa6cf90624750f50e514ff96c44dde1ef85b6
MD5 f52382362e240dfd2d5949ee43dfd14f
BLAKE2b-256 967a72ddb4df342e1b8a7d143729b1594a4354ac0cf61af821cd7284771bd588

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page