Motiv, Simple and Efficient processing pipelining.
Simple and Efficient processing pipelining.
Motiv is a framework to ease and enables building minimal pipelines and pipeline-like applications. It abstracts away communication, messaging patterns and execution patterns.
When to use Motiv?
Motiv is for minimal pipelining and pipeline-like applications, you can build a tightly-coupled or a loosely-coupled pipeline using it. However, if you're pipeline's purpose is data processing, motiv is a good start. If the purpose is big data processing, then you're better off using a big data processing engine (e.g. Spark).
Motiv is simple, it provides many messaging-patterns, it manages them for you and it provides you direct control over them. Motiv is stateless, and so should the applications that use it. Motiv uses efficient and very fast communication/message-passing (zmq) for the underlying communication.
$ pip install motiv
$ git clone email@example.com:SaadTalaat/motiv.git $ cd motiv $ python3 setup.py install
Motiv has two type of patterns that are glued together to form a component in a pipeline. A stream (messaging-pattern) and an actor (execution-pattern). Streams can be a
Sink. an actor can be
Streams are boilerplate messaging-patterns that either send out messages or receive messages. for example an
Emitter has a typical publisher behavior. Unlike the
Ventilator distributes messages between listening components (workers).
An execution pattern defines how a component should execute, for example, a component can simply be a
Proxy between two streams that forwards messages received over subscriber stream to a ventilator, or from a worker to a publisher, or from a worker to a sink. It can also be a
Ticker which is execute-till-halt event-loop with each cycle calling
Ticker.tick method function, this type of behavior is convenient for processing incoming messages and sending out results to different actors or do an action depending on these messages.
Streams + Execution Pattern
An execution pattern or an actor expects to have an input stream or output stream or both for it to be runnable.
See examples: motiv/examples/
$ git clone firstname.lastname@example.org:SaadTalaat/motiv.git $ cd motiv $ python3 motiv/examples/workers.py .... $ python3 motiv/examples/pubsub.py
Motiv is still in development, master might not be stable, features will be rolled out fast. I encourage you to fix any code you see wrong, raise any issues that concern you and actively participate in developing Motiv.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.