Reactive data science
Project description
Stream Processing Made Easy
Maki Nage is a Python stream processing library and framework. It provides expressive and extensible APIs. Maki Nage speeds up the development of stream applications. It can be used to process stream and batch data. More than that, it allows to develop an application with batch data, and deploy it as a Kafka micro-service.
Read the doc to learn more.
Main Features
Expressive and Extensible APIs: Maki-Nage is based on ReactiveX.
Deployment Ready: Maki-Nage uses Kafka to scale the workload, and be resilient to errors.
Unifies Batch and Stream processing: The same APIs work on both sources of data.
Flexible: Start working on your laptop, continue on a server, deploy on a cluster.
ML Streaming Serving: Serve your machine learning model as a Kafka micro-service.
Installation
Maki Nage is available on PyPI:
pip install makinage
Getting started
Write your data transforms
import rx
import rxsci as rs
def rolling_mean():
return rx.pipe(
rs.data.roll(window=3, stride=3, pipeline=rx.pipe(
rs.math.mean(reduce=True),
)),
)
Test your code on batch data
You can test your code from any python data or CSV file.
data = [1, 2, 3, 4, 5, 6, 7]
rx.from_(data).pipe(
rs.state.with_memory_store(rx.pipe(
rolling_mean(),
)),
).subscribe(
on_next=print
)
2.0
5.0
Deploy your code as a Kafka micro-service
To deploy the code, package it as a function:
def my_app(config, data):
roll_mean = rx.from_(data).pipe(
rs.state.with_memory_store(rx.pipe(
rolling_mean(),
)),
)
return roll_mean,
Create a configuration file:
application:
name: my_app
kafka:
endpoint: "localhost"
topics:
- name: data
- name: features
operators:
compute_features:
factory: my_app:my_app
sources:
- data
sinks:
- features
And start it!
makinage --config myconfig.yaml
Serve Machine Learning Models
Maki Nage contains a model serving tool. With it, serving a machine learning model in streaming mode just requires a configuration file:
application:
name: my_model_serving
Kafka:
endpoint: "localhost"
topics:
- name: data
encoder: makinage.encoding.json
- name: model
encoder: makinage.encoding.none
start_from: last
- name: predict
encoder: makinage.encoding.json
operators:
serve:
factory: makinage.serve:serve
sources:
- model
- data
sinks:
- predict
config:
serve: {}
And then serving the model it done the same way than any makinage application:
makinage --config config.serve.yaml
Some pre and post processing steps are possible if input features or predictions must be modified before/after the inference:
Read the book to learn more.
Publications
Toward Data Science: Stream Processing Made Easy
KDnuggets: Real-Time Histogram Plots on Unbounded Data
License
Maki Nage is publised under the MIT License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.