Main Features

• Expressive and Extensible APIs: Maki-Nage is based on ReactiveX.

• Deployment Ready: Maki-Nage uses Kafka to scale the workload, and be resilient to errors.

• Unifies Batch and Stream processing: The same APIs work on both sources of data.

• Flexible: Start working on your laptop, continue on a server, deploy on a cluster.

• ML Streaming Serving: Serve your machine learning model as a Kafka micro-service.

Installation

Maki Nage is available on PyPI:

pip install makinage

Getting started

import rx
import rxsci as rs

def rolling_mean():
    return rx.pipe(
        rs.data.roll(window=3, stride=3, pipeline=rx.pipe(
            rs.math.mean(reduce=True),
        )),
    )

data = [1, 2, 3, 4, 5, 6, 7]

rx.from_(data).pipe(
    rs.state.with_memory_store(rx.pipe(
        rolling_mean(),
    )),
).subscribe(
    on_next=print
)

2.0
5.0

def my_app(config, data):
    roll_mean = rx.from_(data).pipe(
        rs.state.with_memory_store(rx.pipe(
            rolling_mean(),
        )),
    )

    return roll_mean,

application:
    name: my_app
kafka:
    endpoint: "localhost"
topics:
    - name: data
    - name: features
operators:
    compute_features:
        factory: my_app:my_app
        sources:
            - data
        sinks:
            - features

makinage --config myconfig.yaml

Serve Machine Learning Models

Maki Nage contains a model serving tool. With it, serving a machine learning model in streaming mode just requires a configuration file:

application:
    name: my_model_serving
Kafka:
    endpoint: "localhost"
topics:
- name: data
  encoder: makinage.encoding.json
- name: model
  encoder: makinage.encoding.none
  start_from: last
- name: predict
  encoder: makinage.encoding.json
operators:
  serve:
    factory: makinage.serve:serve
    sources:
      - model
      - data
    sinks:
      - predict
config:
  serve: {}

And then serving the model it done the same way than any makinage application:

makinage --config config.serve.yaml

Some pre and post processing steps are possible if input features or predictions must be modified before/after the inference:

Read the book to learn more.

Publications

• Toward Data Science: Stream Processing Made Easy

• KDnuggets: Real-Time Histogram Plots on Unbounded Data

License

Maki Nage is publised under the MIT License.