Skip to main content

Build multimodal AI services via cloud native technologies · Neural Search · Generative AI · MLOps

Project description

Cmon logo: Build multimodal AI services via cloud native technologies · Neural Search · Generative AI · Cloud Native

Build multimodal AI services with cloud native technologies

PyPI PyPI - Downloads from official pypistats Github CD status

Cmon lets you build multimodal AI services and pipelines that communicate via gRPC, HTTP and WebSockets, then scale them up and deploy to production. You can focus on your logic and algorithms, without worrying about the infrastructure complexity.

Cmon provides a smooth Pythonic experience transitioning from local deployment to advanced orchestration frameworks like Docker-Compose, Kubernetes, or Cmon AI Cloud. Cmon makes advanced solution engineering and cloud-native technologies accessible to every developer.

  • Build applications for any data type, any mainstream deep learning framework, and any protocol.
  • Design high-performance microservices, with easy scaling, duplex client-server streaming, and async/non-blocking data processing over dynamic flows.
  • Docker container integration via Executor Hub, OpenTelemetry/Prometheus observability, and fast Kubernetes/Docker-Compose deployment.
  • CPU/GPU hosting via Cmon AI Cloud.
Wait, how is Cmon different from FastAPI? Cmon's value proposition may seem quite similar to that of FastAPI. However, there are several fundamental differences:

Data structure and communication protocols

  • FastAPI communication relies on Pydantic and Cmon relies on docarray allowing Cmon to support multiple protocols to expose its services.

Advanced orchestration and scaling capabilities

  • Cmon lets you deploy applications formed from multiple microservices that can be containerized and scaled independently.
  • Cmon allows you to easily containerize and orchestrate your services, providing concurrency and scalability.

Journey to the cloud

  • Cmon provides a smooth transition from local development (using docarray) to local serving using (Cmon's orchestration layer) to having production-ready services by using Kubernetes capacity to orchestrate the lifetime of containers.
  • By using Cmon AI Cloud you have access to scalable and serverless deployments of your applications in one command.

Documentation

Install

Note: (Windows) not supported at this moment! You may require to install it on (WSL). [ UPDATE ] : Windows Support added!

pip install cmon-ai

Find more install options on Apple Silicon

Get Started

Basic Concepts

Cmon has four fundamental concepts:

  • A Document (from docarray) is the input/output format in Cmon.
  • An Executor is a Python class that transforms and processes Documents.
  • A Deployment serves a single Executor, while a Flow serves Executors chained into a pipeline.

The full glossary is explained here.

Build AI Services

Let's build a fast, reliable and scalable gRPC-based AI service. In Cmon we call this an Executor. Our simple Executor will wrap the StableLM LLM from Stability AI. We'll then use a Deployment to serve it.

Note A Deployment serves just one Executor. To combine multiple Executors into a pipeline and serve that, use a Flow.

Let's implement the service's logic:

executor.py
from cmon import Executor, requests
from docarray import DocumentArray

from transformers import pipeline


class StableLM(Executor):

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.generator = pipeline('text-generation', model='stablelm-3b')

    @requests
    def generate(self, docs: DocumentArray, **kwargs):
        generated_text = self.generator(docs.texts)
        docs.texts = [gen[0]['generated_text'] for gen in generated_text]

Then we deploy it with either the Python API or YAML:

Python API: deployment.py YAML: deployment.yml
from cmon import Deployment
from executor import StableLM

dep = Deployment(uses=StableLM, timeout_ready=-1, port=12345)

with dep:
    dep.block()
jtype: Deployment
with:
  uses: StableLM
  py_modules:
    - executor.py
  timeout_ready: -1
  port: 12345

And run the YAML Deployment with the CLI: cmon deployment --uses deployment.yml

Use Cmon Client to make requests to the service:

from docarray import Document
from cmon import Client

prompt = Document(
    tags = {'prompt': 'suggest an interesting image generation prompt for a mona lisa variant'}
)

client = Client(port=12345)  # use port from output above
response = client.post(on='/', inputs=[prompt])

print(response[0].text)
a steampunk version of the Mona Lisa, incorporating mechanical gears, brass elements, and Victorian era clothing details

Note In a notebook, you can't use deployment.block() and then make requests to the client. Please refer to the Colab link above for reproducible Jupyter Notebook code snippets.

Build a pipeline

Sometimes you want to chain microservices together into a pipeline. That's where a Flow comes in.

A Flow is a DAG pipeline, composed of a set of steps, It orchestrates a set of Executors and a Gateway to offer an end-to-end service.

Note If you just want to serve a single Executor, you can use a Deployment.

For instance, let's combine our StableLM language model with a Stable Diffusion image generation service from Cmon AI's Executor Hub. Chaining these services together into a Flow will give us a service that will generate images based on a prompt generated by the LLM.

Build the Flow with either Python or YAML:

Python API: flow.py YAML: flow.yml
from cmon import Flow
from executor import StableLM

flow = (
    Flow()
    .add(uses=StableLM, timeout_ready=-1, port=12345)
    .add(
        uses='cmonai://cmon.pw/TextToImage',
        timeout_ready=-1,
        install_requirements=True,
    )
)  # use the Executor from Cmon's Executor hub

with flow:
    flow.block()
jtype: Flow
with:
    port: 12345
executors:
  - uses: StableLM
    timeout_ready: -1
    py_modules:
      - executor.py
  - uses: cmonai://cmon.pw/TextToImage
    timeout_ready: -1
    install_requirements: true

Then run the YAML Flow with the CLI: cmon flow --uses flow.yml

Then, use Cmon Client to make requests to the Flow:

from cmon import Client, Document

client = Client(port=12345)

prompt = Document(
    tags = {'prompt': 'suggest an interesting image generation prompt for a mona lisa variant'}
)

response = client.post(on='/', inputs=[prompt])

response[0].display()

Deploy to the cloud

You can also deploy a Flow to JCloud.

First, turn the flow.yml file into a JCloud-compatible YAML by specifying resource requirements and using containerized Hub Executors.

Then, use cmon cloud deploy command to deploy to the cloud:

wget https://raw.githubusercontent.com/cmon.pw/cmon/master/.github/getting-started/jcloud-flow.yml
cmon cloud deploy jcloud-flow.yml

Warning

Make sure to delete/clean up the Flow once you are done with this tutorial to save resources and credits.

Read more about deploying Flows to JCloud.

Check the getting-started project source code.

Easy scalability and concurrency

Why not just use standard Python to build that microservice and pipeline? Cmon accelerates time to market of your application by making it more scalable and cloud-native. Cmon also handles the infrastructure complexity in production and other Day-2 operations so that you can focus on the data application itself.

Increase your application's throughput with scalability features out of the box, like replicas, shards and dynamic batching.

Let's scale a Stable Diffusion Executor deployment with replicas and dynamic batching:

  • Create two replicas, with a GPU assigned for each.
  • Enable dynamic batching to process incoming parallel requests together with the same model inference.
Normal Deployment Scaled Deployment
jtype: Deployment
with:
  timeout_ready: -1
  uses: cmonai://cmon.pw/TextToImage
  install_requirements: true
jtype: Deployment
with:
  timeout_ready: -1
  uses: cmonai://cmon.pw/TextToImage
  install_requirements: true
  env:
   CUDA_VISIBLE_DEVICES: RR
  replicas: 2
  uses_dynamic_batching: # configure dynamic batching
    /default:
      preferred_batch_size: 10
      timeout: 200

Assuming your machine has two GPUs, using the scaled deployment YAML will give better throughput compared to the normal deployment.

These features apply to both Deployment YAML and Flow YAML. Thanks to the YAML syntax, you can inject deployment configurations regardless of Executor code.

Get on the fast lane to cloud-native

Using Kubernetes with Cmon is easy:

cmon export kubernetes flow.yml ./my-k8s
kubectl apply -R -f my-k8s

And so is Docker Compose:

cmon export docker-compose flow.yml docker-compose.yml
docker-compose up

Note You can also export Deployment YAML to Kubernetes and Docker Compose.

That's not all. We also support OpenTelemetry, Prometheus, and Jaeger.

What cloud-native technology is still challenging to you? Tell us and we'll handle the complexity and make it easy for you.

Support

Join Us

Cmon is backed by Cmon AI and licensed under Apache-2.0.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cmon-ai-0.42.6.tar.gz (577.5 kB view details)

Uploaded Source

File details

Details for the file cmon-ai-0.42.6.tar.gz.

File metadata

  • Download URL: cmon-ai-0.42.6.tar.gz
  • Upload date:
  • Size: 577.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.1

File hashes

Hashes for cmon-ai-0.42.6.tar.gz
Algorithm Hash digest
SHA256 ccc579d10d60098cfa75496737eb4bfd5a06ae2c78b414c1f2e3f7843871b0ab
MD5 98500c664b52e42e2cbe434968c02100
BLAKE2b-256 3142f4484b0cfbaa3366a4f66d12d455e3f0143df6dbc80248f7f9f08ac417b0

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page