zenml 0.30.0

ZenML: Write production-ready ML code.

Build portable, production-ready MLOps pipelines.

A simple yet powerful open-source framework that integrates all your ML tools.
Explore the docs »

Join our Slack Community and be part of the ZenML family.

Features · Roadmap · Report Bug · Vote New Features · Read Blog · Meet the Team
🎉 Version 0.30.0 is out. Check out the release notes here.

🏁 Table of Contents

1. Why ZenML?
2. What is ZenML?
3. Getting Started
   • Installation
   • First run
   • Collaborate with ZenML
   • ZenBytes
   • ZenFiles
4. Infrastructure Requirements
   • Deploy ZenML
   • MLOps Stack Recipes
5. Roadmap
6. Contributing and Community
7. Meet the Team
8. Getting Help
9. License
10. Learn More

🤖 Why ZenML?

🤹 Are you an ML engineer or data scientist shipping models to production and juggling a plethora of tools? ZenML is here to help transition your production ML pipelines to production.

Everyone loves to train ML models, but few talks about shipping them into production, and even fewer can do it well. At ZenML, we believe the journey from model development to production doesn't need to be long and painful.

If you come from unstructured notebooks or scripts with lots of manual processes, ZenML will make the path to production easier and faster for you and your team. Using ZenML allows you to own the entire pipeline - from experimentation to production.

Read more here.

💡 What is ZenML?

ZenML is an extensible, open-source MLOps framework for creating portable, production-ready MLOps pipelines. It's built for Data Scientists, ML Engineers, and MLOps Developers to collaborate as they develop to production.

ZenML offers a simple and flexible syntax, is cloud- and tool-agnostic, and has interfaces/abstractions catered toward ML workflows. With ZenML you'll have all your favorite tools in one place, so you can tailor a workflow that caters to your specific needs.

Read more on all tools you can readily use in the integrations section. Can't find your tool? You can always write your own integration to use it with ZenML.

🤸 Getting Started

💾 Installation

Option 1 - Install ZenML via PyPI:

pip install zenml

Note - ZenML supports Python 3.7, 3.8, and 3.9.

In order to get access to the dashboard locally, install the optional dependencies for the ZenML server:

pip install "zenml[server]"

Option 2 - If you’re feeling adventurous, try out the bleeding-edge installation:

pip install git+https://github.com/zenml-io/zenml.git@develop --upgrade

Warning - Fire dragons ahead. Proceed at your own risk!

Option 3 - Install via a Docker image hosted publicly on DockerHub:

docker run -it zenmldocker/zenml /bin/bash

or just run the ZenML server directly with Docker:

docker run -it -d -p 8080:80 zenmldocker/zenml-server

Warning

Known installation issues for M1 Mac users

If you have an M1 Mac machine and encounter an installation error, try setting up brew and pyenv with Rosetta 2 and then install ZenML. The issue arises because some dependencies aren’t fully compatible with the vanilla ARM64 Architecture. The following links may be helpful (Thank you @Reid Falconer) :

• Pyenv with Apple Silicon
• Install Python Under Rosetta 2

Read full installation instructions in the docs.

🏇 First run

If you're here for the first time, we recommend running:

zenml go

This spins up a Jupyter notebook that walks you through various functionalities of ZenML at a high level.

By the end, you'll get a glimpse of how to use ZenML to register a stack:

# Register a MLflow experiment tracker
zenml experiment-tracker register mlflow_tracker --flavor=mlflow

# Register an Airflow orchestrator
zenml orchestrator register airflow_orchestrator --flavor=airflow

# Create a stack from the components
zenml stack register airflow_stack -o airflow_orchestrator -a default -e mlflow_tracker

And run a simple pipeline on Airflow (or an orchestrator of your choice) creating steps like this:

import mlflow
from sklearn.base import ClassifierMixin
from sklearn.svm import SVC
from zenml.client import Client

experiment_tracker = Client().active_stack.experiment_tracker

@step(experiment_tracker=experiment_tracker.name)
def svc_trainer_mlflow(
    X_train: pd.DataFrame,
    y_train: pd.Series,
) -> ClassifierMixin:
    """Train a sklearn SVC classifier and log to MLflow."""
    mlflow.sklearn.autolog()  # log all model hparams and metrics to MLflow
    model = SVC(gamma=0.01)
    model.fit(X_train.to_numpy(), y_train.to_numpy())
    train_acc = model.score(X_train.to_numpy(), y_train.to_numpy())
    print(f"Train accuracy: {train_acc}")
    return model

You can also run your first pipeline right in Google Colab

👭 Collaborate with ZenML

ZenML is built to support teams working together. The underlying infrastructure on which your ML workflows run can be shared, as can the data, assets, and artifacts in your workflow.

To visually see this in action, ZenML ships with a fully-featured dashboard, that can be launched locally with the simple command:

zenml up

The dashboard can also be deployed with a server on any cloud service (see Deploy ZenML section).

🍰 ZenBytes

New to MLOps? Get up to speed by visiting the ZenBytes repo.

ZenBytes is a series of short practical MLOps lessons taught using ZenML. It covers many of the core concepts widely used in ZenML and MLOps in general.

📜 ZenFiles

Already comfortable with ZenML and wish to elevate your pipeline into production mode? Check out ZenFiles.

ZenFiles is a collection of production-grade ML use-cases powered by ZenML. They are fully fleshed out, end-to-end projects that showcase ZenML's capabilities. They can also serve as a template from which to start similar projects.

☁️ Infrastructure Requirements

🔋 Deploy ZenML

While ZenML works perfectly fine locally, it should be deployed on the cloud to enable collaborative features as the central MLOps interface for teams.

The easiest and fastest way to get running on the cloud is by using the deploy CLI command. It currently only supports deploying to Kubernetes on managed cloud services. Read more about the command here.

zenml deploy

Naturally, the docs contain detailed guides on how to manually set up ZenML on the cloud in case more control is required.

The following architectural diagram illustrates a common ZenML deployment:

👨‍🍳 Open Source MLOps Stack Recipes

Apart from the infrastructure required to run ZenML itself, ZenML also boasts a ton of integrations into popular MLOps tools. The ZenML Stack concept ensures that these tools work nicely together, therefore bringing structure and standardization into the MLOps workflow.

However, ZenML assumes that the stack infrastructure for these tools is already provisioned. If you do not have deployed infrastructure, and want to quickly spin up combinations of tools on the cloud, the MLOps stack sister repository contains a series of Terraform-based recipes to provision such stacks. These recipes can be used directly with ZenML:

zenml stack recipe deploy <NAME_OF_STACK_RECIPE> --import

The above command not only provisions the given tools, but also automatically creates a ZenML stack with the configuration of the deployed recipe!

🗺 Roadmap

ZenML is being built in public. The roadmap is a regularly updated source of truth for the ZenML community to understand where the product is going in the short, medium, and long term.

ZenML is managed by a core team of developers that are responsible for making key decisions and incorporating feedback from the community. The team oversees feedback via various channels, and you can directly influence the roadmap as follows:

• Vote on your most wanted feature on our Discussion board.
• Start a thread in our Slack channel.
• Create an issue on our Github repo.

🙌 Contributing and Community

We would love to develop ZenML together with our community! Best way to get started is to select any issue from the good-first-issue label. If you would like to contribute, please review our Contributing Guide for all relevant details.

👩‍👩‍👧‍👦 Meet the Team

Have a question that's too hard to express on our Slack? Is it just too much effort to say everything on a long GitHub issue? Or are you just curious about what ZenML has been up to in the past week? Well, register now for the ZenML Office (Half) Hour to get your answers and more! It's free and open to everyone.

Every week, part of the ZenML core team will pop in for 30 minutes to interact directly with the community. Sometimes we'll be presenting a feature. Other times we just take questions and have fun. Join us if you are curious about ZenML, or just want to talk shop about MLOps.

We will host the gathering every Wednesday 8:30AM PT (5:30PM CET). Register now through this link, or subscribe to the public events calendar to get notified before every community gathering.

🆘 Getting Help

The first point of call should be our Slack group. Ask your questions about bugs or specific use cases, and someone from the core team will respond. Or, if you prefer, open an issue on our GitHub repo.

📜 License

ZenML is distributed under the terms of the Apache License Version 2.0. A complete version of the license is available in the LICENSE file in this repository. Any contribution made to this project will be licensed under the Apache License Version 2.0.

📖 Learn More

ZenML Resources	Description
🧘‍♀️ ZenML 101	New to ZenML? Here's everything you need to know!
⚛️ Core Concepts	Some key terms and concepts we use.
🚀 Our latest release	New features, bug fixes.
🗳 Vote for Features	Pick what we work on next!
📓 Docs	Full documentation for creating your own ZenML pipelines.
📒 API Reference	Detailed reference on ZenML's API.
🍰 ZenBytes	A guided and in-depth tutorial on MLOps and ZenML.
🗂️️ ZenFiles	End-to-end projects using ZenML.
👨‍🍳 MLOps Stacks	Terraform based infrastructure recipes for pre-made ZenML stacks.
⚽️ Examples	Learn best through examples where ZenML is used? We've got you covered.
📬 Blog	Use cases of ZenML and technical deep dives on how we built it.
🔈 Podcast	Conversations with leaders in ML, released every 2 weeks.
📣 Newsletter	We build ZenML in public. Subscribe to learn how we work.
💬 Join Slack	Need help with your specific use case? Say hi on Slack!
🗺 Roadmap	See where ZenML is working to build new features.
🙋‍♀️ Contribute	How to contribute to the ZenML project and code base.