Skip to main content

An MLOps Platform for Model Evaluation

Project description

An MLOps/LLMOps Platform

๐Ÿš€ ๏ธโ˜๏ธ Starwhale Cloud is now open to the public, try it! ๐ŸŽ‰๐Ÿป

Artifact Hub PyPI - Python Version Client/SDK UT Server UT Starwhale E2E Test Codecov Codecov

English | ไธญๆ–‡

What is Starwhale

Starwhale is an MLOps/LLMOps platform that brings efficiency and standardization to machine learning operations. It streamlines the model development liftcycle, enabling teams to optimize their workflows around key areas like model building, evaluation, release and fine-tuning.

products

Starwhale meets diverse deployment needs with three flexible configurations:

  • ๐Ÿฅ Standalone - Deployed in a local development environment, managed by the swcli command-line tool, meeting development and debugging needs.
  • ๐Ÿฆ… Server - Deployed in a private data center, relying on a Kubernetes cluster, providing centralized, web-based, and secure services.
  • ๐Ÿฆ‰ Cloud - Hosted on a public cloud, with the access address https://cloud.starwhale.cn. The Starwhale team is responsible for maintenance, and no installation is required. You can start using it after registering an account.

As its core, Starwhale abstracts Model, Runtime and Dataset as first-class citizens - providing the fundamentals for streamlined operations. Starwhale further delivers tailored capabilities for common workflow scenarios including:

  • ๐Ÿ”ฅ Models Evaluation - Implement robust, production-scale evaluations with minimal coding through the Python SDK.
  • ๐ŸŒŸ Live Demo - Interactively assess model performance through user-friendly web interfaces.
  • ๐ŸŒŠ LLM Fine-tuning - End-to-end toolchain from efficient fine-tuning to comparative benchmarking and publishing.

Starwhale is also an open source platform, using the Apache-2.0 license. The Starwhale framework is designed for clarity and ease of use, empowering developers to build customized MLOps features tailored to their needs.

framework

Key Concepts

๐Ÿ˜ Starwhale Dataset

Starwhale Dataset offers efficient data storage, loading, and visualization capabilities, making it a dedicated data management tool tailored for the field of machine learning and deep learning

dataset overview

import torch
from starwhale import dataset, Image

# build dataset for starwhale cloud instance
with dataset("https://cloud.starwhale.cn/project/starwhale:public/dataset/test-image", create="empty") as ds:
    for i in range(100):
        ds.append({"image": Image(f"{i}.png"), "label": i})
    ds.commit()

# load dataset
ds = dataset("https://cloud.starwhale.cn/project/starwhale:public/dataset/test-image")
print(len(ds))
print(ds[0].features.image.to_pil())
print(ds[0].features.label)

torch_ds = ds.to_pytorch()
torch_loader = torch.utils.data.DataLoader(torch_ds, batch_size=5)
print(next(iter(torch_loader)))

๐Ÿ‡ Starwhale Model

Starwhale Model is a standard format for packaging machine learning models that can be used for various purposes, like model fine-tuning, model evaluation, and online serving. A Starwhale Model contains the model file, inference codes, configuration files, and any other files required to run the model.

overview

# model build
swcli model build . --module mnist.evaluate --runtime pytorch/version/v1 --name mnist

# model copy from standalone to cloud
swcli model cp mnist https://cloud.starwhale.cn/project/starwhale:public

# model run
swcli model run --uri mnist --runtime pytorch --dataset mnist
swcli model run --workdir . --module mnist.evaluator --handler mnist.evaluator:MNISTInference.cmp

๐ŸŒ Starwhale Runtime

Starwhale Runtime aims to provide a reproducible and sharable running environment for python programs. You can easily share your working environment with your teammates or outsiders, and vice versa. Furthermore, you can run your programs on Starwhale Server or Starwhale Cloud without bothering with the dependencies.

overview

# build from runtime.yaml, conda env, docker image or shell
swcli runtime build --yaml runtime.yaml
swcli runtime build --conda pytorch --name pytorch-runtime --cuda 11.4
swcli runtime build --docker pytorch/pytorch:1.9.0-cuda11.1-cudnn8-runtime
swcli runtime build --shell --name pytorch-runtime

# runtime activate
swcli runtime activate pytorch

# integrated with model and dataset
swcli model run --uri test --runtime pytorch
swcli model build . --runtime pytorch
swcli dataset build --runtime pytorch

๐Ÿ„ Starwhale Evaluation

Starwhale Evaluation enables users to evaluate sophisticated, production-ready distributed models by writing just a few lines of code with Starwhale Python SDK.

import typing as t
import gradio
from starwhale import evaluation
from starwhale.api.service import api

def model_generate(image):
    ...
    return predict_value, probability_matrix

@evaluation.predict(
    resources={"nvidia.com/gpu": 1},
    replicas=4,
)
def predict_image(data: dict, external: dict) -> None:
    return model_generate(data["image"])

@evaluation.evaluate(use_predict_auto_log=True, needs=[predict_image])
def evaluate_results(predict_result_iter: t.Iterator):
    for _data in predict_result_iter:
        ...
    evaluation.log_summary({"accuracy": 0.95, "benchmark": "test"})

@api(gradio.File(), gradio.Label())
def predict_view(file: t.Any) -> t.Any:
    with open(file.name, "rb") as f:
        data = Image(f.read(), shape=(28, 28, 1))
    _, prob = predict_image({"image": data})
    return {i: p for i, p in enumerate(prob)}

๐Ÿฆ Starwhale Fine-tuning

Starwhale Fine-tuning provides a full workflow for Large Language Model(LLM) tuning, including batch model evaluation, live demo and model release capabilities. Starwhale Fine-tuning Python SDK is very simple.

import typing as t
from starwhale import finetune, Dataset
from transformers import Trainer

@finetune(
    resources={"nvidia.com/gpu":4, "memory": "32G"},
    require_train_datasets=True,
    require_validation_datasets=True,
    model_modules=["evaluation", "finetune"],
)
def lora_finetune(train_datasets: t.List[Dataset], val_datasets: t.List[Dataset]) -> None:
    # init model and tokenizer
    trainer = Trainer(
        model=model, tokenizer=tokenizer,
        train_dataset=train_datasets[0].to_pytorch(), # convert Starwhale Dataset into Pytorch Dataset
        eval_dataset=val_datasets[0].to_pytorch())
    trainer.train()
    trainer.save_state()
    trainer.save_model()
    # save weights, then Starwhale SDK will package them into Starwhale Model

Installation

๐Ÿ‰ Starwhale Standalone

Requirements: Python 3.7~3.11 in the Linux or macOS os.

python3 -m pip install starwhale

๐Ÿฅญ Starwhale Server

Starwhale Server is delivered as a Docker image, which can be run with Docker directly or deployed to a Kubernetes cluster. For the laptop environment, using swcli server start command is a appropriate choice that depends on Docker and Docker-Compose.

swcli server start

Quick Tour

We use MNIST as the hello world example to show the basic Starwhale Model workflow.

๐Ÿช… MNIST Evaluation in Starwhale Standalone

๐Ÿช† MNIST Evaluation in Starwhale Server

Examples

Documentation, Community, and Support

Contributing

๐ŸŒผ๐Ÿ‘PRs are always welcomed ๐Ÿ‘๐Ÿบ. See Contribution to Starwhale for more details.

License

Starwhale is licensed under the Apache License 2.0.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

starwhale-0.6.15.tar.gz (8.5 MB view details)

Uploaded Source

Built Distribution

starwhale-0.6.15-py3-none-any.whl (8.6 MB view details)

Uploaded Python 3

File details

Details for the file starwhale-0.6.15.tar.gz.

File metadata

  • Download URL: starwhale-0.6.15.tar.gz
  • Upload date:
  • Size: 8.5 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.19

File hashes

Hashes for starwhale-0.6.15.tar.gz
Algorithm Hash digest
SHA256 21f84e4a175c1b3854b872500db4bbfa9350707fca3c906d8dec0700cab11603
MD5 a9e474a2084f0098112d4f5d63988589
BLAKE2b-256 9f489d672554741e030e725fe08bdd23b89e940281d93bb9e1f055faa8cb2bbe

See more details on using hashes here.

File details

Details for the file starwhale-0.6.15-py3-none-any.whl.

File metadata

  • Download URL: starwhale-0.6.15-py3-none-any.whl
  • Upload date:
  • Size: 8.6 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.19

File hashes

Hashes for starwhale-0.6.15-py3-none-any.whl
Algorithm Hash digest
SHA256 517e53d68687692061e6ba2e1396aa42549cb789671e720344c9562f432412a4
MD5 6e3303976f46a114d02ea0bcd1481ede
BLAKE2b-256 34bb16b3e8f3d26b65308f4840a578f19a707868f2321b2fc60f805ddb3c2280

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page