An MLOps Platform for Model Evaluation
Project description
An MLOps/LLMOps Platform
๐ ๏ธโ๏ธ Starwhale Cloud is now open to the public, try it! ๐๐ป
English | ไธญๆ
What is Starwhale
Starwhale is an MLOps/LLMOps platform that make your model creation, evaluation and publication much easier. It aims to create a handy tool for data scientists and machine learning engineers. Starwhale helps you:
- ๐๏ธ Keep track of your training/testing dataset history including data items and their labels, so that you can easily access them.
- ๐งณ Manage your model packages that you can share across your team.
- ๐ Run your models in different environments, either on a Nvidia GPU server or on an embedded device like Cherry Pi.
- ๐ฅ Create a online service with interactive Web UI for your models.
Key Concepts
๐ฆ Starwhale Instance
Each deployment of Starwhale is called an instance. All instances can be managed by the Starwhale Client (swcli). You can start using Starwhale with one of the following instance types:
- ๐ป Starwhale Standalone: Rather than a running service, Starwhale Standalone is actually a repository that resides in your local file system. It is created and managed by the Starwhale Client (SWCLI). You only need to install SWCLI to use it. Currently, each user on a single machine can have only ONE Starwhale Standalone instance. We recommend you use the Starwhale Standalone to build and test your datasets, runtime, and models before pushing them to Starwhale Server/Cloud instances.
- ๐ Starwhale Server: Starwhale Server is a service deployed on your local server. Besides text-only results from the Starwhale Client (SWCLI), Starwhale Server provides Web UI for you to manage your datasets and models, evaluate your models in your local Kubernetes cluster, and review the evaluation results.
- โ๏ธ Starwhale Cloud: Starwhale Cloud is a managed service hosted on public clouds. By registering an account on https://cloud.starwhale.cn , you are ready to use Starwhale without needing to install, operate, and maintain your own instances. Starwhale Cloud also provides public resources for you to download, like datasets, runtimes, and models. Check the "starwhale/public" project on Starwhale Cloud for more details.
Starwhale tries to keep concepts consistent across different types of instances. In this way, people can easily exchange data and migrate between them.
๐ Starwhale Dataset
Starwhale Dataset offers efficient data storage, loading, and visualization capabilities, making it a dedicated data management tool tailored for the field of machine learning and deep learning
import torch
from starwhale import dataset, Image
# build dataset for starwhale cloud instance
with dataset("https://cloud.starwhale.cn/project/starwhale:public/dataset/test-image", create="empty") as ds:
for i in range(100):
ds.append({"image": Image(f"{i}.png"), "label": i})
ds.commit()
# load dataset
ds = dataset("https://cloud.starwhale.cn/project/starwhale:public/dataset/test-image")
print(len(ds))
print(ds[0].features.image.to_pil())
print(ds[0].features.label)
torch_ds = ds.to_pytorch()
torch_loader = torch.utils.data.DataLoader(torch_ds, batch_size=5)
print(next(iter(torch_loader)))
๐ Starwhale Model
Starwhale Model is a standard format for packaging machine learning models that can be used for various purposes, like model fine-tuning, model evaluation, and online serving. A Starwhale Model contains the model file, inference codes, configuration files, and any other files required to run the model.
# model build
swcli model build . --module mnist.evaluate --runtime pytorch/version/v1 --name mnist
# model copy from standalone to cloud
swcli model cp mnist https://cloud.starwhale.cn/project/starwhale:public
# model run
swcli model run --uri mnist --runtime pytorch --dataset mnist
swcli model run --workdir . --module mnist.evaluator --handler mnist.evaluator:MNISTInference.cmp
๐ Starwhale Runtime
Starwhale Runtime aims to provide a reproducible and sharable running environment for python programs. You can easily share your working environment with your teammates or outsiders, and vice versa. Furthermore, you can run your programs on Starwhale Server or Starwhale Cloud without bothering with the dependencies.
# build from runtime.yaml, conda env, docker image or shell
swcli runtime build --yaml runtime.yaml
swcli runtime build --conda pytorch --name pytorch-runtime --cuda 11.4
swcli runtime build --docker pytorch/pytorch:1.9.0-cuda11.1-cudnn8-runtime
swcli runtime build --shell --name pytorch-runtime
# runtime activate
swcli runtime activate pytorch
# integrated with model and dataset
swcli model run --uri test --runtime pytorch
swcli model build . --runtime pytorch
swcli dataset build --runtime pytorch
๐ Starwhale Evaluation
Starwhale Evaluation enables users to evaluate sophisticated, production-ready distributed models by writing just a few lines of code with Starwhale Python SDK.
import typing as t
import gradio
from starwhale import evaluation
from starwhale.api.service import api
def model_generate(image):
...
return predict_value, probability_matrix
@evaluation.predict(
resources={"nvidia.com/gpu": 1},
replicas=4,
)
def predict_image(data: dict, external: dict) -> None:
return model_generate(data["image"])
@evaluation.evaluate(use_predict_auto_log=True, needs=[predict_image])
def evaluate_results(predict_result_iter: t.Iterator):
for _data in predict_result_iter:
...
evaluation.log_summary({"accuracy": 0.95, "benchmark": "test"})
@api(gradio.File(), gradio.Label())
def predict_view(file: t.Any) -> t.Any:
with open(file.name, "rb") as f:
data = Image(f.read(), shape=(28, 28, 1))
_, prob = predict_image({"image": data})
return {i: p for i, p in enumerate(prob)}
๐ฆ Starwhale Fine-tuning
Starwhale Fine-tuning provides a full workflow for Large Language Model(LLM) tuning, including batch model evaluation, live demo and model release capabilities. Starwhale Fine-tuning Python SDK is very simple.
import typing as t
from starwhale import finetune, Dataset
from transformers import Trainer
@finetune(
resources={"nvidia.com/gpu":4, "memory": "32G"},
require_train_datasets=True,
require_validation_datasets=True,
model_modules=["evaluation", "finetune"],
)
def lora_finetune(train_datasets: t.List[Dataset], val_datasets: t.List[Dataset]) -> None:
# init model and tokenizer
trainer = Trainer(
model=model, tokenizer=tokenizer,
train_dataset=train_datasets[0].to_pytorch(), # convert Starwhale Dataset into Pytorch Dataset
eval_dataset=val_datasets[0].to_pytorch())
trainer.train()
trainer.save_state()
trainer.save_model()
# save weights, then Starwhale SDK will package them into Starwhale Model
Installation
๐ Starwhale Standalone
Requirements: Python 3.7~3.11 in the Linux or macOS os.
python3 -m pip install starwhale
๐ฅญ Starwhale Server
Starwhale Server is delivered as a Docker image, which can be run with Docker directly or deployed to a Kubernetes cluster. For the laptop environment, using swcli server start
command is a appropriate choice that depends on Docker and Docker-Compose.
swcli server start
Quick Tour
We use MNIST as the hello world example to show the basic Starwhale Model workflow.
๐ช MNIST Evaluation in Starwhale Standalone
- Use your own Python environment, follow the Standalone quickstart doc.
- Use Google Colab environment, follow the Jupyter notebook example.
๐ช MNIST Evaluation in Starwhale Server
- Run it in the your private Starwhale Server instance, please read Server installation(minikube) and Server quickstart docs.
- Run it in the Starwhale Cloud, please read Cloud quickstart doc.
Examples
-
๐ LLM:
- ๐ OpenSource LLMs Leaderboard: Evaluation, Code
- ๐ข Llama2: Run llama2 chat in five minutes, Code
- ๐ฆ Stable Diffusion: Cloud Demo, Code
- ๐ฆ LLAMA evaluation and fine-tune
- ๐น Text-to-Music: Cloud Demo, Code
- ๐ Code Generation: Cloud Demo, Code
-
๐ Fine-tuning:
- ๐ Baichuan2: Cloud Demo, Code
- ๐ซ ChatGLM3: Cloud Demo, Code
- ๐ฆ Stable Diffusion: Cloud Demo, Code
-
๐ฆฆ Image Classification:
- ๐ปโโ๏ธ MNIST: Cloud Demo, Code.
- ๐ฆซ CIFAR10
- ๐ฆ Vision Transformer(ViT): Cloud Demo, Code
-
๐ Image Segmentation:
- Segment Anything(SAM): Cloud Demo, Code
-
๐ฆ Object Detection:
- ๐ฆ YOLO: Cloud Demo, Code
- ๐ฏ Pedestrian Detection
-
๐ฝ๏ธ Video Recognition: UCF101
-
๐ฆ Machine Translation: Neural machine translation
-
๐ Text Classification: AG News
-
๐๏ธ Speech Recognition: Speech Command
Documentation, Community, and Support
-
Visit Starwhale HomePage.
-
More information in the official documentation.
-
For general questions and support, join the Slack.
-
For bug reports and feature requests, please use Github Issue.
-
To get community updates, follow @starwhaleai on Twitter.
-
For Starwhale artifacts, please visit:
- Python Package on Pypi.
- Helm Charts on Artifacthub.
- Docker Images on Docker Hub, Github Packages and Starwhale Registry.
-
Additionally, you can always find us at developer@starwhale.ai.
Contributing
๐ผ๐PRs are always welcomed ๐๐บ. See Contribution to Starwhale for more details.
License
Starwhale is licensed under the Apache License 2.0.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for starwhale-0.6.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f28fbf4ad8870b19903b185405e8a40a708673953211619ec0f8a36f00d65cb2 |
|
MD5 | 1de2f94abe2bc78dc724cde3395c7302 |
|
BLAKE2b-256 | c50f0133b2335c4812be43d074238f45f4944b6776fd1911f9f8c8599a4857ce |